Latent Concept-based Explanation of NLP Models

Interpreting and understanding the predictions made by deep learning models poses a formidable challenge due to their inherently opaque nature. Many previous efforts to explain these predictions rely on input features, specifically, the words within NLP models. However, such explanations are often less informative due to the discrete nature of the words and their lack of contextual verbosity. To address this limitation, we introduce Latent Concept Attribution (LACOAT), which generates explanations for predictions based on latent concepts. Our intuition is that a word can exhibit multiple facets depending on the context in which it is used. Therefore, given a word in context, the latent space derived from our training process reflects a specific facet of that word. LACOAT functions by mapping the representations of salient input words into the training latent space, enabling it to provide latent context-based explanations of the prediction.