How explainable AI affects understandability, trust and usability

Disinformation is false or misleading information that is intentionally spread to mislead the public.

When it comes to disinformation, AI is a bit of a double-edged sword. On the one hand, AI is making it easier than ever to generate disinformation in the form of text and deepfake imagery. At the same time, it also makes it easier to combat disinformation, which previously would have required a lot of manual work.

However, using AI for potentially sensitive topics such as disinformation comes with challenges relating to transparency, reliability, and user acceptance of algorithmic decisions. Being technically accurate is not enough, as the AI must also be able to overcome scepticism, concerns about political bias, censorship, and fairness (side note: These issues are less apparent with misinformation, which differs from disinformation because it is shared without malicious intent.).

Explainable AI (XAI) is a set of efforts to improve the transparency and trustworthiness of AI by making its inner workings more understandable to laypeople, for instance by providing contextual explanations about disinformation debunking results (side note: The AI4Debunk project, which our university of applied sciences is part of, uses this approach as well.).

This week’s paper describes how design science research (DSR) was used to iteratively design and test a front-end design for XAI. What follows are some insights about how users perceive XAI, and a set of design guidelines for developing responsible XAI systems.

The paper starts with a systematic review of literature on explainable AI, which shows that XAI is a relatively new phenomenon, mostly applied within healthcare (16%) and deception detection (14%) (side note: I’m not entirely sure what this means, especially because the authors explicitly mention that only three papers in the latter category are about online disinformation.). Most XAI deals with textual input. The reviewed sample included only one study that focused on visual inputs, for identifying deepfake videos.

In general, the literature suggests that explanations must be informative but not overwhelming, as this may result in frustration or even outright rejection of AI-based systems. The findings can be summarised into the following design guidelines for XAI to detect disinformation on digital platforms:

1. Preserve the original GUI of the platform so that users do not have to transition to and learn different interfaces.

2. Balance simplicity and clarity: explanations must be clear and comprehensible without becoming overly complex.

3. Empower inexperienced users by making sure that features are accessible to them, and that they remain in control of decision making.

4. Use confidence scores to show how certain an AI model is about its prediction.

5. Make use of colour for critical insights, e.g. by highlighting important keywords in text and using clear colour schemes.

6. Provide natural language explanations on demand, which are initially hidden but can be expanded only for users who need detailed information.

7. Exclude conversational agents at first to prevent user frustration (side note: This guideline is based on papers that were published before ChatGPT became a thing. Make of that what you wish.). Focus on delivering clear and direct explanations through other features instead.

8. Evaluate explainability features to assess their effectiveness and impact in real-world scenarios.

9. Iterative development and improvement helps your application to adapt to evolving needs and technological advancements.

Based on these guidelines, the researchers designed mockups for an XAI-based disinformation detection tool for digital platforms. I won’t go into the details here (side note: The original article has screenshots, so you can see for yourself.), but the designs include warnings that may be shown either before, above, or below posts and that contain elements such as confidence scores in the form of percentages or on a simplified ordinal scale (low, medium or high) and texts with highlights.

A small qualitative user test with eight participants showed that most prefer having warnings displayed above posts. When asked to choose between brief and detailed explanations for AI classifications, all eight participants prefer the longer version of the text.

Opinions about confidence scores, on the other hand, were mixed. Some participants always find them useful, others only if they exceed some specific threshold – ranging from as low as 20% to as high as 80% – or not at all. This suggests that confidence scores may pose challenges for laypersons. Regardless, when participants have to choose between percentages or an ordinal scale, all except one favour a percentage display.

Unlike the confidence scores, there is full consensus about the importance of highlighting text passages that are relevant for classification, with many claiming that it helps them better understand the content and the claims made by AI. Such text passages can either be highlighted within the original text or cited in an explanatory text. Most participants favour the first.

For the second cycle, a quantitative test was conducted with 344 participants to determine the effectiveness using a refined set of prototypes. Now, the tool either shows a binary classification without any explanations, with explanations in an expandable window, or with a confidence score in addition to the explanations. Participants are randomly assigned to one of these three versions.

The results show no significant differences in understandability between the three versions, which suggests that the inclusion of XAI doesn’t actually enhance understandability. A small but significant difference in trust scores was found between the control group and the two groups that saw a version with explanations, but the presence of XAI does not lead to higher trust overall. There are no significant differences in usability scores across all three versions, suggesting that XAI does not enhance usability over a basic AI system. Finally, the results suggest that introducing explanations actually reduces participants’ agreement with the system’s classifications.

What impact do demographic and personal characteristics have on these scores? Interestingly, academic background and prior experience with AI have no real effect on the scores. Age, on the other hand, does; older individuals tend to find the tool less understandable, trustworthy and usable. Individuals with a higher propensity to trust perceive greater understandability and report higher levels of trust. Interestingly, female participants also report significantly lower levels of usability compared to male participants.

Overall, the findings suggest that factors such as demographic and individual characteristics (trust) play a larger role in shaping user experiences and perceptions than the inclusion of XAI. This could imply that tool developers should carefully tailor explanation formats to specific user profiles. Another reading is that the provided explanations may have introduced additional uncertainty or complexity, making the AI’s decision process more transparent but also more challenging to interpret. The level of detail in explanations may also have prompted users to scrutinise the AI’s classifications more critically than they would have otherwise.

These findings lead the researchers to come up with the following refined guidelines for XAI:

1. Integrate explanations seamlessly into the user experience, focusing mostly on presentation.

2. Simplify explanations to avoid cognitive overload, i.e. make sure they are straightforward and relevant to the user’s current context.

3. Prioritise trustworthiness in design to build credibility for inexperienced users, with enough supportive elements that reinforce trust and reliability.

4. Make explanations optional by offering customisable explanation features so that details are only shown when needed.

5. Consider user trust and cognitive factors, e.g. by accounting for cognitive differences such as those related to age.

6. Address demographic and individual differences through adaptability in design. Targeted user research may be needed to tailor explanations effectively.

7. Refine and test explanation mechanisms continuously, based on user feedback and iterative testing.