When a conversational AI character in an immersive environment says something false, how should the system alert the user? Researchers have now explored this question in a proof-of-concept study, which tested three techniques for alerting users to hallucinated statements from “embodied” AI agents in virtual reality (VR) systems.
While all three techniques improved the ability of users to identify untrue statements, they differed in how users experienced and interpreted the warnings. The findings can inform developers’ decisions about how to help users recognize potentially unreliable statements and improve their experience with embodied AI agents in applications such as educational programs. This is the first study to assess the performance of hallucination awareness cues with regard to embodied conversational agents. Embodied agents are AI-powered conversational agents that have a graphical representation capable of using gestures, facial expressions and speech to enhance computer-human interaction.
AI-powered conversational agents are increasingly common in applications ranging from customer service to educational tools. However, the generative AI systems that power these agents are subject to making untrue statements, known as “hallucinations.” To address this problem, many conversational agents use additional verification methods to check the agent’s statements. For example, if a verification method identifies a statement as potentially unreliable, the system might highlight the relevant text or display a warning to the user.
“These verification cues are relatively straightforward in text-based conversational interfaces,” says Qiao Jin, corresponding author of a paper on the work and an assistant professor of computer science at North Carolina State University. “For example, you can highlight the relevant words or place a source link beside them, and users can return to the text later.
“However, it is less clear how to communicate uncertainty and provenance cues in immersive environments like VR, where conversational agents are embodied as human-like characters that speak directly to users and their spoken responses pass by quickly.”
“Our evaluation of these cues looks not only at which cues are most effective at alerting users to potential hallucinations, but also which techniques best preserve the user’s sense of immersion in the VR experience,” says Xiaoran Yang, first author of the paper and a Ph.D. student at NC State.
For this study, the researchers recruited 24 college students and had them interact with a VR system featuring an embodied conversational agent that resembles a woman wearing a suit. Study participants were faced with multiple hallucinated statements under four different conditions.
One condition included embodied cues, in which the conversational agent made physical gestures associated with the hallucinated statement – such as crossing its arms when it was uncertain. A second condition included icon cues, in which an icon would appear next to the agent notifying users of additional information they should check regarding a statement. A third condition included text cues, with all of the agent’s statements appearing as subtitles that can be highlighted with different colors to give users information about potential uncertainties. The fourth condition was the control, in which no additional cues were given.
The researchers assessed each study participant’s ability to detect hallucinations under each condition and conducted an after-study questionnaire and interview to learn more about how well participants interpreted the various cues – as well as the extent to which the cues disrupted each participant’s experience in the VR.
“All three cues outperformed the control at helping users identify hallucinations,” says Yang. “But each type of cue had its own particular strengths.”
The researchers found that embodied cues were associated with higher levels of immersion in the VR experience and higher trust in the embodied agent. Text cues offered the clearest interpretability, meaning it did the best job of helping users identify hallucinations. However, text cues also distracted users from following other things taking place in the virtual environment. Icon cues offered a middle ground. The icons had better interpretability than the embodied cues while causing less disruption to immersion than text cues.
“Our findings here suggest that there is a lot of potential for how we can use these cues to improve the user experience,” says Yang.
“By choosing different cues, or combinations of cues, in different contexts, we can improve a system’s ability to communicate clearly and effectively with users about potential hallucinations while limiting the impact on the user’s experience,” says Jin.
“It’s also worth noting that additional work will need to be done with regard to the embodied cues,” says Jin. “We want to make sure that the body language of the cues can be understood by users with different cultural backgrounds. For example, avoiding eye contact may indicate uncertainty in one cultural context, but indicate respect or deference in another.”
The paper, “Signals of AI Hallucination: Designing Hallucination-Aware Cues for Embodied Conversational Agents in VR,” will be presented Oct. 8 at the IEEE International Symposium on Mixed and Augmented Reality (ISMAR) in Bari, Italy. The paper was co-authored by Yang Zhan, Yuxuan Huang, Yichen Yu and Noboru Matsuda of NC State; Xie He of Carnegie Mellon University; and Zhuo Wang of Xi’an Jiaotong-Liverpool University.