Understanding speech from lip movements alone is difficult, particularly in noisy environments. A research team at the Korea Advanced Institute of Science and Technology (KAIST) has developed an AI technology that first recognizes words it can identify with high confidence and then uses the surrounding context to resolve ambiguous parts of the sentence. The technology is expected to enable communication without relying on sound. For example, it could convert speech obscured by noise into text or help people who are deaf or hard of hearing follow conversations.
On November 15, KAIST (President Choongsik Bae) announced that a research team led by Professor Yong Man Ro from the School of Electrical Engineering had developed a technology that improves the accuracy of visual speech recognition (VSR), which converts lip movements into text. The team used a diffusion large language model (DLLM), which generates a sentence by repeatedly filling in masked positions. This allows the model to decode ambiguous lip movements using words that appear both before and after them.
Lip reading is challenging because different sounds can produce very similar lip movements. In English, for example, /p/, /b/, and /m/ are nearly indistinguishable based on visual information alone. Widely used AI models generate sentences in a fixed order, from the first word to the last. As a result, if the model selects an incorrect word early in a sentence, the error can affect the words that follow.
The research team replaced this first-to-last order with an approach that decodes the most certain parts first. The model first identifies words associated with clear lip movements and then uses those words as context to decode the remaining positions. By considering information from both earlier and later parts of the sentence, the model progressively resolves ambiguous positions.
The researchers also improved the training process by dividing it into two stages. In the first stage, the model learns to align lip movements in the video with the corresponding text. In the second stage, it learns to model sentence length. During decoding, the model estimates several plausible sentence lengths based on the video duration and generates a candidate sentence for each length. It then evaluates the candidates based on the likelihood of each sentence length and the model’s confidence in its predictions, selecting the most reliable sentence.
The team trained the model using LRS3, a dataset containing 433 hours of lip-reading video, as its labeled training data, meaning that the videos are paired with ground-truth transcripts. The model achieved a word error rate of 19.4%. Word error rate measures the proportion of words that are incorrectly recognized, omitted, or added compared with the correct transcript. A lower value indicates better performance. This is the lowest error rate reported among existing methods that use only LRS3 as labeled training data.
The technology could be used in communication tools that convert lip movements into text in situations where speech is difficult to hear. For example, it could help determine what workers are saying at industrial sites with loud machinery. It could also be developed into devices that support communication for people who are deaf or hard of hearing.
The team plans to continue research on reducing the size and computational cost of the AI model so that it can be deployed on a wider range of devices. A further goal is to enable the technology to run in real time on wearable devices and robots without an internet connection.
“This study is meaningful because it demonstrates that flexibly controlling the order in which AI generates words can, by itself, overcome a fundamental limitation of speech recognition caused by visual ambiguity,” said Professor Yong Man Ro. He added that the research would lay the foundation for physical AI that people can reliably use on a variety of devices in everyday life and at industrial sites.
Dr. Jeong Hun Yeo, a postdoctoral researcher in the School of Electrical Engineering at KAIST, is the first author. Ph.D. candidates Chae Won Kim and Hyeongseop Rha are co-authors, and Professor Yong Man Ro is the corresponding author. The paper received an Outstanding Paper Award at the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP 2026), a top-tier international conference in natural language processing.
Paper title: Diffusion Large Language Models for Visual Speech Recognition
DOI:
https://doi.org/10.48550/arXiv.2605.28456
This research was supported in part by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (Ministry of Science and ICT) (No. RS-2020-II200004, Development of Previsional Intelligence based on Long-Term Visual Memory Network) and the KAIST Jang Young Sil Fellow Program.