Wednesday, November 18, 2026

KAIST Boosts Accuracy of AI That Reads Lips...Wins EMNLP Outstanding Paper Award

Input
2026-11-15 12:00:00
Updated
2026-11-15 12:00:04
Professor Yong-man Noh (left), the corresponding author on the research team led by Professor Noh of the Korea Advanced Institute of Science and Technology (KAIST)'s School of Electrical and Electronic Engineering, and postdoctoral researcher Jeong-hun Yeo, the first author, pose for a commemorative photo holding the certificate for the Outstanding Paper Award at EMNLP 2026. Provided by KAIST

[Financial News] South Korean researchers have developed artificial intelligence (AI) technology that can accurately understand what someone is saying from lip movements alone in noisy places. The technology is expected to be used in communication tools that convey speech masked by noise as text or help people with hearing disabilities or hearing loss communicate.
The Korea Advanced Institute of Science and Technology (KAIST) said on the 15th that a research team led by Professor Yong-man Noh of its School of Electrical and Electronic Engineering had developed technology to improve the accuracy of visual speech recognition (VSR), which converts speech into text by observing lip movements.
The team used a diffusion large language model (DLLM), which repeatedly fills in blanks in a sentence, to decode ambiguous lip movements using the surrounding context. Existing AI systems in widespread use generated sentences sequentially, starting with the first word, so choosing an incorrect word at the beginning could affect the rest of the sentence.
The team changed the approach from “reading in order from the beginning” to “reading the clearest parts first.” It first identifies words with clear lip movements, then uses them as clues to decode the remaining parts. Like fitting together the pieces with the clearest shapes in a puzzle before filling in the surrounding pieces, the system gradually resolves ambiguous parts by considering information from both before and after them in the sentence.
The team also improved the training method. It divided training into two stages: first teaching the model to connect lip movements in videos with written text, then teaching it sentence length. When decoding actual speech, the model also estimates several possible sentence lengths based on the video length, compares the results, and selects the sentence it considers more reliable.
As a result, the team recorded a word error rate of 19.4% using the 433-hour lip-reading video dataset LRS3 as training data with reference transcripts. Word error rate measures how often words are incorrectly recognized, omitted, or added compared with the reference sentence; a lower rate indicates better performance. This is the lowest error rate among existing methods that used only LRS3 as training data with reference transcripts.
The technology could be used in communication aids that convey lip movements as text in environments where it is difficult to hear. For example, it could be developed into a device that understands workers' speech at industrial sites with loud machinery or helps people with hearing disabilities or hearing loss communicate.
The team plans to continue its research by reducing the size and computational requirements of the AI model so it can be used on a range of devices. A next step is to enable it to operate in real time without an internet connection on wearable devices or robots.
Professor Yong-man Noh said, “The significance of this research lies in demonstrating that simply flexibly controlling the order in which AI generates words can overcome the fundamental limitations of speech recognition involving visual ambiguity,” adding, “It will lay the foundation for physical AI that can be used with confidence across a range of devices in everyday life and industrial settings.”
The study was conducted by postdoctoral researcher Jeong-hun Yeo of KAIST's School of Electrical and Electronic Engineering as first author, doctoral researchers Chae-won Kim and Hyeong-seop Ra as co-authors, and Professor Yong-man Noh as corresponding author. The paper received the Outstanding Paper Award at EMNLP 2026, a major international conference in natural language processing.

[email protected] Yeon Ji-an Reporter