“AI should be able to say ‘I’m Not Sure’ on its own.”

A new approach has been proposed to address the problem of “overconfidence”—one of the most critical risks of artificial intelligence (AI) in areas such as autonomous driving and medical diagnosis, where AI shows high confidence in incorrect predictions. A KAIST research team has developed a training method that enables AI to recognize situations involving unfamiliar or unseen knowledge, laying the foundation for reducing overconfidence and improving reliability.

KAIST (President Kwang Hyung Lee) announced on the 27th of April that a research team led by Distinguished Professor Se-Bum Paik from the Department of Brain and Cognitive Sciences has identified that random initialization—widely used in deep learning (an AI technique that learns from data using artificial neural networks)—may be a fundamental cause of overconfidence in AI.

To address this, the research team proposed a “warm-up” strategy in which the neural network is briefly trained using random noise (meaningless arbitrary input data) before learning from real data.

The research team found that AI overconfidence already appears at the initialization stage, which can propagate and cause significant errors during subsequent training. In fact, when random data were input into a randomly initialized neural network, the model exhibited high confidence despite not having learned anything. This characteristic can lead to hallucination in generative AI, where false information is produced in a plausible manner.

The research team found clues for solving this issue in the biological brain. The human brain forms neural circuits through “spontaneous neural activity”—brain signals generated without external input—even before birth.

Applying this concept to artificial neural networks, the researchers introduced a “warm-up phase” in which the network undergoes brief pre-training with random noise inputs before actual learning. This corresponds to a process in which AI adjusts its own uncertainty before starting data learning. After the warm-up process, the AI model’s initial confidence is aligned to a low level close to chance, significantly reducing the overconfidence bias observed in conventional initialization.

In other words, before learning from real data, the model first learns the state of “I don’t know anything yet.”
As a result, the model’s accuracy (how often predictions are correct) and confidence (how strongly the model believes its predictions) naturally become aligned.

A notable difference was also observed in responses to unseen data. While conventional models tend to give incorrect answers with high confidence even for data they have not encountered during training, models with warm-up training showed a clear improvement in their ability to lower confidence and recognize that they “do not know.”

This also led to strong performance in out-of-distribution detection, which refers to identifying data that differ from the training distribution.

This study suggests the possibility that AI can go beyond simply producing correct answers and develop the ability to distinguish “what it knows” from “what it does not know”—that is, meta-cognition, the ability to recognize its own cognitive state.

Professor Se-Bum Paik stated, “This study demonstrates that by incorporating key principles of brain development, AI can recognize its own knowledge state in a way that is more similar to humans,” adding, “This is important because it helps AI understand when it is uncertain or might be mistaken, not just improve how often it gives the right answer.”

This technology is expected to be applied not only to fields requiring high reliability, such as autonomous driving, medical AI, and generative AI, but also to the initialization methods of nearly all deep learning models, making it a key technology for improving overall AI reliability.

This study, with Jeonghwan Cheon, a master’s student in the Department of Brain and Cognitive Sciences at KAIST (currently serving as a Private in the Republic of Korea Army), as the first author, was published online on April 9, 2026, in the international journal Nature Machine Intelligence, and was selected as a notable paper and featured in News & Views.
※ Paper title: “Brain-inspired warm-up training with random noise for uncertainty calibration,” DOI: 10.1038/s42256-026-01215-x
※ News & Views article: Learning to be uncertain before learning from data, DOI: 10.1038/s42256-026-01205-z

This research was supported by the Basic Science Research Program of the National Research Foundation of Korea and the KAIST Singularity Professor Research Program.