UOsaka researchers quantitatively compare the ways humans and machines conceptualize words

Osaka, Japan – By now, it’s no secret that large language models (LLMs) are experts at mimicking natural language. Trained on vast troves of data, these models have proven themselves capable of generating text so convincing that it regularly appears humanlike to readers. But is there any difference between how we think about a word and how an LLM does?
In an article published this month in Behavior Research Methods, a team of researchers from The University of Osaka studied the “impressions” LLMs form of words along several different dimensions and compared them with how humans conceptualize words.

The experiment was straightforward: the research team took 695 words acquired early in life by children and asked different LLMs to rate each one in terms of attributes such as concreteness, socialness, and arousal. They then quantitatively compared these model-based ratings with human norms from prior studies.

“We observe relatively strong correlations between the two groups for some attributes, such as concreteness, imageability, and body-object interaction,” says Hiromichi Hagihara, lead author of the study. “These findings suggest that LLMs, despite not interacting directly with the world, may hold certain forms of embedded knowledge because humans implicitly encode information about the world into language.”

However, although some attributes showed strong correlations in the ratings, humans and LLMs did not always agree. “As an example, evaluations of iconicity, which is the degree to which a word’s form and meaning are similar, diverge substantially between humans and LLMs,” explains Kazuki Miyazawa, a co-author of the study. “For many attributes, the discrepancy is greatest for words like prepositions and conjunctions. Even for attributes like concreteness, where there is high overall agreement, human ratings vary widely across these words, whereas LLMs tend to assign consistently low ratings. This suggests that LLMs do not perceive these kinds of words the way we do.”

Taking these results one step further, the team examined how well human- and LLM-based attribute ratings predicted the age of acquisition of these words. They found that while some attributes exhibited similar predictive patterns across humans and LLMs, systematic biases also emerged, depending on the model.

“In some cases, LLM-based ratings tend to overestimate how strongly certain word features are related to how early words are learned, compared with human ratings,” says Hagihara. “For example, concreteness is negatively correlated with age of acquisition in human data, which means children tend to learn more concrete words earlier. While LLM-based ratings capture this general trend, some models exaggerate the strength of these relationships for particular features.”

The team believes that the results of this study can be used to build LLMs that approximate humans cognitively or serve as complementary tools in psychological studies. One day, such LLMs could offer new perspectives on how humans learn and process language.
###
The article, “How well do large language models mirror human cognition of word concepts? A comparison of psychological ratings for early-acquired English words,” was published in Behavior Research Methods at https://doi.org/10.3758/s13428-025-02938-2

About The University of Osaka
The University of Osaka was founded in 1931 as one of the seven imperial universities of Japan and is now one of Japan's leading comprehensive universities with a broad disciplinary spectrum. This strength is coupled with a singular drive for innovation that extends throughout the scientific process, from fundamental research to the creation of applied technology with positive economic impacts. Its commitment to innovation has been recognized in Japan and around the world. Now, The University of Osaka is leveraging its role as a Designated National University Corporation selected by the Ministry of Education, Culture, Sports, Science and Technology to contribute to innovation for human welfare, sustainable development of society, and social transformation.
Website: https://resou.osaka-u.ac.jp/en