Large language models (LLMs), the artificial intelligence (AI) systems supporting ChatGPT and similar conversational platforms, are now used by many people global to find and summarize information and create different types of text. In spite of their widespread use, these models still have notable limitations.
When generating text or solutions to user queries, current LLMs do now not depend only on patterns observed and data analyzed during training. They also can retrieve information from external resources, such as websites, databases and search engines.
Giving LLMs access to external resources permits them to generate responses which are up to date and more comprehensive. If a model cannot reliably judge the reliability of external sources of information, moreover, it can create text content that is untrustworthy or misguided.
Researchers at the University of Michigan currently evolved Learn2Discern (L2D), a latest framework that can be used to evaluate whether or not LLMs update their “beliefs” correctly when offered with new information. Using this framework, which was provided in a paper posted to the arXiv preprint server, they showed that many leading conversational AI tools still struggle to discern between dependable sources of information and sources which might be merely well-known or broadly cited.
“LLMs are increasingly used with external knowledge sources just like the internet. Do they weigh information correctly—updating more for reliable sources (source discernment) and more when claims bring priors to the truth (fact discernment)?” wrote Joshua Ashkinaze, Laura Kurek and their colleagues in their paper. “We formalize this as information discernment and introduce L2D, an experimental framework and benchmark grounded in three normative axioms with interpretable metrics.”
Measuring how properly LLMs compare information
Ashkinaze and his collaborators need to broaden new tools that could be used to test how well LLMs assess new information and update their “beliefs” accordingly. First, they developed L2D, a framework and benchmark that can be used to quantify how reliably LLMs incorporate new external information into their existing knowledge.
The L2D framework describe key principles for a how an LLM need to ideally behave after getting new information from external sources. Rather than of certainly searching at whether or not an LLM’s answers are correct, the framework examines how a model revises its answers after incorporating external information.
To compare the validity in their proposed framework, the team executed an initial examine regarding 299 human participants. Participants were asked whether LLMs should follow the principles delineated by L2D when updating their “beliefs” in response to new information.
“To establish external validity, a pre-registered, quota-matched user study confirms that real LLM users endorse all 3 axioms and report that violations lessen their trust and usage intent,” wrote the authors.
Once they confirmed that their framework extensively showed how users thought AI must behave, the researchers used L2D to evaluate 13 unique LLMs. The assessed models covered Claude 3.5 Sonnet, Gemini 2.0 Flash, Gemini 2.5 Flash, GPT 3.5 Turbo, GPT-4.1, GPT-4.1 Mini, GPT-4o, GPT-4o-Mini, GPT-5, GPT-5-Mini, Mixtral 8x7b, Qwen 2.5-14b and Qwen 2.5-7b.
“Across 13 models and almost 670K trials, we find consistent failures across both dimensions: models carry out near chance on source and truth discernment, depend upon source popularity twice as a much as source reliability, and update roughly equally whether a claim improves or worsens their position relative to the ground truth.”
Toward more dependable AI systems
Essentially, Ashkinaze and his colleagues found that most of the LLMs they examined did not constantly give greater weight to information originating from dependable sources than to information from less reliable sources. In addition, the models often updated their “beliefs” by similar amounts regardless of whether or not new information moved them closer or further from the correct answer.
LLMs seemed to be better at incorporating external knowledge when their initial “beliefs” have been already relatively accurate. While large and more modern models were better at discerning among true and false information, they commonly failed to reliably gauge the reliability of external sources.
The survey responses collected via the researchers and the dataset they compiled during their experiments are available online. In the future, their framework may be used to evaluate other LLMs’ ability to discern among unreliable and reliable resources of information, probably contributing to the development of increasingly truthful conversational AI platforms.
