The way the public consumes information about climate science has shifted significantly in recent years. While earlier studies focused on traditional sources like search engine results, newspapers, and social media feeds, large language models are now a major part of the information landscape. A new investigation seeks to understand how these AI systems handle climate change topics, a subject that requires both factual accuracy and nuanced interpretation.
Researchers from University College London and the United Nations Educational, Scientific, and Cultural Organization, known as UNESCO, led this study. A primary challenge in this field is that AI interactions are ephemeral, meaning the specific information served to a user is confined to their individual chat session and is not easily tracked. To overcome this, the team developed a multi-angled approach that tested various prompts and evaluated the resulting outputs.
The testing protocol included benchmarks for simple factual questions, which may already be embedded in the training data of many current systems. However, the study also examined more complex interactions. This included scenarios where users pushed back on an answer, believing themselves to be correct, to see how the models handled the dialogue.
A common criticism in AI research is that it often focuses on a single model, which may become outdated quickly. To address this, the study utilized a broad range of 33 different models. This included multiple versions of major frontier models as well as open-weight alternatives. The flagship models tested included Claude Fable 5, Claude Opus 5, Gemini 3.1 Pro Preview, Gemini 3.6 Flash, and GPT-5.6 Sol.
All participating models were scored based on their ability to answer 600 questions drawn from several benchmark sets. These questions covered multiple choice items from textbooks, reading comprehension tasks based on snippets from scientific papers, and queries regarding the confidence levels of climate scientists as stated in IPCC reports.
The questions regarding scientific confidence proved to be the most difficult for the AI systems. The flagship frontier models answered approximately 78 percent of these questions correctly. In contrast, the smallest open-weight models managed to answer only about 47 percent correctly, highlighting a significant gap in performance for this specific type of query.
Performance on other benchmarks showed a different pattern. For the remaining question sets, both frontier models and large open-weight models generally achieved scores around 90 percent. The smaller open-weight models performed moderately, with scores in the 70s. This suggests that while smaller models can handle standard factual recall, they struggle more with nuanced scientific interpretations.
The findings indicate that while large language models are becoming a primary source of information, their reliability varies depending on the complexity of the question. Users seeking detailed scientific consensus or confidence levels should be aware that even advanced AI systems may provide incorrect or incomplete information in these areas.
The study underscores the need for continued evaluation of AI models as they are integrated into daily information consumption. As these tools become more prevalent, understanding their specific weaknesses in handling scientific data is crucial for both developers and the general public.
The full list of models and detailed results can be found in the published report, offering a comprehensive view of the current state of AI performance in climate science communication.







