Supervisors:
Primary: Nicolas Baumard (DR CNRS, Institut Jean Nicod, ENS-PSL)
Co-advisor: Hugo Mercier (DR CNRS, Institut Jean Nicod, ENS-PSL)
- Context
Funding and environment – This PhD will be conducted within the framework of PR[AI]RIE-PSAI, at the interface between cognitive science and experimental psychology. It will benefit from the interdisciplinary research environment of the Institut Jean Nicod (ENS-PSL / CNRS / EHESS), where close collaboration between cognitive scientists, AI researchers, and social scientists enable the development of novel approaches to understanding human–AI interaction. The project builds directly on a line of work developed within the host institution on epistemic vigilance (i.e., the cognitive mechanisms by which humans evaluate the reliability of communicated information) and extends it to the novel challenge posed by AI-generated content.
Scientific context – Large language models (LLMs) have rapidly become primary tools for information seeking, including in political and news context. In AI-augmented search engines and chatbots connected to real-time data, LLMs now act as curators of information for billions of users, directly shaping what content is seen and how it is interpreted (Prama & Islam, 2025). While this lowers barriers to access, it also introduces significant risks: LLMs can generate convincing but inaccurate or unverifiable claims (“hallucinations”) and reproduce biases, often with high confidence (Massenon et al., 2025; Weidinger et al., 2021).
Large-scale empirical studies have shown that between 25–30% of political statements produced by LLMs contain inaccurate claims, and that these claims can be just as persuasive as accurate ones (Hackenburg et al., 2025). At the same time, LLMs have been shown to exhibit systematic political leanings and to influence users’ political views (Potter et al., 2024), which can be exploited to generate persuasive misinformation at scale (Pan et al., 2023). Nonetheless, we lack systematic, ecologically valid evidence on how individuals evaluate and respond to LLM-generated political content in realistic interactive settings.
This gap is particularly important because the mechanisms by which humans evaluate LLM-generated information likely differ from those studied in prior work on misinformation and persuasion. LLMs present novel epistemic challenges: they are perceived as authoritative, they generate content in response to individual queries, and they can engage in multi-turn dialogue. Understanding how individuals calibrate their trust in such systems, and how this translates into belief change, is an important scientific question with direct implications for the governance of AI-based information systems.
- Objectives and Scientific Roadmap
The central objective of this PhD is to develop an integrated empirical and computational framework for understanding how LLM-generated political content is produced, evaluated, and internalized. The research program is structured along three complementary axes.
Axis 1: Automated Evaluation of LLM-Generated Political Content
The first axis builds on prior work (Hackenburg et al., 2025) to develop tools to automatically evaluate the reliability of political content produced by LLMs, with the technical support of two engineers. LLM outputs will be collected under systematically varied prompts, ranging from neutral queries to persuasive or adversarial instructions. Factual claims will be extracted and assessed using a hybrid pipeline combining LLM-based verification, web retrieval, and human validation. Outputs will also be annotated for accuracy, political slant, verifiability, and other relevant traits. This produces a structured, continuously updatable database of LLM output features across models and topics, which serves as the empirical foundation for the behavioral studies in Axes 2 and 3.
Axis 2: Experimental Study of Credibility Judgement
The second axis uses controlled experiments to examine how individuals evaluate LLM outputs: what features of LLM-generated content make it more or less believable? Participants will be exposed to vignette-style content generated by LLMs, with systematic variation along dimensions identified in Axis 1: factual accuracy, verifiability, plausibility, and political congruence with the user’s prior beliefs. Participants’ judgments of truthfulness and credibility will be recorded and analyzed using regression and mixed-effects models. These experiments will provide some of the first systematic evidence on how individuals evaluate and respond to AI-generated information in realistic interactive settings—a question that remains largely unaddressed in the existing literature.
Axis 3: Interactive Experiments on Belief Updating
The third axis integrates the first two by placing participants in naturalistic multi-turn interactions with LLMs on political topics. This design prioritizes ecologically valid settings by employing standard, unprompted generalist systems; this context corresponds to ordinary real-world use and so enables linking between naturally occurring variation in outputs to real life belief updating and political belief trajectory. Participants’ opinions are measured before and after these interactions. The generated content is analyzed with the tools from Axis 1, providing estimates of its accuracy, verifiability, and political slant, and the credibility mechanisms identified in Axis 2 are used to interpret observed belief change. This axis supplies evidence on how LLMs influence users’ beliefs, and identifies the specific output features through which belief updating occurs.
- Timeline and Candidate Profile
Timeline – The first year will focus on acquiring the necessary background in automated fact-checking, developing the content evaluation pipeline in collaboration with two engineers (Axis 1), and piloting experimental materials. The second year will be dedicated to the controlled vignette experiments (Axis 2) and refinement of computational tools. The third year will concentrate on the interactive belief-updating experiments (Axis 3), integration of findings across axes, and writing of the dissertation.
Candidate profile – The project requires a strong background in cognitive science or experimental psychology, combined with a solid interest in AI systems. Candidates should have experience with behavioral experimentation and statistical analysis (R), and ideally familiarity with research on belief formation and credibility judgments. A strong interest in interdisciplinary research at the boundary of cognitive science and AI is essential.
Non-discrimination, openness, and transparency – All PR[AI]RIE-PSAI partners are committed to supporting and promoting equality, diversity, and inclusion within their communities. We encourage applications from a wide range of backgrounds, which we will ensure are selected through an open and transparent recruitment process.
Application Process
Candidates interested in this PhD position should submit the following materials:
- A detailed curriculum vitae, including academic background and relevant experience;
- A one-page motivation letter, describing the candidate’s interest in the proposed research topic, their scientific ambitions, and the relevance of their background to the project;
- Copies of academic transcripts and diplomas.
Applications should be sent directly to the supervisors at the following email addresses: hugo.mercier@gmail.com, sacha.altay@gmail.com.
The deadline for applications is May 1st, and candidates should note that the selection process will be conducted in two phases, with results communicated between the end of May and mid-June 2026.
References
Prama, T.T. & Islam, M.S. (2025). Evaluating Credibility and Political Bias in LLMs for News Outlets in Bangladesh. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 4, 665—677.
Massenon, R., Gambo, I., Khan, J.A., Agbonkhese, C., & Alwadain, A. (2025). “My AI is Lying to Me”: User-reported LLM hallucinations in AI mobile apps reviews. Nature: Scientific Reports, 15(30397).
Weidinger, L., Mellor, J., Rauth, M., Griffin, C., Uesato, J., Huang, P-S., Cheng, M., Glaese, B., Balle, B., Kasirzadeh, A., Kenton, Z., Brown, S., Hawkins, W., Stepleton, T., Biles, C., Birhane, A., Haas, J., Rimell, L., Hendricks, L.A., Isaac, W., …, Gabriel, I. (2021). Ethical and social risks of harm from Language models. PsyArXiv. https://doi.org/10.48550/arXiv.2112.04359
Potter, Y., Lai, S., Kim, J., Evans, J., & Song, D. (2024). Hidden Persuaders: LLMs’ Political Leaning and Their Influence on Voters. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 4244—4275.
Pan, Y., Pan, L., Chen, W., Nakov, P., Kan, M-Y., & Wang, W. (2023). On the Risk of Misinformation Pollution with Large Language Models. Findings of the Association for Computational Linguistics, 1389—1403.Hackeburg, K., Tappin, B.M., Hewitt, L., Saunders, E., Black, S., Lin, H., Fist, C., Margetts, H., Rand, D.G., & Summerfield, C. (2025). The levers of political persuasion with conversational artificial intelligence. Science, 390(6777).
Nicolas Baumard 