OECD Digital Education Outlook 2026: Exploring Effective Uses of Generative AI in Education

GenAI holds the promise to generate assessment items. Although GenAI can produce a wide range of content, its use in standardised assessment requires generating items that meet psychometric standards of validity and reliability (Kaldaras, Akaeze and Reckase, 2024). Emerging evidence suggests this is feasible (von Davier, 2026). For example, Bhandari et al. (2024) showed that ChatGPT can generate psychometrically sound items for Algebra, while Attali et al. (2022) demonstrated similar success for reading tasks. The work by Attali et al. (2022) underpins the automated item generation process used in the Duolingo English Test that is a widely recognised language proficiency exam. GenAI can be used to evaluate the quality of assessment items. Work at Duolingo emphasised the importance of the human- in- the-loop to perform item quality review and sensitive review as part of quality assurance and before checking for psychometric properties of the generated items (Hao et al., 2024). This is also aligned with the recommendations by Moore et al. (2024) combining human judgements with LLMs to produce high quality multiple choice questions and short answer questions.

There is growing evidence of the potential of the use of GenAI in existing assessment practices. Existing research shows that the use of GenAI can be particularly effective when fine-tuned LLMs are used for automatic scoring of open-ended responses, demonstrating accuracy comparable or superior to models based on conventional machine and deep learning approaches. Latif and Zhai (2024), for instance, showed that a fine-tuned version of GPT-3.5 significantly outperformed BERT in scoring multi-label and multi-class science education tasks, achieving up to a 10.6% accuracy improvement. Similarly, GPT-4 has shown strong alignment (Quadratic Weighted Kappa (QWK) over 0.8) with contemporary writing evaluation tools in high-stakes language assessment contexts for L2 English learners, especially when provided with a single calibration example for each rating category (Yancey et al., 2023). However, Mansour et al. (2024) showed that conventional approaches dramatically overperformed ChatGPT-3.5 Turbo and Llama2 (average QWK of 0.817 vs 0.313 and 0.201) on automatic essay scoring of English essays from the Automated Student Assessment Prize dataset, which contains essays written in English by U.S. secondary school students in grades 7–10 for whom English is the first language on persuasive, source-dependent, and narrative writing tasks. The results indicate that although LLMs can potentially be useful for some types of automated scoring tasks, they may not be for others. It is therefore important to extend the existing body of knowledge to understand the types of tasks LLMs can be effective for to inform educational practice and policy. Likewise, educators need to be careful in their choices of relying on GenAI for automatic scoring.

Several studies have examined the extent to which GenAI can automatically assess responses to open-ended questions in standardised assessments and identify effective prompting strategies. For example, Rodrigues et al. (2024) evaluated GPT-4 across 738 open-ended questions drawn from high school Biology, Earth Science, and Physics tasks categorised by Bloom’s taxonomy (BLOOM, 1984). The model produced high-quality responses overall, though its performance declined on questions requiring factual recall or creative reasoning. Chan et al. (2025) analysed LLMs in standardised STEM assessments and showed that chain-of-thought prompting significantly improved accuracy, particularly for reasoning-intensive problems. In higher education, Moore et al. (2022) explored GPT-3’s ability to evaluate student-generated short-answer chemistry questions in online college courses and found only modest alignment (32-40%) with expert judgments. Together, these studies show that while GenAI can complement human grading in structured educational contexts, its reliability still varies by domain, cognitive demand, and prompt design, highlighting the continued need for human oversight in both item generation and scoring.