From Chalkboards to Chatbots: Evaluating the Impact of Generative AI on Learning Outcomes in Nigeria

Cost-Effectiveness

This section provides a cost-effectiveness analysis of the pilot, drawing comparisons with other high-dosage programs, and discusses some challenges and opportunities for scalability. We measure the program’s nominal cost using planning and budget data, and estimate its implicit costs on pro rata basis (Valdivia Teixeira, 2019). Implementing the 6-week pilot for 657 students, had a per-pupil cost of approximately $48, and the marginal cost is estimated at $9. Furthermore, extending the pilot over the four academic quarters would cost $124 per pupil, which can be particularly useful to frame policy discussions about the return on investment of a longer intervention even without further improvement in learning outcomes (a conservative approach considering our dosage effects results). Table 13 provides a breakdown of the pilot’s costs and our estimation for a four-quarter program. As the pilot was implemented, fixed costs accounted for 43% of overall costs (39% in a hypothetical four-quarter program), which provides an idea of the potential cost reduction in a second round of the program.

To analyze the effectiveness of the pilot, we follow the methodology used in Evans and

Yuan (2019). To estimate effectiveness, size effects are translated into EYOS, which expresses the years of ’business-as-usual’ schooling in terms of learning outcomes delivered by a given intervention. Our ITT effect of 0.238 standard deviation in English is equivalent to increasing 1.5 years of ’business-as-usual’ schooling in Nigeria, and the total score gain of 0.31 standard deviation is equivalent to 2 additional years of schooling in the country. This is more than twice the effect of some of the most effective interventions in education, such as structured pedagogy, which is typically implemented for the entire school year. For the remainder of this section, we use effects on English given that this is our main outcome of interest. The intervention’s cost-effectiveness, measured in EYOS per $100 (theoretically) invested per participant, is estimated to yield 3.2 EYOS under the assumption of constant returns.

We convert gains in test scores into increased wages (Evans and Yuan, 2019). We estimate that improvements in English would result in an increase of 14% in wages. The additional annual income ranges between $392 and $630. Over their work-life, each participant has a present discounted value of income gains of $7,767 to $12,517 When considering the long-term wage effects and the cost of our pilot, the benefit-cost ratio of our pilot program is 161 to 260. As a reference point, we calculate that running the pilot for one year without further improvements in learning outcomes would still yield a benefit-cost ratio as high as 62 to 100.

For comparison purposes, the return on investment of the program is very high compared to recent evidence from high-dosage personalized tutoring programs with and without technology in the United States. Guryan et al. (2023), Bhatt et al. (2024), Fryer and Howard-Noveck (2020) yield benefit-cost ratios that range from 2.4 to 8. However, evidence from low-income countries (LICs) and lower-middle-income countries (LMICs) shows benefit-cost ratios that vary between 8 and 156 (Glewwe et al. (2010), Duflo et al. (2011), Banerjee et al. (2007), Evans and Yuan (2019)). In line with our results, a recent review of 150 interventions in global education (Angrist et al., 2023) found that programs teaching at the right level with a technology component generate the largest benefit-cost

18 ratios across LICs and LMICs, with a weighted average of 65.

Like in any other lower-middle income country, the productivity of a ’business as usual’ day in a Nigerian school is lower than in a high-performing country. To facilitate cross-country comparisons, we calculate the LAYS (Angrist et al., 2025; Filmer et al., 2020), which adjusts the learning gains in years of schooling of our program by the quality of learning in Nigeria. Using the learning gain in English skills (0.24 σ), we estimate the LAYS under two scenarios. If the effects last only one year, the intervention produces 0.3 LAYS. Conversely, if the impacts last for the remaining school life expectancy, the program generates an additional 0.9 year of high-quality schooling for each participant. In other words, our Nigerian participants gained (on average) up to 0.9 years of a top-performing country education. Finally, our program yields between 0.6 and 1.9 LAYS per $100. If instead of the estimates based on the 6-week program we used the estimates calculated in the dosage effect section for English, the LAYS for one year of the program -accounting for the observed attendance ratewould be 1.25.

Beyond comparing small quantitative differences, it is important to assess results as ordinal rather than cardinal, given the varying underlying assumptions, imprecision in estimations, and contextual conditions of each study. Through this lens, analysis can inform broad trade-offs in policy making, budget allocation, and program design. Consequently, the pilot program’s benefit-cost ratios and other cost-effectiveness metrics are situated at the upper end or above benchmarks, highlighting its potential as a cost-effective solution for addressing learning crises in low-resource settings.

Future Research Directions

Several potential avenues for future research emerge from the findings of this study. First, extending the duration of the program beyond six weeks could provide insight into whether a longer intervention leads to more substantial or sustained improvements in learning outcomes and what the shape of the learning curve is as time progresses. A more extended program might also allow for more complex interactions with the chatbot, further enhancing its educational benefits. These extended studies should be complemented with a more qualitative assessment of the interaction between the students and the AI tool, in order to understand the causal mechanism driving the improvements in learning, and the specific ways that students benefit from the virtual tutoring.

Second, expanding the study to include a more diverse set of schools, particularly those in rural areas, would improve the external validity of the findings. By investigating the effectiveness of the program in various educational settings, it could be possible to assess its scalability and adaptability to different contexts. Both a longer duration of the intervention and a broader set of schools could show insights into what Rodriguez-Segura (2022) calls ”general equilibrium” effects after the roll-out of educational technology interventions, which could include potential changes in teacher attitudes, effort, and behavior, even as part of the regular instruction.

Another promising avenue for exploration is the addition of an additional treatment arm that offers one-on-one tutoring by teachers without the use of technology. This would allow for a direct comparison between the effectiveness of LLM-powered tutoring and traditional teacher-led tutoring, providing valuable insights into cost-effectiveness and pedagogical efficacy, and helping to calculate the productivity enhancement effect that technology might have on teachers. Similarly, additional arms can help disentangle the multiple causal mechanisms that might be driving the effect, including additional instructional time and the interaction with the chatbot with teacher support.

Understanding the long-term impacts of the intervention is also crucial. Future studies should investigate whether the positive effects observed in the short term persist over time, contributing to lasting improvements in students’ academic trajectories. Similarly, from a policy perspective, it would be valuable to assess whether an after-school program like this leads to a long-term shift in effort or time away from productive in-school activities, and whether an in-school program might offer even greater effectiveness as an alternative.

Finally, further research could explore whether students are transferring -without explicit instructiontheir skills in using AI tools from one subject area to another. For example, future studies could examine whether familiarity with AI tools in English language lessons enhances student performance in other subjects, such as mathematics or science. This cross-curricular application of AI would provide insights into the broader academic potential of LLMs in education.

Policy Implications

The findings of this pilot intervention highlight several promising policy implications for addressing the learning crisis in developing countries, particularly in Sub-Saharan Africa.

First, the intervention demonstrated substantial impacts on learning outcomes, despite some implementation challenges, such as internet disruptions and power outages. This is particularly encouraging for countries grappling with severe teacher shortages, high population growth, and increasing teacher attrition rates. One key takeaway for policymakers is that investing in AI-powered tutoring programs powered by LLMs, could significantly increase teacher productivity, which is consistent with recent qualitative evidence (Keppler et al., 2024). By supplementing traditional classroom instruction with AI-based support, education systems can deliver personalized learning experiences, particularly in contexts where human resources are stretched thin.

Second, the program shows that LLMs can improve learning when used properly. A recent debate in the literature seems to show that LLMs can harm learning when used as shortcuts, in other words, when used to facilitate responses to students’ questions without encouraging them to think. These are results found in studies such as Bastani et al. (2024). Some studies have also shown that the use of LLMs can lead to lower-quality reasoning and argumentation when students use them to search for information (Stadler et al., 2024). Contrarily, the intervention we assessed seems to suggest that LLMs can improve learning when used specifically as tutors adapted to specific use cases and contexts via prompting. Thus, the findings are consistent with the idea emphasized in Gerlich (2025) that educational strategies should promote critical engagement with AI technologies in order to avoid cognitive offloading, which might reduce critical thinking skills. The intervention evaluated in this paper leveraged third key mechanisms to achieve effective tutoring. First, the prompts were intentionally crafted to guide the LLM to provide explanations and support grounded in the principles of the science of learning, rather than simply supplying direct answers. Second, teachers played an essential role in monitoring and guiding students’ use of the LLM to ensure it was used appropriately and productively. Third, the content of each session was aligned with the official curriculum.

In other words, we interpret that the intervention as a whole -which includes the interaction with the LLM and teacher guidance with specific promptsis driving the results. We have reasons to believe that the effects are not driven solely by the additional time

21 with teachers, given that the impact of human tutoring tends to be very low when is not one-on-one or in small groups (Nickow et al., 2020; Kraft and Lovison, 2024). This interpretation suggests that there might be complementarities between teachers and technology and the way technology is used and deployed is critical to understand its impact. Following Muralidharan et al. (2019), our results can also be interpreted as showing the extent to which using technology -and in particular, LLMsin education can raise the productivity of an instructor.

Third, although this intervention was conducted on a pilot scale, its cost-effectiveness makes it a promising candidate for large-scale implementation. While the effect sizes of interventions typically decrease as sample sizes grow (Evans and Yuan, 2022), the rapid advancements in LLMs and the potential for improving the implementation process suggest that future iterations of the program could be even more impactful. Moreover, the fact that the intervention was implemented with local staff (including teachers and monitors), might help with its scalability. Similarly, the use of a free tool, rather than traditional subscription-based computer-adaptive software, can significantly lower marginal costs. Moreover, LLMs offer a distinct advantage for adaptive learning: they eliminate the need to develop extensive question banks with varying difficulty levels to accurately categorize students into performance tiers, a requirement that Rodriguez-Segura (2022) emphasizes is essential for traditional adaptive software. This potential for scalability is particularly important for policy makers seeking affordable, efficient ways to address learning gaps in resource-constrained environments.

Fourth, while AI interventions have the potential to reduce learning gaps, policy makers must be vigilant about areas where such programs could inadvertently widen inequities. Although the intervention may offer Pareto-optimal benefits, disparities in digital literacy and access to technology could exacerbate existing inequalities. A prerequisite to ensure that all students can benefit from AI-powered tutors, digital skills and AI literacy programs should be introduced early in the curriculum in a practical and inclusive manner and teachers should receive training to leverage digital skills to improve their pedagogical practice and support students to be digital and AI literate. Additionally, significant investments in infrastructure and equipment are needed to provide equitable access to technology across regions. Policy makers must ensure that efforts to integrate AI into

22 education are accompanied by initiatives that address the digital divide, particularly in low-income and rural areas. This may require a realignment of priorities across sectors, as education budgets are often heavily weighted toward recurrent expenditures such as salaries.

Finally, the rapid development of generative AI presents a unique opportunity to tackle the global learning crisis. By harnessing the responsible use of AI to provide personalized, adaptive learning at scale, governments can take decisive steps toward improving learning outcomes in contexts that have traditionally faced significant educational challenges.

Technical Report,” arXiv preprint arXiv:2303.08774.

A-U, D., C. A, M. C (2023): Can Digital Personalized

Learning for Mathematics Remediation Level the Playing Field in Higher Education? Experimental Evidence from Ecuador, The World Bank.

A, J. D. G. W. I (1995): “Two-Stage Least Squares Estimation of Average Causal Effects in Models with Variable Treatment Intensity,” Journal of the American Statistical Association, 90, 431–442.

R. N, B. W (2023): “Improving Learning in Low- and Lower-Middle-Income Countries,” Journal of Benefit-Cost Analysis, 14, 55–80.

(2025): “How to Improve Education Outcomes Most Efficiently? A Review of the Evidence Using a Unified Metric,” Journal of Development Economics, 172, 103382.

B, Y., D. M, L. Z, M. B, S. R (2016): “The Impact of Integrating ICT with Teaching: Evidence from a Randomized Controlled Trial in Rural Schools in China,” Computers & Education, 96, 1–14.

, M. W (2016): “Mainstreaming an Effective Intervention: Evidence from Randomized Evaluations of “Teaching at the Right Level” in India,” Tech. rep., National Bureau of Economic Research.

falls of Participatory Programs: Evidence from a Randomized Evaluation in Education in India,” Working Paper 14311, National Bureau of Economic Research.

B, A. V., S. C, E. D, L. L (2007): “Remedying Education:

Evidence from Two Randomized Experiments in India,” The Quarterly Journal of Economics, 122, 1235–1264.

“Generative AI Can Harm Learning,” Available at SSRN, 4895486.

“Can Technology Facilitate Scale? Evidence from a Randomized Evaluation of High Dosage Tutoring,” Working Paper 32510, National Bureau of Economic Research.

B, R. A. (1994): “Memory and metamemory considerations in the training of human beings,”.

B, B. S. (1984): “The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring,” Educational Researcher, 13, 4–16.

B, M. D. MK (2009): “In pursuit of balance: Randomization in practice in development field experiments,” American economic journal: applied economics, 1, 200– 232.

Relative Effectiveness of Teachers and Learning Software: Evidence from a Field Experiment in El Salvador,” Journal of Labor Economics, 40, 737–777.

D, A. S. (2024): “Chapter 4 - KoboToolbox,” in Open Electronic Data Capture Tools for

Medical and Biomedical Research and Medical Allied Professionals, ed. by A. Pundhir, A. K. Mehto, and A. Jaiswal, Academic Press, 241–329.

D, R., M. J, X. Y, Y. L, S. L (2024): “Does ChatGPT Enhance Student

Learning? A Systematic Review and Meta-Analysis of Experimental Studies,” Computers & Education, 105224.

D, E., P. D, M. K (2011): “Peer Effects, Teacher Incentives, and the Impact of Tracking: Evidence from a Randomized Evaluation in Kenya,” American Economic Review, 101, 1739–1774.

(2013): “Improving students’ learning with effective learning techniques: Promising directions from cognitive and educational psychology,” Psychological Science in the Public interest, 14, 4–58.

E, D. F. Y (2019): “Equivalent Years of Schooling: A Metric to Communicate Learning Gains in Concrete Terms,” World Bank Policy Research Working Paper.

E, D. K. A. M A (forthcoming): “Teacher Demand in African

Countries: Trends and Challenges,” CGD Working Paper, Center for Global Development.

E, D. K. F. Y (2022): “How Big Are Effect Sizes in International Education

Studies?” Educational Evaluation and Policy Analysis, 44, 532–540.

F, R. C., R. I, M. P. T (2015): “The Next Generation of the

Penn World Table,” American Economic Review, 105, 3150–3182.

F, D., H. R, N. A, S. S (2020): “Learning-Adjusted

Years of Schooling (LAYS): Defining a New Macro Measure of Education,” Economics of Education Review, 77, 101971.

F, R. G. M. H-N (2020): “High-Dosage Tutoring and Reading

Achievement: Evidence from New York City,” Journal of Labor Economics, 38, 421–452.

G, M. (2025): “AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking,” Societies, 15, 6.

G, P., N. I, M. K (2010): “Teacher Incentives,” American Economic

Journal: Applied Economics, 2, 205–227.

G, J., J. L, M. P. B, P. J. C, J. M. V. D, K. D, G. F, (2023): “Not Too Late: Improving Academic Outcomes among Adolescents,” American Economic Review, 113, 738–765.

fective and Scalable Math Support: Experimental Evidence on the Impact of an AIMath Tutor in Ghana,” in International Conference on Artificial Intelligence in Education, Springer, 373–381.

I, H., K. K, H. N, M. N (2021): “Does Computer-Aided

Instruction Improve Children’s Cognitive and Noncognitive Skills?” Asian Development Review, 38, 98–118.

K, S. H. (2016): “Spaced repetition promotes efficient and effective learning: Policy implications for instruction,” Policy Insights from the Behavioral and Brain Sciences, 3, 12– 19.

K, S., W. P. S, C. S (2024): “Backwards Planning with Generative AI: Case Study Evidence from US K12 Teachers,” Available at SSRN.

K, G., K. M, A. K, T. M, G. P (2024): “AI Tutoring

Outperforms Active Learning,” Research Square.

K, M. A. V. S. L (2024): “The Effect of Student-Tutor Ratios: Experimental Evidence from a Pilot Online Math Tutoring Program. EdWorkingPaper No. 24-976,” Annenberg Institute for School Reform at Brown University.

Education with Large Language Models: Peril or Promise?” Available at SSRN.

L, F., R. L, L. Z, X. H, S. R (2015a): “Does Computer-Assisted Learning Improve Learning Outcomes? Evidence from a Randomized Experiment in Migrant Schools in Beijing,” Economics of Education Review, 47, 34–48.

“Teaching the Language of Wider Communication, Minority Students, and Overall Educational Performance: Evidence from a Randomized Experiment in Qinghai Province, China,” Economic Development and Cultural Change, 63, 753–776.

L, D. S. (2005): “Training, Wages, and Sample Selection: Estimating Sharp Bounds on

Treatment Effects,”.

L, M., P. B. C, F. J. S (2024): “AI Meets the Classroom:

When Does ChatGPT Harm Learning?” arXiv preprint arXiv:2409.09047.

M, Y., R. F, P. L, S. R (2024): “Isolating the “Tech” from

Edtech: Experimental Evidence on Computer-Assisted Learning in China,” Economic Development and Cultural Change, 72, 1923–1962.

MD (2014): “Both multiple-choice and short-answer quizzes enhance later exam performance in middle and high school classes.” Journal of Experimental Psychology: Applied, 20, 3.

M, D., L. Z, R. L, Q. Q, W. H, J. W, Y. Q, M. B, S. R (2014): “Integrating computer-assisted learning into a regular curriculum: Evidence from a randomised experiment in rural schools in Shaanxi,” Journal of Development Effectiveness, 6, 300–323.

M, E. L. M (2023a): “Assigning AI: Seven Approaches for Students, with Prompts,” arXiv preprint arXiv:2306.10052.

M, E. R. L. M (2023b): “Using AI to Implement Effective Teaching

Strategies in Classrooms: Five Strategies, Including Prompts,” The Wharton School Research Paper.

M, K. (2017): “Field experiments in education in developing countries,”

in Handbook of Economic Field Experiments, Elsevier, vol. 2, 323–385.

M, K., A. S, A. J. G (2019): “Disrupting Education?

Experimental Evidence on Technology-Aided Instruction in India,” American Economic Review, 109, 1426–1460.

N, A., P. O, V. Q (2020): “The Impressive Effects of Tutoring on PreK-12 Learning: A Systematic Review and Meta-Analysis of the Experimental Evidence,” Working paper, National Bureau of Economic Research.

P, M. D. A (2019): “The Impact of a Mathematics Computer-Assisted

Learning Platform on Students’ Mathematics Test Scores,” MERIT Working Papers 2019-007, United Nations University - Maastricht Economic and Social Research Institute on Innovation and Technology (MERIT).

R-S, D. (2022): “EdTech in Developing Countries: A Review of the Evidence,” The World Bank Research Observer, 37, 171–203.

S, M., M. B, M. S (2024): “Cognitive Ease at a Cost: LLMs

Reduce Mental Effort but Compromise Depth in Student Scientific Inquiry,” Computers in Human Behavior, 160, 108386.

V T, S. (2019): “Capturing Cost Data,” https://thedocs.worldbank.

V L, W. J. R. K. H (2015): Handbook of Item Response Theory, CRC Press.

V, A., S. P. C, M. S (2024): “GPT-4 as a Homework

Tutor Can Improve Student Engagement and Learning Outcomes,” arXiv preprint arXiv:2409.15981.

V, S. L. K (2006): “Constructing socio-economic status indices:

how to use principal components analysis,” Health Policy and Planning, 21, 459–468.

tor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise,” ArXiv preprint arXiv:2410.03017.

W, J., D. Gˇ ´, P. A. K (2025): “ChatGPT in education: An effect in search of a cause,”.

W, Y., C. R. M, M. A. S (2018): “Teaching the science of learning,” Cognitive research: principles and implications, 3, 1–17.

W B (2022): “The State of Global Learning Poverty Report: 2022 Update,” Tech.

rep., World Bank, accessed: 2024-09-20.

Total Score English Skills Digital Skills AI Skills (Weighted) Score Score Score

Not eligible for Lottery

First Term Exam Second Term Exam (1) (2)