From Chalkboards to Chatbots: Evaluating the Impact of Generative AI on Learning Outcomes in Nigeria

The global education sector is grappling with a learning crisis. According to the Learning Poverty Index, approximately 70% of 10-year-olds in low- and middle-income countries cannot read and understand an age-appropriate text (World Bank, 2022). These deficits in learning accumulate and become particularly acute at the secondary school level, as evidenced by numerous international, regional, and national assessments.

In his seminal 1984 study, Bloom demonstrated that students receiving one-on-one tutoring outperformed their peers in traditional classroom settings by an average of two standard deviations (Bloom, 1984). Subsequent studies have consistently confirmed the significant benefits of one-on-one tutoring (Nickow et al., 2020). The challenge, however, is that implementing one-on-one tutoring at scale is costly and unaffordable for most education systems. Bloom referred to this challenge as the “two-sigma problem”: how to replicate the gains of personalized tutoring at scale in a cost-effective manner.

This paper examines whether generative artificial intelligence, specifically large language models (LLMs), can help solve that problem. We evaluate a six-week after-school tutoring program in Nigeria that used a publicly available LLM (ChatGPT-4) to support students in learning English. First-year secondary students from nine public schools in Benin City were invited to participate; from this pool, 52% of eligible students expressed interest, and participants were randomly selected from among them. Those assigned to the intervention attended twelve 90-minute sessions in computer labs, engaging in curriculumaligned activities guided by teachers. We use a randomized controlled trial (RCT) design to estimate the causal impact of the program on learning outcomes.

We present three main sets of results. First, we show that students selected to participate in the program score 0.31 standard deviation higher in the final assessment that was delivered at the end of the intervention. We find strong statistically-significant intent-to-treat (ITT) effects on all sections of that assessment: English skills (which included the majority of questions, 0.24 σ), digital skills (0.14 σ), AI skills (0.31 σ) and an Item Response Theory (IRT) composite score of each student’s exam (0.26 σ). We also show that the intervention yielded strong positive results on the regular English curricular exam of the third term. This result is important because the content evaluated in that exam was broader than the one covered during the six weeks of the intervention and included the content of the entire year. We calculate an ITT effect of being selected for the program on the performance in the third-term exam of 0.21 standard deviations.

Second, we examine heterogeneity of the effects by certain pre-treatment characteristics.

Treatment effects were positive and statistically significant across all levels of baseline performance, but stronger among students with better prior performance. Similarly, treatment effects were positive and statistically significant over the entire distribution of a proxy for socioeconomic status, but stronger among students with a higher one. Lastly, treatment effects were stronger among female students, compensating for a deficit in their baseline performance.

Third, we conduct dose-response analysis. We estimate Local Average Treatment Effect (LATE) estimates, focusing on the impact of actual attendance to the intervention sessions, which averaged 72% among the treatment group. Using attendance data, we estimate a dose-response relationship, finding a strong linear association between days attended and improved learning outcomes, with an effect size of approximately 0.031 standard deviation per additional day of attendance. Further analysis predicts substantial gains with extended program duration, estimating an increase of between 1.2 and 2.2 standard deviations for a full academic year of participation, depending on attendance rates.

The findings, combined with a cost analysis, seem to indicate that the program was highly cost-effective. The six-week pilot generated learning gains that take between 1.5 and 2 years in a business-as-usual scenario. The program achieved 3.2 equivalent years of schooling (EYOS) per $100 invested, surpassing many comparable interventions. Using Learning-adjusted years of schooling (LAYS) as the metric for the analysis, the program generates up to 0.9 year of high-performance education. When benchmarked against evidence from both low- and middle-income countries, the pilot program ranks among the most cost-effective solutions for addressing learning crises.

Our study contributes to different strands of the literature that aim to identify the effect of programs that try to customize instruction to the level of students, both with and without technology. Efforts to address this challenge have included the development of the ”Teaching at the Right Level” (TaRL) approach, which has shown to improve learning outcomes in contexts such as India, Kenya, Ghana, and Zambia (Banerjee et al., 2016). Implementation modalities of TaRL have varied, ranging from pulling students out of class (Banerjee et al., 2007), tracking classrooms (Duflo et al., 2011), providing extra instructional time outside of school (Banerjee et al., 2016), and employing volunteers instead of teachers (Banerjee et al., 2008). However, scaling TaRL programs remains difficult due to their labor-intensive nature. This challenge is particularly pronounced given the global shortage of teachers, which is particularly pronounced in Sub-Saharan Africa. Recent estimates suggest that by 2040, countries in the region will need 21% more secondary school teachers per year (Evans and Mendez Acosta, forthcoming). Teacher shortages are

3 further compounded by high attrition rates, and the need for specialized knowledge at the secondary level makes TaRL programs even more difficult to implement.

In recent years, adaptive learning software has emerged as a potential solution to the scalability of tutoring programs by using technology to mimic one-on-one tutoring. Evidence suggests that computer-adaptive learning systems can improve learning outcomes. For example, a study of personalized, technology-aided after-school instruction for middle school students in India reported gains of 0.37 standard deviation in math and 0.23 standard deviation in Hindi over a 4.5-month period (Muralidharan et al., 2019). A study in Cambodia that focused on math instruction for primary school students found impacts on cognitive skills due to students’ increased learning productivity per hour (Ito et al., 2021). In El Salvador, the use of software for adaptive learning proved effective in an environment with heterogeneous classes and poorly qualified teachers (B¨uchel et al., 2022). Experiments in China have also found positive effects on standardized math scores (Lai et al., 2015a) and on Mandarin (Lai et al., 2015b), including when implemented during regular school hours (Mo et al., 2014). In Ecuador, the possibility to use an adaptive-learning software for 4 months led to large positive impact on standardized test scores in math (Angel-Urdinola et al., 2023). Other studies that do not follow experimental approaches have also estimated positive effects of similar software programs, such as a program in Uruguay that showed gains of 0.2 standard deviation on mathematics test scores (Perera and Aboal, 2019).

Despite these successes, adaptive learning programs face several challenges. First, most are not deployed in the world’s most challenging educational contexts, particularly in Sub-Saharan Africa, raising questions about external validity. Second, these programs often rely on proprietary software, which typically involves both fixed and per-student costs, making them difficult to scale in resource-constrained environments.

Some adaptive-learning options are developed using artificial intelligence (AI) to adjust to the level of the students, but they primarily rely on pattern recognition and predictive algorithms, to provide students with exercises adjusted to their level based on a pool of thousands of items. The recent advances in generative artificial intelligence offer a promising avenue to use software to teach students while maintaining a more humanlike interaction with students through the use of natural language.

Most of the studies that have examined generative AI in education have been conducted in developed countries and lab settings, assessing the short-term effects of brief interactions (Kumar et al., 2023). In Italy, studies have found positive effects of Large Language Models (LLMs) on learning outcomes through homework support (Vanzo et al., 2024). In

4 the United States, a human-AI approach with expert guidance through language models supports tutors instead of providing direct help to students, and found that students working on mathematics with tutors randomly assigned to have access to a tutor co-pilot are 4 percentage points more likely to master topics (Wang et al., 2024). A study carried out among undergraduate students at Harvard University showed that those who benefited from an AI-powered tutor at home performed better than those exposed only to active learning classes (Kestin et al., 2024).

Only a few studies evaluate the effect of generative AI to support students through tutoring. In Ghana, students who were given access to a phone for one hour a week and were allowed to use an AI-powered math tutor via a messaging app to independently study math improved their scores much more than those without access, with an effect size of 0.36 (Henkel et al., 2024). A recent study in T¨urkiye of an intervention that included only four sessions showed that while LLMs can improve mathematics learning outcomes, they can also be detrimental to learning in the long term if they are used as ”crutches” rather than as tutors (Bastani et al., 2024). A similar effect was found for coding classes in a lab setting (Lehmann et al., 2024). This study showed more positive impacts with the LLM used with prompts to safeguard learning (Bastani et al., 2024).

Thus, this paper contributes to this recent literature by examining the impact of one of the first programs to leverage LLMs for educational purposes in a developing country context using a real experimental design in Sub-Saharan Africa. It also aims to address some of the challenges identified in recent reviews of emerging studies on the effect of LLMs on learning: the lack of objective measures to complement subjective assessments of impact, weaknesses in the definition of the control and treatment groups (Weidlich et al., 2025), and the lack of power analysis to determine adequate sample sizes (Deng et al., 2024). Furthermore, the intervention used a free, off-the-shelf model, requiring minimal customization and no pre-built question banks, which might facilitate its scalability.

The findings of this intervention underscore several critical policy implications for addressing the learning crisis in developing countries, particularly in Sub-Saharan Africa. The program demonstrated significant impacts on learning outcomes, even amid challenges such as internet disruptions and power outages, highlighting its potential in contexts with severe teacher shortages and resource constraints. AI-powered tutoring programs using LLMs can complement traditional teaching by enhancing teacher productivity and delivering personalized learning experiences, particularly when paired with guided prompts, teacher oversight, and alignment with the curriculum. The intervention’s cost-effectiveness and scalability are promising, leveraging local staffand free tools

5 to minimize costs while eliminating the need for extensive question banks required by traditional adaptive software. However, policymakers must address potential inequities arising from disparities in digital literacy and access to technology. Investments in infrastructure, teacher training, and inclusive digital education are essential to ensure equitable access and mitigate the risk of exacerbating inequalities. Given the nascent application of LLMs in education, numerous questions remain unanswered, underscoring the importance of replicating this study, including with small variations.

The rest of this paper is organized as follows. Section 2 describes the intervention and the experimental design, including the data used. Section 3 presents our main results, including a heterogeneity and dosage analysis, as well as a robustness analysis. Section 4 discusses cost effectiveness, proposes future research directions, and presents policy implications.