From Chalkboards to Chatbots: Evaluating the Impact of Generative AI on Learning Outcomes in Nigeria

Model and Main Results

We estimate the intent-to-treat (ITT) effects of the intervention using the following regression:

The dependent variable Y represents student i’s test score in school k and subject s. The specification includes a coefficient γ for the student’s pre-treatment exam score from the second term (Baseline), and a term β for our primary variable of interest—an indicator for whether the student won the treatment lottery. To account for unobserved school-level heterogeneity, we include school fixed effects (ϕ).

Although the difference in academic performance before the intervention between the treatment and the control group was not statistically significant, it was still positive in favor of the treatment group. Therefore, we followed a conservative approach to ensure the robustness of our results, including the performance in the second term exam as a control variable. This approach is also recommended by Muralidharan (2017).

Table 2 reports the intent-to-treat (ITT) effects of the intervention on three main outcomes: the total score of the final assessment (weighted and IRT scaled) and the score from the third term exam. The coefficient for the second-term exam is significant across all models, reflecting the predictive validity of prior performance. School-fixed effects are included in all models, and the number of observations ranges between 636 and 654, depending on the outcome.

The treatment effect on the total score (weighted) is 0.31 standard deviation (SE = 0.068), and the effect remains positive and significant at 0.263 standard deviation (SE = 0.068) when scaled using Item Response Theory (IRT). These results indicate that the intervention led to substantial improvements in students’ performance on the assessment directly tied to the program. Importantly, treatment also had a positive and significant impact on the third term exam score, with an effect size of 0.206 standard deviation (SE = 0.067), although this exam was not limited to the intervention’s specific content. This suggests that the intervention may have fostered generalizable skills or improved learning outcomes beyond the targeted content.

Table 3 provides a more granular analysis by disaggregating the total score into English skills, digital skills, and AI skills. The results reveal that the intervention had the largest effect on AI knowledge, with a coefficient of 0.309 standard deviations (SE = 0.077), followed by English skills (0.238 σ, SE = 0.068) and digital skills (0.139 σ, SE = 0.076). Effects on English skills (our main outcome of interest) and on AI skills are statistically significant at the 1% level. The effects on digital skills are statistically significant at the 10% level. The positive and significant effects on AI and digital skills further indicate potential spillovers to other skill areas, even though these were not the primary targets of the program. As with Table 2, the second term exam scores are strong predictors of outcomes, and school fixed effects are included to account for unobserved factors at the school level.

The results of this pilot study are particularly notable given that several factors likely attenuated the estimated treatment effects. First, the randomization was conducted at the student level rather than at the school level, a design feature that may have led to spillover effects, as students in the control group may have interacted with peers in the treatment group during regular school hours, potentially diffusing the impact of the intervention. Second, monitoring and evaluation data indicate that some control group students inadvertently gained access to the after-school sessions, due to the lack of willingness of some teachers to enforce the distinction, especially during the first weeks. Additionally, significant implementation challenges occurred during the first weeks of the program, with many students encountering difficulties creating accounts and engaging with the LLM. Despite these challenges, the intervention yielded positive and significant results, suggesting that the observed effects should be interpreted as conservative estimates of the intervention’s impact.

A recent review of randomized controlled trials in pre-primary, primary, and secondary education in low and middle-income countries found a median effect of 0.10 standard deviations in overall test scores and 0.14 in reading (Evans and Yuan, 2022). Thus, the results found in this study are situated at least at the 80th percentile of all RCTs, even though effect sizes in secondary education tend to be lower than in primary. Even when considering only RCTs that had between 500 and 1000 participants, the results are still higher than 80% of the other studies. When considering only the effects on language outcomes, the results are near the 70th percentile of all studies.

Heterogeneity

As a first step, we conducted quantile regressions to examine the treatment effect at different points in the outcome distribution. The analysis indicates that the treatment has a positive and statistically significant effect across all quantiles, suggesting broad benefits for students regardless of their initial performance levels. Table 4 examines the heterogeneity in treatment effects by gender, SES, and baseline academic performance. Column (1) explores heterogeneity by gender, using an interaction term between the treatment indicator and a female dummy variable. While the main effect of treatment is not statistically significant for this specification, the interaction term for treatment and being female is positive and significant (0.420) at the 5% level, indicating that the intervention had a larger positive effect on female students compared to their male counterparts. This result should be interpreted with caution, as it appears to be influenced by the inclusion of a girls-only school that performed worse than others in the sample prior to the intervention.

Column (2) considers heterogeneity by baseline academic performance, as measured by the second-term exam score. The interaction term between treatment and the second-term exam score is positive and significant at the 5% level, indicating that students with higher prior academic performance benefited more from the intervention. Column (3) examines heterogeneity by socio-economic status, using an interaction term between treatment and the SES index. The interaction term is positive (0.113) and significant, also at the 5% level, suggesting that students from higher SES backgrounds experienced larger treatment effects. This finding aligns with anecdotal evidence indicating that, for students from poorer households, this was often one of their first experiences using computers. While these students still experienced significant gains relative to the counterfactual of not participating, the initial unfamiliarity with technology may have moderated the magnitude of the intervention’s impact.

Dosage Effects

All the results presented thus far are ITT estimates, which are based on an average attendance rate of approximately 72% among participants in the treatment group. In this section, we present LATE and OLS estimates, which measure the impact of actually attending the sessions. These estimates leverage attendance data collected as part of the program’s monitoring and evaluation efforts (see Figure 5). Additionally, under further assumptions, we provide predicted treatment effects at varying levels of program expo-13 sure. We estimate the dose-response relationship between days of attendance and value-added using the following model:

The dependent variable Y represents student i’s test score in subject s in school k. The specification includes a coefficient γ for the student’s second-term pre-treatment exam score (Baseline), and a term µ for our primary variable of interest—the number of days the student attended intervention sessions, which takes a value of zero for the control group (Attendance).

Table 5 shows that higher attendance is strongly associated with improved learning outcomes, providing evidence of a dose-response effect, with an estimated effect size of approximately d=0.033 for each additional day of attendance. This finding underscores the importance of consistent participation, as greater exposure to the program results in meaningful improvements in student outcomes.

These estimates capture the average causal response (ACR) of the treatment, which represents a weighted average of the causal effects of a one-unit change in treatment (in this case, an additional day of attendance) for individuals whose treatment status is influenced by the instrument (Angrist and Imbens, 1995). However, using these Instrumental Variable (IV) estimates to predict the effects of different levels of attendance requires additional assumptions, as stated in Muralidharan et al. (2019): (i) assumptions about how treatment effects vary across students, as the ACR is identified only for a subset of compliers (those who attend at least one session) and not the entire sample, and (ii) assumptions about the functional form of the relationship between days attended and treatment effects, since the ACR reflects an average across varying levels of treatment intensity.

For the first assumption, we cannot assume that the effect on the non-compliers would have been the same as the compliers. This is because we have evidence showing that low performing students tend to benefit less from the program (Table 4), and that better performance in previous examinations correlates positively with attendance (Table 9). Therefore, we follow a conservative approach and assume that the non-compliers -those who are assigned to the treatment group but do not attend any sessionswould not benefit at all had they attended the program. Under that assumption, and given that the non-compliers represent only 3.5% of those assigned to the treatment group, the estimated effect size of each additional day of attendance is of 0.031.

We follow this conservative approach despite the fact that, following Muralidharan et al.

14 (2019), two other pieces of evidence would be consistent with expecting the ACR to apply even to the non-compliers. First, we cannot reject the equality of the IV estimates of equation (3) and the ordinary least squares (OLS) estimates using a value-added (VA) specification, which suggests that the average treatment effects and local average treatment effects (ATE and LATE) may be similar. Second, the constant term in the OLS VA specifications (corresponding to 0 attendance) is similar when estimated using the full sample and when estimated using only the data in the treatment group. This suggests equality of potential outcomes across students with different compliance rates.

Regarding the functional form of the relationship between days attended and the treatment effect, a graphic representation suggests a linear relationship (Figure 4). Moreover, while the value added of the scores is strongly correlated with the number of days attended in a linear specification, adding a quadratic term does not improve the fit, and the quadratic term is not significant, as shown on Table 7. From a more theoretical point of view, the linear effect is also expected given the adaptive nature of LLMs. Thus, it seems fair to assume that, while effect is not the same across students, the effect for each student does not show diminishing returns to program exposure.

Qualitative evidence from the intervention aligns with these results, suggesting that the first days of the program had little to no measurable impact on learning outcomes. This lag may reflect the time required for students to familiarize themselves with the technology and adjust to the new instructional format. Beyond this initial acclimation period, the effects of additional days of attendance remained consistently positive, with no evidence of plateauing. This suggests that extending the implementation period beyond the six weeks of the intervention could have further amplified learning gains. It also suggests that the LATE estimates might be underestimated, given the null effects of the first few days of participation.

Under the assumptions of constant treatment effects diminished by the lack of effect on non-compliers and a linear dose-response relationship—both of which appear reasonable in this context— and following the methodology used in (Muralidharan et al., 2019), our instrumental variables analysis predicts that attending the program for 36 weeks (the equivalent of a school year) would lead to gains of approximately 2.23 standard deviations. A more conservative estimate based on an attendance rate of 72% -the empirical value in our sample-, predicts that attending the program for 21 weeks (over a theoretical total of 36) would lead to gains in the range of 1.55 standard deviations. Under a more pessimistic scenario of only 50% attendance rates, the estimate would still be 1.2 standard deviations. These findings highlight the potential of sustained implementation to yield

15 transformative impacts on learning outcomes.

Robustness

To test the robustness of the results, we performed several checks, progressively adjusting the baseline model to improve robustness and account for potential sources of bias. First, for all the models presented, we employed robust standard errors to address heteroskedasticity in the data. Second, we included school-fixed effects in all the models presented, with robust standard errors, to control for unobservable school-level characteristics that might influence student outcomes. Third, we incorporated students’ performance in the second-term exam, conducted prior to the intervention, as a control variable. Although the difference in performance between the treatment and control groups was not statistically significant, the treatment students exhibited slightly higher scores. As mentioned above, adding this variable allows us to adopt a conservative approach, mitigating the risk of any potential, albeit unlikely, selection bias. The models with all these specifications are the basis for our main results, presented in Table 2.

Furthermore, we estimated the models using alternative specifications of the dependent variable to assess the robustness of our findings. For each outcome of interest, total score, English language proficiency, digital skills, and AI knowledge, we analyzed specifications with and without the experts’ weighted difficulty of questions. Additionally, we employed IRT as an alternative approach to scoring the assessments. This approach ensures that the estimated outcomes are not unduly influenced by the design or composition of the test. We presented the main results for the total score weighted (Column (1) of Table 2), which are 0.31 standard deviation, and the IRT scaled results (Column (2) of Table 2), of 0.263 standard deviation.

Table 8 presents a sensitivity analysis to test the robustness of the treatment effects by iteratively excluding one school at a time from the sample. The dependent variable shown is the total score (weighted) from the final learning assessment, but similar results are found with other specifications. The estimated treatment effects range from 0.156 (SE = 0.085) when excluding Idia College to 0.360 (SE = 0.072) when excluding Imaguero College. While the effect remains statistically significant at the 1% level in most cases, the exclusion of Idia College reduces the effect size and significance to the 10% level. This is, however, understandable, given the large size of Idia College (219 of 657, or 33% of the sample), whose exclusion reduces the power of the calculation. For all other schools, the estimated coefficients are stable around 0.30 and maintaining statistical significance at

16 the 1% level. Overall, the results demonstrate that the treatment effects remain consistent and statistically significant across most specifications, indicating that the findings are not driven by the influence of any single school.

Finally, since the difference in attrition between the treatment and control groups is significant, we first provide Lee-bounds estimates of ITT effects on the outcome variables. These bounds indicate the range of the ITT effects estimated using Lee (2005) bounding approach. This method accounts for possible bias due to attrition or selection into the program. The analysis shows that the treatment effects are always positive and significant, even following this conservative approach (Table 10). In addition, we assess the robustness of our findings to attrition by modeling the likelihood of participation in the endline based on observed characteristics. We then calculate inverse probability weighted treatment effects, finding that the estimated ITT effects remain largely unchanged (Table 11). Thus, our main conclusions hold even in the presence of non-random attrition at endline.