From Chalkboards to Chatbots: Evaluating the Impact of Generative AI on Learning Outcomes in Nigeria

The Intervention

The study analyzes the effects of an after-school program in which students interacted with a large language model twice per week to improve their English skills, following the national curriculum. The intervention was implemented in Benin City, Nigeria, using Copilot, an LLM powered by the GPT-4 model at the time of implementation. The program was implemented over a six-week period between June and July 2024, targeting first-year senior secondary school students, who are typically 15 years old. The intervention aimed to improve learning outcomes in English language classes using an AI chatbot as a virtual tutor. The selected tool was Microsoft Copilot, powered by ChatGPT-4, which was freely available and required only student registration. The program was conducted in nine schools and the students were grouped according to the number of computers in each school lab, with an average of 30 students per session. Each student was allowed to participate in a maximum of two 1.5-hour after-school sessions per week.

The selection of schools was based on the availability of computer labs. These labs varied in the types of devices they used, ranging from laptops to desktop computers. Internet access, essential for real-time interaction with the LLM, was provided through routers

6 and mobile telephone signals. However, internet disruptions and power outages were common challenges faced during the intervention. Despite these issues, students were able to interact with the chatbot for the majority of the sessions.

All students’ guardians signed consent forms, agreeing to their children’s participation in the pilot program. Students worked in pairs, sharing a computer, and engaged in dialogue with the AI tool to enhance their learning. Teachers, who played a critical role in guiding the students but did not provide direct instruction, participated in a single three-day training program with one cohort. This training introduced teachers to the functionalities of the LLM and equipped them with pedagogical techniques to ensure their responsible use and supervise students during the sessions. It also made them aware of potential risks, such as hallucinations and biases, that the LLM could have.

In the first session, teachers familiarized students with Microsoft Copilot, emphasizing both its educational benefits and potential risks, such as over-reliance on the model and the possibility of hallucinations and biased outputs. The goal was to foster responsible usage, encouraging students to complement their learning with the AI tool while retaining critical thinking skills.

Each subsequent session focused on a topic from the first-year English language curriculum, aligned with the material that students covered during their regular classes. The sessions began with a teacher-provided prompt, followed by free interaction between the student pairs and the AI tool. Teachers circulated the classroom, ensuring students’ interactions remained relevant and on task. Each teacher was provided with a three-part implementation toolkit which included: a) curated online learning resources on the use of Copilot and LLMs; b) a handbook focused on AI literacy and potential risks and benefits; and c) session guidelines, including suggested initial prompts and potential follow-up questions to assist students if needed. Teachers were also provided with contacts in case they faced any problems with the program implementation, and a group-chat was created to streamline communications. The students also had a customized guide, which included the initial prompts.

The lesson guides and their prompts were carefully crafted to position the LLM as a tutor, focusing on facilitating learning rather than simply providing direct answers. These prompts were informed by principles from the science of learning and were tailored to the cultural context of southern Nigeria, incorporating familiar names and customs to resonate with students. Some of the prompt structures were derived from Mollick and

Mollick (2023a). This design aimed to encourage the LLM to adapt to each student’s individual learning level, providing pedagogical support through contextually relevant examples and diverse teaching techniques. Students interacted with the LLM by asking questions, completing exercises, and receiving personalized feedback. At the end of each session, the students were encouraged to reflect and discuss lessons learned and challenges encountered during session to facilitate knowledge sharing among the group.

To ensure the fidelity of program implementation, monitors were first trained, provided with monitoring guidelines, and then assigned to track student attendance and gather information about each session using Kobo Toolbox. This system allowed for real-time data collection, ensuring that the intervention was carried out as intended in each school and offered the opportunity to respond promptly to any challenges.

Sample and Randomization

The randomization for the pilot program was conducted at the student level in the nine selected schools. All first-year senior secondary school students in these schools were informed about the program through information sessions and given a window of ten days to express their interest in participating. Only students who voluntarily expressed interest within this period were included in the randomization pool.

To assess whether students who expressed interest in the after-school program differed systematically from those who did not, we compare pre-program exam scores between students who were eligible for the lottery (i.e., those who later expressed interest) and those who were not. Table 12 reports estimates from regressions of baseline academic outcomes on eligibility status. In the first term, students who would later express interest scored 0.085 standard deviations higher than their peers (p ¡ 0.1) (see Figure 6). However, by the second term—still prior to the lottery—this relationship reverses: students who

8 did not express interest scored 0.147 standard deviations higher (p ¡ 0.01) (see Figure 7). The absence of a consistent directional pattern across terms suggests that selection into the program was not strongly or systematically correlated with academic performance. While our analysis focuses on treatment effects among those who expressed interest, the lack of clear academic selection implies that results may generalize beyond this group. Nevertheless, we lack demographic data on non-interested students, which limits our ability to assess representativeness along other dimensions.

Once the period to express interest closed, the randomization was carried out using simple random sampling without replacement among interested students to assign them either to the treatment group, which participated in the program, or to the control group, which did not receive any intervention but continued their regular learning in the classroom. The students completed a baseline survey and an end-line survey with sociodemographic information. Initially, 657 students were assigned to the treatment group and 671 to the control group. However, only 422 students in the treatment group and 337 in the control group completed the final assessment, which constitutes the final sample used for the analysis.

Table 1 provides summary statistics and balance tests for key observable characteristics of the two groups. Demographic variables include gender, age, and a socio-economic status (SES) index. This index was derived from a principal components analysis of household characteristics, such as access to goods (computers, phones), services (internet connection), study spaces, and parental education. The SES index, as well as other variables such as the proportion of female students and age, shows that the sample is balanced across the treatment and control groups, with differences that are small and not statistically significant. These results confirm that the randomization process achieved balance in key characteristics, supporting the validity of subsequent comparisons between the treatment and control groups.

In addition to sociodemographic information, baseline academic performance was measured using scores from the First and Second Term Exams conducted prior to the intervention. The difference between treatment and control group students in mean baseline scores for the First Term Exam is 0.131 (SE = 0.073), and for the Second Term Exam, it

9 is 0.096 (SE = 0.073). These differences are also statistically insignificant, indicating that students in both groups had comparable academic performance before the program.

Learning Data used as the Dependent Variable

At the end of the six-week intervention, participating and non-participating students completed a standardized assessment designed to measure three key outcomes: (a) English language proficiency aligned with the Nigerian curriculum for the corresponding period (our main outcome of interest), (b) knowledge of AI, and (c) understanding of basic digital concepts (from now on referred as ”digital skills” for convenience). The majority of the questions aimed to assess English language. To minimize the risk of cheating, multiple versions of the assessment were created, each with a randomized order of questions. Additionally, monitors were stationed at the schools to oversee the administration of the assessments and ensure compliance with testing protocols. The assessment was administered in a traditional pencil-and-paper format and consisted of multiple-choice questions designed by experts based on the Nigerian curriculum.

For each student, a simple score was generated based on the percentage of correct answers across all topics, as well as separate scores for each of the three domains (English language, AI knowledge, and digital skills). In addition to these unweighted scores, weighted scores were calculated for each domain and for the overall assessment. The weights were based on the ex-ante difficulty of each test item, which was determined by the test designers prior to the administration.

An additional score for proficiency in each theme was calculated using Item Response Theory (IRT). This method allowed for comparability across students by placing their performance on a common scale, taking into account the actual difficulty of each question as observed in the sample. The IRT-derived score provided a more nuanced measurement of English language ability -as well as knowledge of AI and digital skillsby considering both the students’ responses and the varying difficulty levels of the assessment items.

In addition to the intervention-specific assessment, an additional dependent variable was derived from the student’s final English exam scores. This exam, which was conducted independently by the school, covered the entire term’s content, which extended beyond the six-week period of the after-school program.