Scaling up successfully: Lessons from Kenya’s Tusome national literacy program

RQ1: Setting expectations

Research question 1 asks whether Tusome made expectations for learning outcomes clear. To answer this question, we investigated the program’s design documents and the training manuals used in coach and teacher professional development. We found clear evidence that Tusome communicated the Ministry of Education’s benchmarks for literacy outcomes within the design documents and training materials, but more importantly, we found that those benchmarks were widely disseminated, understood, and reinforced. With the support of PRIMR and, later, Tusome, the Ministry of Education has defined expectations for reading achievement by setting benchmarks for proficient reading in both Kiswahili and English in the first years of primary school (Piper et al. 2016b), as shown in Table 1 below. Analyzing Tusome training documents from two national-scale trainings in 2015, three national-scale training programs in 2016, and two more national-scale trainings in 2017 showed that teachers were trained on the existence of the literacy benchmarks and that actual data from Kenya were consistently shared during termly trainings to demonstrate and reinforce how Kenya’s children were performing relative to these expectations. Evidence from training reports generated during these sessions suggests that the benchmarks were typically understood by the participants and that they could recall the benchmarks and what the literacy results meant by grade and language in terms of the progress being made (or failing to be made).

Second, instructional expectations are embodied in the teachers’ guides distributed to teachers across the program. The guides (and the training in using the guides) explicitly communicate to teachers the sequence of lessons they are to implement to build the reading skills expected by the national primary curriculum. The external evaluation results showed that these teachers’ guides were widely available, with 95% of teachers at the national level using the guides during classroom observations (Freudenberger and Davis 2017).

Third, the Tusome training program encouraged teachers to use their guides consistently to deliver high-quality lessons. The emphasis in Tusome was on program fidelity, and during training, the fact that teachers should expect to see progress toward literacy benchmarks based on their consistent use of the lessons as laid out in the teachers’ guides was specifically highlighted. To reinforce this message, classroom support was similarly focused on the daily implementation of the lessons in the teachers’ guides.

Fourth, CSOs were expected to regularly visit schools and were given tools and training to enable them to monitor and support teachers in adopting the instructional methods and using the provided Tusome learning materials. During these visits, CSOs uploaded the results of a benchmark assessment designed by the Ministry of Education to be specific to the material covered each term and administered to three children in each classroom.Footnote 4

To determine whether Tusome was able to support Kenyan government officers to focus on fidelity, we reviewed whether the instructional support function had been internalized by the government system. Tusome’s work has been done primarily by government officers, and a key focus has been working with the Teachers’ Service Commission to reinvigorate the classroom instructional support role of the CSOs. CSOs represent the front line of the public education system in Kenya—the place where the government’s administrative apparatus interacts most directly with schools.

Under the PRIMR pilot program, the project supported the Teachers’ Service Commission to evaluate and reform the role of the TAC Tutors (now CSOs). In 2016, using a new job description for CSOs, the TSC re-interviewed and rehired CSOs under a job description that focused more specifically on instructional support. This job description emphasized visiting schools regularly, observing teachers’ lessons, and providing constructive feedback on their instructional practice—a significant departure from the previous de facto role of CSOs that stressed supervising school administration.

Under Tusome, CSOs were trained three times a year, given tablets with instructional support tools to help teachers, and reimbursed for travel to enable them to visit schools, record their observations, advise teachers and head teachers, and do spot assessments of student progress. In the 2016 academic year, CSOs across Kenya recorded and uploaded 113,604 classroom lesson observations at schools during the seven key instructional months in the Kenyan calendar, corresponding to an average of more than 90 visits per CSO; this rate of school supervision and support far surpassed what the education system was doing before (Kisirkoi 2012). Encouragingly, the 2017 observation data indicated even more classroom visits than in the prior year. The more frequent visits, and the observation of teachers during visits, communicated and reinforced the new expectations for teachers. Having CSOs show up regularly to observe their lessons let teachers know that the education system expected them to deliver a specific sequence of lessons and employ the instructional techniques on which they had been trained.

RQ2: Monitoring progress

To determine whether Tusome supported the government’s core function of monitoring progress toward the new system, teacher, and student expectations, we reviewed program documents and the large-scale database of classroom outcomes shared in the Tusome dashboard. The work of the CSOs described above communicated and reinforced expectations and monitored the extent to which those expectations were met. We found that Tusome had built system capacity to monitor expectations in four important ways.

First, as mentioned above, at the end of each classroom observation, the CSOs randomly selected and evaluated the oral reading fluency of three students. Although a small number of students were sampled at each visit, cumulatively, over the course of several visits within a sub-county or county, the number of students added up to a reasonably representative sample for that geographic area. These data were centralized and viewable on the dashboard, which presented both national and county-level data, as shown in the sample in Fig. 1. The dashboard is shared each term with all CSOs, in front of the Ministry of Education’s County Directors of Education to whom they are responsible; as we have argued elsewhere (Piper et al. 2017a), this is a meaningful expansion of the accountability function of the Ministry of Education.

The dashboard in Fig. 1 shows the average oral reading fluencies and percentages of students meeting performance benchmarks in English and Kiswahili in Classes 1–3 in each county. For example, as of September 2017, across all the counties in Kenya, 57% of Class 1 students tested were reading at or above the benchmark in English, and 70% were meeting the benchmark in Kiswahili. However, when we look at each county, this dashboard shows considerable variation, especially in performance on the reading fluency measure, with some counties performing particularly poorly. How data such as these are or are not being used to direct and target support resources is discussed below.

Second, in addition to monitoring student outcomes, Tusome equipped and trained CSOs to monitor teacher adoption and application of the desired instructional practices. At each visit, the CSO recorded which lesson the teacher was delivering that day, noted whether each component of the lesson was well delivered, and indicated whether the teacher employed the techniques on which he or she had been trained. This information formed the basis for the feedback offered by the CSO and was aggregated to generate a picture of whether teachers in a district were meeting expectations for improved instruction. In addition, the Tusome technical team often followed the CSOs to the classroom and to their subsequent one-on-one sessions with teachers and then gave feedback on the quality of the CSOs’ instructional support. Based on those visits, the Tusome field staff shared daily updates with the appropriate government officers on teachers’ progress in delivering the expected sequence of daily lessons. Education officers then gave instructions to the CSOs—and head teachers—to support those teachers who were falling behind in Tusome lesson delivery.

Figure 2 presents the progress through the sequence of lessons during classroom observations in October 2016, the last month of Tusome’s second year of implementation. The figure shows small variations by grade and by language, but on average, Class 1 and 2 teachers had taught more than 124 Tusome lessons by that point in the year, meaning that they were within 10 lessons (i.e., two school weeks) of where they should have been. The lowest quartile of teacher performance on this measure corresponded to those teachers who were 4 weeks behind what would be appropriate for October, which is the middle of Term 3.

While CSOs monitored teachers and assessed students as described above, the CSOs were also monitored. Each time a CSO visited a school, the tablet they used to record their visit and lesson observations took a global positioning system (GPS) reading to verify the location. The classroom observation data and GPS location were then uploaded and aggregated into monthly reports showing whether the CSO made the target number of visits to that location every month.

The data on the number of classroom visits were used to determine the travel reimbursement each CSO received—depending on whether they met the target number of visits for each school (i.e., not simply visiting the nearest schools multiple times while ignoring harder-to-reach locations). This type of monitoring, coupled with the explicit incentive created by linking travel allowances to the fulfillment of specific school-visit responsibilities, greatly increased the accountability of CSOs for meeting the expectations placed on them for school and teacher support. The accountability structure of Tusome also expanded beyond the CSOs, with the dashboard showing the percentages of target visits at the county and national levels. These data were consistently used by Ministry of Education leadership to draw attention to and increase accountability for instructional support. The Tusome program has supported the use of these data by both national-level education leaders and the county-level officers who are much closer to the actual CSOs whose behavior the program is trying to influence.

Fourth, our analysis of the Tusome data suggested that the accountability structures available in Tusome have driven demand for similar accountability structures in other programs. Kenya’s Primary Education Development project (2015–2019), an early grade mathematics program operating with funds from the Global Partnership for Education, has created mathematics benchmarks for Classes 1 and 2 and a similar tablet-based, GPS-tracked tool for CSO observations of mathematics classrooms. Furthermore, the quality assurance and standards system, which serves as an inspectorate in Kenya, also created a set of tablet-based tools and a dashboard for their system. Finally, the sub-county directors of education, similar to district education officers, were provided with a Ministry of Education-designed tool that is available on the Tusome dashboard platform. The demand for benchmarking and accountability systems in these other areas is evidence that Tusome is helping Kenya institutionalize mechanisms to monitor and reinforce expectations for decentralized school support, teacher instructional behavior, and student outcomes.

The demand for the accountability structures in Tusome is encouraging, but implementation still requires improvement. For example, the number of visits by CSOs to mathematics classrooms typically corresponded to just over 10% of the number of visits to Tusome-supported literacy classrooms. Moreover, quality assurance officers and sub-county directors of education undertook only a fraction of the expected numbers of class and school visits, and most of these officers did not upload observation data consistently.

RQ3: Providing differentiated support

If the education system in Kenya is indeed creating new expectations for teachers and students, then the system must also provide teachers and students the resources they need. The external evaluation of Tusome (Freudenberger and Davis 2017) showed that Tusome had assisted the Ministry of Education in ensuring provision of the basic inputs necessary to implement an early grade reading program at scale. In more than 97% of classrooms across the country, teachers had the teachers’ guides containing the lesson plans for their grade, 99% of classrooms had a 1:1 ratio of books to students in both subjects, and teachers reported having participated in training on early grade reading instruction frequently, with the median teacher reporting having been involved in four to five training sessions. The provision of coaching, as mentioned above, was also being assured, albeit with somewhat less fidelity than the provision of training and materials. More than 80% of Class 1 and 2 teachers reported being observed by a CSO at least once per term (compared to the official target of three times per term).

Data gathered from ongoing support visits and aggregated at the county and then zonal level indicate the instructional challenges teachers are facing as they work to adopt the Tusome approach and could be used to better structure future trainings and immediate support. Our findings were mixed in the area of providing differentiated support. We found that, in fact, Tusome used the large-scale data collected from classrooms to inform the design of trainings that were responsive to the needs of teachers. Specifically, we found that the classroom observation briefs collected by Ministry of Education officers and Tusome staff inspired a robust email-based discussion of these findings on a daily basis. These data included specific feedback to individual CSOs on how to respond to the challenges that teachers encountered during their observations and summaries of the challenges the teachers described. The trainings organized during the term breaks took those challenges into account and could be focused in response to teachers’ needs within each county and zone.

While the data above showed how Tusome responded to monitoring data via its aggregate training programs, the review of the Tusome program showed no evidence that data were used to inform county- or zone-specific interventions when the data showed weak levels of implementation. The monitoring data did reveal to each CSO the schools in their jurisdiction where teachers were not meeting instructional expectations and/or where students were underperforming in reading. The dashboard mentioned above could be used to indicate to individual CSOs which schools need more support and to Tusome and Ministry of Education officers which CSOs need support. However, the evidence suggests that this support was not provided frequently, if at all, and that no plan based on classroom visit data was implemented to help CSOs know which teachers to visit more frequently.

RQ4: Impact on learning outcomes

To answer RQ4, we reviewed the external evaluation report prepared by MSI (Freudenberger and Davis 2017) to investigate the impact of Tusome on learning outcomes. Given that Tusome was implemented nationally and that there is no counterfactual, the external evaluation report compared the learning outcomes in a nationally representative sample of schools between 2015 and 2016. Table 2 shows the impact of Tusome on Kiswahili reading outcomes (for an explanation of the reading subtasks assessed at midline, see Annex III of Freudenberger and Davis 2017, p. 65). The external evaluation assessed children in both Class 1 and Class 2 on six Kiswahili subtasks. The data showed statistically significant increases in national-level learning outcomes for each of the six subtasks and for both Classes 1 and 2. The table also shows the effect sizes of the differences between baseline and midline. The effect sizes were 0.50 SD or higher for all estimates, except for invented word decoding, which was 0.45 SD in Class 1. The average effect size for Class 1 was 0.63 SD, and that for Class 2 was 0.76 SD. These effects are considered large in the education literature that looks primarily at small-scale pilot programs (J-PAL 2018) and are significantly larger than the Kiswahili effects in the PRIMR pilot studies (Piper et al. 2016b).

Table 3 presents the impact of Tusome on English in Classes 1 and 2 and was drawn from findings presented in Freudenberger and Davis (2017). Similar to Kiswahili, the results showed statistically significant gains in each of the eight tasks and in both grades. The magnitude of the impacts was typically large, with effect sizes above 0.50 SD except for vocabulary (Classes 1 and 2) and a short reading comprehension task (Classes 1 and 2). The average effect size for English Class 1 was 0.67 SD, and for Class 2, it was 1.04 SD. These impacts were significantly larger than that of PRIMR on English outcomes determined in previous studies (Piper and Zuilkowski 2015; Piper et al. 2014, 2018).

Encouragingly, the percentage of Class 2 children who met the emergent benchmarks increased from 34 to 65% for English and from 37 to 66% for Kiswahili (Freudenberger and Davis 2017), nearly doubling both proportions. These are massive gains for research on educational interventions in general but even more so considering that Tusome is implemented at the national level.