Scaling up successfully: Lessons from Kenya’s Tusome national literacy program

Several countries have recently begun large-scale educational interventions to respond to low learning outcomes. The evidence used to justify national or large-scale programs typically has emanated from pilot initiatives that increasingly have provided rigorous evidence of impact on learning outcomes. For example, several recent meta-analyses showed a dramatic increase in results from pilot programs focused on improving learning outcomes while also using rigorous research designs (Conn 2017; McEwan 2015; Murnane and Willett 2011).

Although an increased dependence on causal evidence to justify large-scale implementation strengthens the research base for large programs, the decisions to take these programs to scale have largely been undertaken without a body of literature that examines the barriers to successful large-scale educational implementation or whether the external validity assumptions for these comparisons hold (Bates and Glennerster 2017). This failure to develop robust scale-up literature and practice has allowed the field of educational development to focus heavily on proof of concept, with randomized controlled trial (RCT) studies estimating the program impact of small- or medium-scale interventions in several contexts. We found a handful of recent studies and research that made explicit a model by which evidence from pilot programs should interact with scale-up designs. However, this evidence is too recent to influence the design decisions made by governments. Instead, based on the guidance of educational researchers, governments often depend on pilot evidence as sufficient for scale-up decisions, without undertaking a thorough investigation of the aspects of system capacity that will be most critical to producing similar impact at scale.

The scale-up literature is pessimistic about both the initial take-up of educational actors and the long-term impact on learning outcomes, to say the least. Scale-up efforts in a variety of areas have proven ineffectual, including school-based teacher professional development initiatives (Chege 2011), cluster-based teacher support initiatives (Piper 2009), in-service teacher professional development programs (Villegas-Reimers 1998), the development of head teachers’ instructional leadership skills (Cambridge Education Consultants 1998), the expansion of mathematics and science instructional capacity in East Africa (Kamau et al. 2014), tablet-based interventions for children (Christia et al. 2014), tablet-based school support (Doykos et al. 2015), and television-based classroom learning programs (Assefa 2017). We wish to emphasize how infrequently large-scale educational interventions have had any statistically significant impact on teacher practice (Jukes et al. 2017) and, as a result, how often such interventions have had a negligible impact on student learning.

Some recent exceptions to this pessimistic view of the scalability of educational reforms exist. The Gauteng Primary Language and Mathematics Strategy program in Gauteng Province in South Africa sustained increased learning outcomes across several grades over a few years (Fleisch et al. 2016). Escuela Nueva has shown ongoing improvement in teaching and learning in Colombia and a dozen other countries, mostly in Latin America, and at increasingly large scale (Rincón-Gallardo and Fleisch 2016). Working outside of government systems, Pratham’s work has consistently shown significant effects on literacy in thousands of schools and communities in India (Banerji and Chavan 2016). These examples of successful scale-up were found in a recent issue of the Journal of Educational Change,Footnote 1 book-ended by reflections on improving outcomes at scale by Fullan (2016) and Elmore (2016). However, even when reflecting on the large-scale impacts identified in these and other programs in the special issue, Elmore (2016) questioned the entire enterprise of focusing on systematic instructional reform given the complexity of having individual teachers determine whether to implement the program behind closed doors.

The large-scale interventions reviewed in the special issue of the Journal of Educational Change used various implementation designs. The studies showed that each program faced project-specific hurdles to achieving program impact. Some of the programs failed because they did not sufficiently account for the demand side of the intervention, whereas others failed because they were overly technological solutions to instructional programs. Still others did not succeed because they expected teachers to invest more time and effort than they were willing to provide on a consistent basis, and some failed because they were implemented outside of government systems and, therefore, did not change the day-to-day practice of education officials.

However, the experiences of the programs reviewed in the Journal of Educational Change did point to a set of factors or conditions under which core pedagogical practices can be changed at scale (from Elmore’s 1996 study, also cited in Rincón-Gallardo 2016). Attention to the instructional core is one of them, including setting up the means to articulate and communicate what good pedagogy looks like. Another factor involves reinforcing that core of instructional practice by developing new normative structures and putting in place organizational relationships that intensify teachers’ motivation to adopt those practices. Systems also need structures to promote, provide ample practice with, and thus reproduce the new pedagogical approaches; as well as incentive systems that are tied to those structures.

Fullan and Quinn (2016) argued that the difference between failure and success in large-scale education system improvement is determined by whether interventions focus on the right or wrong drivers of reform. They classified as “wrong drivers” interventions focused on punitive accountability, individualism, technology, and fragmented policies. Their “right drivers” included capacity building, pedagogy, and systemic policies (Fullan and Quinn 2016). Attention to pedagogy as the driver of improved learning outcomes (whether at the micro or system level) was reiterated by the findings of the International Initiative for Impact Evaluation’s (3ie’s) summary review of the impact of education programs, which showed that the largest and most consistent positive effects on learning resulted from programs that used structured approaches to improving pedagogy (Snilstveit et al. 2016). The experience in Gauteng Province in South Africa mentioned above also demonstrated the importance of structured approaches to pedagogy, with additional emphasis placed on the institutional infrastructure needed to support those approaches (Fleisch et al. 2016). That infrastructure includes a variety of high-quality learning materials, implemented in combination with support to teachers (in the form of ongoing coaching) that enables them to utilize those learning materials and that establishes an environment of professional instructional accountability (Fleisch 2017). This notion of institutional infrastructure is consistent with Pritchett’s (2015) idea of a system that is “coherent for learning,” in which the key principal-agent relationships that govern the operation of the education system are aligned to and supportive of specific improvements in teaching and learning (Pritchett 2015, p. 4).

While Fullan (2016) also argued that there are limitations to an overly prescriptive approach to pedagogy, the evidence from 3ie (Snilstveit et al. 2016), McKinsey and Company’s theoretical model (Mourshed et al. 2010), and the United States Agency for International Development’s (USAID’s) Landscape Report on Early Grade Literacy (Kim et al. 2016) indicates that developing countries trying to improve very low-functioning education systems should focus on introducing highly specific approaches to instructional change and putting in place the supports needed for that change to be realized across large numbers of classrooms. In Kenya, Piper et al. (2018) examined the relative impact on learning and cost-effectiveness of training, coaching, 1:1 literacy materials for children, and structured teachers’ guides. They found that the entire package resulted in much greater impacts than books and training or training alone. This growing body of evidence makes a strong case for focusing on curriculum, pedagogy, and materials aligned around a structured sequence of lessons; supporting teachers through relatively frequent coaching in specific instructional techniques, including how to use new materials; and regularly monitoring teacher adoption of those practices and students’ learning outcomes.

Informed by this recent literature’s attention to how education systems do or do not assemble the ingredients for large-scale improvements in learning outcomes, very recent efforts have begun to expand our understanding of large-scale program impact. Five reports in particular have driven a revision of the educational development field’s thinking regarding scale-up. First, the Millions Learning research by the Brookings Institution (Robinson et al. 2016) examined how small pilot programs expanded into more and larger settings. However, that research paid little attention to how and whether the programs were embedded in government systems, which is, in many contexts, a prerequisite for successful large-scale implementation. Second, a framework developed by Management Sciences International (MSI) (Cooley and Kohl 2006) on scale-up provided a typology for how scaling happens. This typology has driven some donor-funded programs to at least address the scalability of interventions. However, few program designs have systematically applied MSI’s complex and linear framework; therefore, its relevance for national or large-scale implementation in education is difficult to assess. Third, a manuscript examining large-scale program implementation in the literacy realm was intended to help USAID missions design and manage such programs for sustainability (DeStefano and Healey 2016). This theoretical framework drew on the Millions Learning research and the MSI frameworks mentioned above and focused specifically on the challenges associated with improving at scale the teaching of reading in the early grades. However, this framework has only nominally been applied to existing programs, and it has yet to be used to guide the initial design or implementation of large-scale education programs. Fourth, the special issue of the Journal of Educational Change reported several recent attempts to implement instructional change at scale, with an opening essay describing the contours of a systematic review of the scale-up literature (Rincón-Gallardo and Fleisch 2016). Fifth and finally, applying the principles of “doing development differently” (The DDD Manifesto Community 2017) to the challenge of achieving large-scale improvements in learning outcomes, Crouch and DeStefano (2017) identified a core set of capacities that education systems should focus on to shift away from bureaucratic administration toward the development of management relationships that are coherent for learning (as argued for by Pritchett 2015) and able to support large-scale instructional change. Crouch and DeStefano (2017) defined the three core functions that education systems need to conduct rapid cycles of planning, action, reflection, and revision: (1) setting and communicating expectations, (2) monitoring and guaranteeing accountability for meeting those expectations, and (3) intervening to ensure the support needed to assist students and schools that are struggling.

Much of this literature on scale-up has focused on how innovations are introduced and then taken to scale. The emphasis is on identifying the conditions needed for innovation to spread and be adopted or adapted in a growing number of sites. However, often, the challenge is not simply how to support or promote the spread of innovation. Successfully scaled programs require building the institutional capacity of an education system to itself first enable a new approach to be implemented at scale and then to support and sustain schools as they work to incorporate that approach into their day-to-day operations. Therefore, we propose an analytical approach that examines how one educational reform program currently being implemented at national scale, the Kenya Tusome Early Grade Reading Activity, has performed in its attention (or lack thereof) to the core functions defined by Crouch and DeStefano (2017). We will demonstrate whether and how Tusome has reinforced education system capacity to better fulfill these core functions. The comparison of this framework to the available evidence in Kenya is supported by an analysis of the Tusome literacy program’s impact on student learning in Kenya.

Tusome, the national literacy program of the Kenyan Ministry of Education, was designed and implemented in a context of growing interest in large-scale program implementation. Based on rigorous evidence from previous research on improving literacy outcomes in Kenya (Gove et al. 2017), and funded by USAID, Tusome began in January 2015 with a formal launch by Kenyan President Uhuru Kenyatta. Tusome supported all of Kenya’s Class 1 (grade 1) students with materials in English and Kiswahili and training for teachers in 2015, expanded to all of Kenya’s Class 2 students in 2016, and expanded again to Class 3 in 2017. What makes Tusome a plausible research subject for the scaling literature is that it drew on the evaluations of several pilot studies in Kenya and enjoyed the benefits of careful design focused on large-scale implementation possibilities. Before Tusome, the Primary Math and Reading (PRIMR) Initiative was a 547-school research study supported by USAID/Kenya and undertaken by the Ministry of Education with technical support from RTI International. Its intent was to assess whether a low-cost intervention implemented primarily through the existing infrastructure could support sufficient initial take-up by teachers to result in increased learning outcomes (Piper and Zuilkowski 2015). Additional pilot conditions were tested in the PRIMR Rural Expansion Programme, funded by the UK Department for International Development (DFID) Kenya, in 847 schools, with additional treatment groups assigned to test various conditions essential to improving outcomes (Piper et al. 2016a, e, 2018). Note that the PRIMR intervention and the subsequent Tusome program focused on a particular set of literacy skills that can be assessed orally and relatively simply, rather than emphasizing a broader definition of literacy’s ability to change an individual’s situation even as it changes the individual (Bartlett 2008) or the potential of literacy to support political freedom (Friere 1970). Some of the large-scale interventions described in the special issue of the Journal of Educational Change included a slightly broader definition of what literacy improvement means, including the Escuela Nueva program in Colombia (Colbert and Arboleda 2016) and the Learning Community Project in Mexico (Rincón-Gallardo 2016).

Tusome was one of the first experiences of taking a piloted literacy program to national scale through government systems. The findings from the pilots were used to design key elements of the national-scale implementation (Piper et al. 2015b, 2016b), and several other large-scale literacy programs were then designed based on Tusome’s experience, even before clear evidence of the program’s scalability or impact was documented. External evaluation data on Tusome’s impact and both school- and classroom-level take-up by individual education actors (including teachers) and learning outcomes data from Tusome’s first year of implementation are now in hand (baseline results, MSI 2015; midline results, Freudenberger and Davis 2017). Thus, Tusome’s transition from a medium-scale pilot to national-scale implementation in 23,000 public primary schools and 1500 low-cost private schools [called Alternative Provision of Basic Education and Training (APBET) institutions in Kenya] offers an interesting case study for the application of the Crouch and DeStefano (2017) core functions framework.