Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise

Generative AI, including Large Language Models (LLMs), has the potential to transform real-world domains like education, healthcare and law, which rely on a skilled workforce to handle complex tasks. For example, educators (e.g., teachers and tutors) are often trained to synthesize curriculum knowledge and recognize student needs in order to provide high-quality learning experiences. Traditionally, experts train novices by sharing their intuition and reasoning refined over years of practice. However, expert-guided training is costly and difficult to scale. Knight and Skrtic (2021) estimate an annual cost of at least US$4,800 per teacher for a coaching program, about 2-4% of district expenditures in the US, with national investments reaching tens of billions of dollars.

Traditional training programs not only demand significant time outside of instructional hours, something many part-time educators cannot manage, but they also tend to follow static curricula that are often misaligned to the practical needs of novice educators. As a result, many novice educators do not have access to expert guidance and must develop their skills on the job, resulting in students likely missing out on valuable learning opportunities, which can be particularly harmful for students who have the most to gain from improved educational experiences.

LLMs may be able to provide real-time guidance for novices at scale, but several challenges must be addressed to make this feasible. LLMs are trained on Web data (e.g., Wikipedia and Reddit) which differ substantially from real-world K-12 interactions, thus out-of-the-box LLMs often fail in real-world learning settings. Current techniques to adapt LLMs for real-world settings (e.g., fine-tuning or prompt-engineering) struggle to elicit appropriate behaviors because these approaches focus on surface-level language patterns and overlook the latent reasoning processes that expert educators have honed through years of practice to guide their decision-making. LLMs also lack real-world knowledge that is important for delivering high-quality learning experiences, such as knowledge of the curriculum, previous interactions, and future learning objectives. In contrast, human educators possess this contextual knowledge. Their knowledge may complement traditional LLMs to produce more effective approaches to supporting novices at scale.

We introduce Tutor CoPilot, a Human-AI approach to scale expertise by providing real-time suggestions to tutors remediating students’ mathematical mistakes. Our work builds on prior research by Wang et al. (2024a), which used think-aloud protocols to capture experienced educators’ reasoning and adapted LLMs to generate expert-like suggestions. Tutor CoPilot aims to enhance the quality of K-12 education at scale by delivering actionable guidance tutors can immediately apply during live sessions. The tool is tutor-facing only and offers multiple pedagogical strategies whenever activated, preserving tutor agency. Tutors retain control over whether and how to use these suggestions, allowing them to leverage their contextual knowledge. This approach can improve the quality and effectiveness of tutors’ responses while also supporting their professional growth. Figure 1 illustrates the user interface for tutors in panels (a) and (d), and the backend process for generating suggestions once the tool is activated in panels (b) and (c). Supplementary Information section A includes further details.

This preregistered study presents the first randomized controlled trial of a Human-AI system in live tutoring. In collaboration with FEV Tutor, a chat-based virtual tutoring provider, and a U.S. Southern school district, we conducted an intervention through an in-school, virtual tutoring program for mathematics with more than 700 tutors and 1000 K-12 students from schools that receive federal funding to support students from low-income families. This study aims to answer four research questions (RQs):

• (1) To what extent does Tutor CoPilot affect student learning?

• (2) Does the effect of Tutor CoPilot on student learning differ by the initial effectiveness of tutors?

• (3) How does Tutor CoPilot change tutoring quality as measured by the language tutors use with students?

• (4) How do tutors perceive Tutor CoPilot?

Our findings reveal that Tutor CoPilot significantly improves student learning outcomes (RQ1): Our intent-to-treat analysis shows students whose tutors have access to Tutor CoPilot are 4 percentage points (p.p.) more likely to master session lesson topics. Tutor CoPilot particularly benefits lower-rated and less-experienced tutors (RQ2), with these tutors improving their students’ mastery by up to 9 percentage points over the control, according to a heterogeneity analysis based on tutor ratings from the tutoring provider. Tutors with access to Tutor CoPilot are more likely to use high-quality strategies that foster student understanding (RQ3), determined by classifiers that identify high- and low-quality pedagogical strategies. Although our study design and implementation were not set up to estimate the direct impact of Tutor CoPilot on end-of-the-year (EOY) student achievement scores on NWEA MAP, a back-of-the-envelope calculation suggests that a full year of tutoring with Tutor CoPilot enabled may improve student performance on the test by 0.024 standard deviations (Supplementary Information section B). Finally, in our interviews with tutors, tutors reported that Tutor CoPilot was helpful but indicated room for improvement in its guidance, such as by generating appropriate grade-level language. With an estimated annual cost of just $20 per tutor based on usage patterns (Supplementary Information section N), Tutor CoPilot offers a scalable and cost-effective alternative to traditional, resource-intensive training programs. Overall, our findings demonstrate that Tutor CoPilot is a promising Human-AI approach combining LLMs with task-specific expertise, enhancing educational quality for students from underserved communities.