Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise

Background

Training Novice Tutors to Use Effective Strategies

One-on-one tutoring is one of the most effective ways to improve student outcomes and reduce educational disparities, particularly in mathematics—a subject strongly linked to college graduation rates and future earnings. Expert tutors leverage deep pedagogical and content knowledge to adaptively respond to individual students needs. They see mistakes as critical sites for learning. They can diagnose and remediate misconceptions with strategies such as Socratic questioning, tailored hints, prompting students to articulate their reasoning and self-explain, and providing elaborated, responsive explanations, all of which stimulate critical thinking and deeper understanding. Good math teaching can be especially challenging, as it requires instructors to engage students in reasoning about abstract concepts and solving complex problems, as well as understanding the developmental progression of problem-solving strategies to interpret students’ thinking on the fly.

Training novice tutors to achieve this level of expertise is challenging. Traditional professional development programs (e.g. workshops) are often costly, time-intensive, and disconnected from the real-time dynamics of tutoring environments, frequently limiting their effectiveness. As a result, students often work with novice tutors who struggle to remediate their mistakes in real time, and provide them with direct answers or overly generic feedback, missing out on critical learning opportunities. For example, tutors at FEV—our study’s context—would often respond to students with generic and scripted messages like “Incorrect”, “Please try again.”, “The answer is …". Our work seeks to address these challenges by providing a scalable, cost-effective solution that provides guidance to novice tutors in their live teaching contexts, helping them navigate the complexities of teaching mathematics. This approach also offers infrastructure for conducting experimental studies on effective tutoring practices.

AI in K-12 Education

Recent advances in AI have sparked excitement about the potential for LLMs to transform K-12 education, from intelligent tutoring systems (ITS) to automated feedback generation. LLMs have the potential to produce adaptive, human-like responses, enhancing earlier generations of ITSs. However, tuning LLMs to behave like expert teachers is challenging. Out-of-the-box LLMs are trained predominantly on internet text (e.g., Wikipedia) that differs greatly from authentic teacher-student interactions. Even strong prompting or fine-tuning techniques struggle to bridge this gap, as they tend to rely on surface-level patterns rather than deeper pedagogical reasoning that expert educators develop through experience. As a result, LLMs often fail to use pedagogical strategies that foster deeper understanding, like probing questions and others listed above, which can hinder students’ educational outcomes. Recent efforts have identified approaches to improving LLMs ability to produce expert-like responses in education, e.g., by providing editable templates and pedagogical instruction following. We build on prior work with Bridge, an adaptation method that captures expert decision-making by transforming raw think-aloud data with experienced educators into effective instructions for LLMs. Bridge has been validated to outperform fine-tuning and prompt-engineering baselines in generating expert-like pedagogical responses.

Approaches to Human-AI Collaboration

Human-computer collaborative systems combine the complementary strengths of humans and computers to enhance performance across various domains like healthcare, writing and data annotation. In education, this collaboration means using AI to support human expertise in situ rather than replacing it. Prior research shows AI assisting in discrete tasks like patient treatment selection, and guiding interactions in more structured, scripted environments such as call centers, where it provides novice workers with immediate, context-specific suggestions. However, far less work has addressed AI assistance for complex interaction tasks like teaching or tutoring, which entail dynamic, open-ended dialogues requiring AI to work in tandem with a human tutor over extended exchanges. Educators bring grounded, context-specific knowledge and empathy that AI lacks, while AI can speedily access vast information to offer real-time guidance. This complementary setup can preserve the critical human connection in teaching mitigate LLM lapses through human oversight—and is thus recommended by many in the field of AI and education. Drawing on prior work, we emphasize the human’s crucial role in selecting expert-curated strategies to guide LLMs in generating effective pedagogical suggestions. Prior findings suggest that without human selection, LLMs often default to repetitive strategies, whereas experienced educators employ richer, more diverse teaching strategies. This observation underscores the human’s critical role in guiding AI to deliver contextually appropriate and impactful instructional support.

Implications of Tutor CoPilot Findings

This study introduces Tutor CoPilot, a human-AI tool to scale real-time expertise in K-12 tutoring. It represents the first randomized controlled trial of AI-supported live tutoring. Our findings demonstrate that AI-generated guidance—based on expert thinking—can significantly improve tutoring quality, particularly for less experienced tutors. Notably, Tutor CoPilot improves student learning at a low cost of $20 per tutor annually, which is far cheaper than traditional training programs costing thousands of dollars. We also find that Tutor CoPilot helps tutors adopt high-quality strategies that promote deeper learning. Thus, our approach offers a practical and scalable solution for improving education, especially for under-served communities that stand to benefit the most from better quality education.

Tutor CoPilot also provides infrastructure for gathering experimental evidence on effective tutoring practices. It allows researchers to answer the question: how does suggesting a particular set of practices to tutors affect student learning? Our results show that students master lessons faster when their tutors are offered a bundle of expert-preferred strategies for mistake remediation. Evidence from this experimental method can support the design of effective tutor training programs.

Limitations

This study presents promising results on Human-AI approaches for scaling real-time expertise. However, we note the following key limitations to our work: generalizability, effectiveness of strategies, dosage and exposure, multi-modality, and user safety.

Generalizability

First, our intervention on Tutor CoPilot is conducted with novice tutors who teach young students from under-served communities (majority Hispanic) based in a Southern school district in the United States. Tutor CoPilot may be less effective for experienced educators, for whom generated suggestions may offer limited benefit. The learning needs of these students may differ significantly from those of students with different demographics, in other parts of the United States, in other countries, or in more affluent learning environments. Additionally, while we observed substantial improvements in students’ proximal learning gains (i.e., the students’ exit ticket performance), we did not find statistically significant improvements in end-of-year math test scores; please refer to Supplementary Information section D for this analysis. This may be due to the little variation in students’ exposure to the treatment and the relatively short duration of the study (two months). Future research would benefit from randomizing students to better assess the impact of the intervention. Although our human-AI collaborative approach has broader potential across domains that rely on human expertise, such as law and healthcare, evaluating this potential requires careful adaptation and validation to meet the demand of those professions.

Effectiveness of Strategies

Our study provides suggestive evidence of the impact of pedagogical strategies on student learning. Tutor CoPilot offers a bundle of strategies from which tutors can choose and edit the suggested responses. We are measuring the causal impact of offering the tool, rather than that of using particular strategies. We opted for this design, in which tutors have the flexibility to choose, to utilize tutor knowledge of the context (in some contexts some strategies may be more effective than they are in other contexts) and to maintain tutor agency, which alone can be important for their learning. Future work might create experimental designs that allow for testing the impact of strategies individually while maintaining tutor agency and for testing the effects of the tool on tutor learning and effectiveness. Such work could also consider measuring key contextual factors that may influence the effectiveness of a strategy at a given moment (e.g., the number and types of attempts a student has made at solving a problem).

Dosage and Exposure

Our study design had to account for the tutoring platform’s almost-random algorithm to match tutors to students each session, which led to our choice of randomizing tutors and focusing on session-level outcomes. Combined with a short implementation period of about 2 months, most students in our sample had very little exposure to tutors with access to the tool that alternated with control tutors, which may also have restricted the effect of the tool on student perceptions measured by our survey and End-of-Year MAP achievement scores. Future research will benefit from a design that allows for higher intensity and contrast of Tutor CoPilot’s exposure between students to measure the tool’s impact on achievement scores and consistent student-tutor matching to measure changes in student perceptions of their tutors.

Modality

Our study collaborates with a chat-based tutoring platform, which is well-suited for real-time use of language models. Future work would benefit from extending Human-AI approaches to include other modalities—such as vision (e.g., whiteboard or student face) and speech (e.g., student and tutor voice)—to incorporate naturalistic modalities for interaction. Including these modalities would provide a more comprehensive understanding of the student and thus enable better expert-like guidance during the interaction. The shift towards multi-modal approaches also raises interesting challenges on how to provide real-time guidance through different modalities, such as via whiteboard drawing or verbal explanations.

User safety

While our study de-identifies names, user identities can still be inadvertently disclosed through non-name information, such as email addresses, phone numbers, or personal anecdotes shared by students and tutors. Similar privacy concerns arise when incorporating vision and speech modalities, highlighting the need for better safeguards. Additionally, we observe a trade-off between performance and safety. In our study, to mitigate over-exposure, we limited the conversation context passed to external APIs; in initial pilot studies, we varied the conversation context between 5 to 10 messages and found that tutors preferred the quality of interactions when more context was included. Balancing these factors will be important for future research.

Future Work

Looking ahead, our work can extend in several exciting directions. First, we aim to explore how well novices retain the skills they acquired from real-time expert guidance. This analysis will shed light on the long-term effects of the use of AI systems in real-world settings. Second, expanding Tutor CoPilot to other skill areas—such as developing collaborative learning—and applying it in different subjects, age groups, or system designs will provide insights into its broader impact. Last, we observed significant variation in how tutors adapted or personalized AI-generated suggestions, highlighting interesting human-AI collaborative dynamics that merit further study.