Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise

We conducted a randomized controlled trial (n=783 tutors) in partnership with FEV Tutor, a virtual tutoring provider, and a large southern U.S. school district to evaluate whether Tutor CoPilot can improve tutoring quality and student outcomes. Tutors in the treatment group received access to Tutor CoPilot during their live tutoring sessions from March 27th to May 17th of 2024 (Methods). Our theory of change centers on the observation that many tutors lack the experience to provide effective responses in the moment. As a result, they generate low-quality responses, such as giving generic feedback (e.g., “Good job”) or giving away the answer after a single student attempt, which can hinder student learning. By offering real-time guidance tailored to the conversation and following high-quality strategies, Tutor CoPilot empowers tutors with more effective language to develop the student’s understanding. Thus, the tool builds tutors’ skills while directly improving learning interactions.

We report the results of our preferred model specifications for each research question below. Supplementary Information section C reports descriptive results, such as the descriptive student numbers and tutor adoption of Tutor CoPilot; Supplementary Information section D reports the student-level analysis; Supplementary Information section E reports the tutor-level analysis; and Supplementary Information section F reports notes on compliance.

RQ1: To What Extent Does Tutor CoPilot Affect Student Learning?

Table 1 shows the results of our intent-to-treat analysis, which includes estimates of the effect of the tutor having access to Tutor CoPilot on session-level outcomes using our preferred model specification. The model includes student-level covariates (baseline MAP math scores and demographic characteristics), school and grade fixed effects, and clusters residuals at the student-tutor pair level to account for correlated observations when students have multiple sessions with the same tutor. We provide Romano-Wolf adjusted p-values to control the inflated type I error rate from multiple hypotheses testing, and present alternative model specifications in Supplementary Information section G. “Participation points” refers to points awarded by the tutor based on student engagement and effort during the session. “Exit Ticket Attempted” indicates whether the student attempted the exit ticket during the session. “Exit Ticket Passed (Conditional)” indicates whether the student passed the exit ticket, limited to sessions in which it was attempted. “Exit Ticket Passed (Unconditional)” captures whether the student passed without restricting to attempted sessions (i.e., students who did not attempt are counted as not passing).

We observed a significant positive treatment effect on students passing their exit tickets: Students working with treatment tutors were 4 percentage points (p.p.) more likely to pass their exit tickets (p<0.01, 62%→66% student passing rate from control to treatment). Supplementary Information section H reports treatment-on-the-treated results using a two-stage least square instrumental variable model to estimate the impact of using Tutor CoPilot during the session, rather than just having access to it, showing students were 14 p.p. more likely to pass their exit tickets (p<0.01) on average. These results remain significant under the Romano-Wolf multiple-hypothesis correction and under different model specifications presented in Supplementary Information section G.

We also found that conditional on attempting an exit ticket, students were 3 p.p. more likely to pass it (p<0.05) when they worked with a tutor with access to Tutor CoPilot. Our estimates also suggest students were 2 p.p. more likely to attempt the exit ticket during the session, but this result is not significant once we account for the multiple hypothesis correction.

We also report treatment effects on participation points and student survey outcomes, neither of which are statistically significant. Although we pre-registered these outcomes and hypothesized positive impacts from Tutor CoPilot, our analysis did not find evidence supporting this expectation. Our study design anticipated that students would frequently match with the same tutor, but this occurred less often than planned. Frequent changes in tutor-student pairings might have made it difficult to influence the student perceptions measured by our surveys, compared to a model where students consistently interact with the same tutor across multiple sessions.

RQ2: Does the Effect of Tutor CoPilot on Student Learning Differ by the Initial Effectiveness of Tutors?

Figure 2 shows how the effect of Tutor CoPilot on student learning varies based on tutors’ initial effectiveness, measured by their quality rating and level of experience at the tutoring platform. The results indicate substantial benefits for tutors with lower initial effectiveness in both categories. In Figure 2(a), lower-rated tutors experienced a 9 p.p. increase in students passing their exit ticket (56%→65% student passing rate from control to treatment). In Figure 2(b), less experienced tutors showed a 7 p.p. increase (61%→68%). We saw these treatment effects diminish with increasing tutor effectiveness, but notably, students of lower-rated or less experienced tutors in the treatment group performed at or above the level of students with higher-rated or more experienced tutors in the control group. This suggests that Tutor CoPilot helped less-effective tutors achieve outcomes comparable to their more-effective peers.

RQ3: How Does Tutor CoPilot Change Tutoring Quality as Measured by the Language Tutors Use with Students?

We investigated whether treatment tutors employed higher-quality instructional strategies compared to control tutors by leveraging NLP methods, in a three-step process. First, we defined the taxonomy of strategies based on Wang et al. (2024a) and salient response patterns in our data. Table 2 reports the final taxonomy. Second, we trained machine learning classifiers to identify these strategies at scale (Supplementary Information section M). Finally, we used these classifiers to identify the strategies in our entire dataset of 350,000+ messages and estimated the difference in strategy use between the two groups using a logit regression. We clustered the standard errors at the student-tutor pair to account for the correlation between same-pair sessions. See Supplementary Information section G.2 for odds ratios and Romano-Wolf p-values and alternative model specification with student and tutor random effects.

Figure 3 reports the log-odds of strategies used between treatment and control tutors, showing the impact of Tutor CoPilot on the types of strategies employed during tutoring sessions.

Based on our preferred specification, we found that treatment tutors were more likely to use strategies such as “prompting the student to explain”, which is aligned with expert-recommended practices for promoting deeper learning. This strategy was used approximately 10% more in treatment sessions compared to control sessions. In contrast, control tutors were more likely to “encourage students in a generic way” compared to treatment tutors, using this strategy 10% more. The higher likelihood of treatment tutors “prompting the student to explain” and control tutors “encouraging students in a generic way” are robust across all specifications and multiple-hypotheses testing adjustments. These findings provide evidence that treatment tutors might have achieved better student learning outcomes through the use of more expert-like teaching strategies (see Supplementary Information section G.2 for differences by tutor quality and experience).

RQ4: How Do Tutors Perceive Tutor CoPilot?

Interviews (n=18) indicated that tutors generally found Tutor CoPilot helpful, particularly for its ability to provide well-phrased explanations and break down complex concepts on the spot. They highlighted its usefulness in explaining difficult topics, such as differentiating between mean and median, and appreciated the clear definitions and hints it generated. However, some tutors noted areas for improvement, mentioning that the tool’s suggestions occasionally lacked alignment with students’ grade levels. A common issue was that the responses were sometimes perceived as “too smart,” requiring tutors to simplify and adapt them for clarity. More details on the tutor interviews can be found in Supplementary Information section I.