Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise

The implementation part of our study lasted for about 7 weeks, starting with the launch of Tutor CoPilot for treatment tutors on the platform on March 27th, and ending on May 17th, 2024, with the end of the school year. We preregistered our primary hypotheses and analysis plan on the Open Science Framework platform prior to accessing the data, which can be found: https://osf.io/8d6ha.

Randomization and Participants

We partnered with FEV Tutor, a chat-based tutoring provider, and a large southern U.S. school district for this study. Students receiving tutoring through FEV Tutors were matched to an available tutor upon their logging into the platform but were not guaranteed to consistently match with the same tutor. Considering that Tutor CoPilot is a tutor-facing tool only that requires training, we opted for randomizing tutors into the treatment (access to Tutor CoPilot) or control (business as usual) conditions. That is, students were not randomized into one of the two conditions, and their treatment status could vary from session to session, depending on the treatment assignment of the tutor with whom they were matched. For this reason, we are not able to estimate the impact of Tutor CoPilot directly on student EOY achievement scores, but rather focus our analysis on session-level outcomes for students, such as session exit tickets.

Tutors

FEV Tutors initially identified 900 tutors who were assigned to work with students from our study’s school district and randomly assigned to the treatment (450) or control (450) conditions. However, due to attrition between the initial assignment time and the study launch, only 872 responded to our pre-launch survey, and of those, 783 (treatment = 388, control = 395) had tutoring sessions with a student from our partner district for this study after Tutor CoPilot was launched. Balance checks show that the control and treatment groups do not differ statistically on any baseline variables; these results are detailed in Supplementary Information section J.

The tutors were paid, full-time professionals employed by FEV Tutor and had approximately two years of working experience with the tutoring company on average (Table 23). All tutors were required to participate in professional development sessions, which included proprietary training modules on math remediation strategies, independent of the Tutor CoPilot tool. This training promoted similar practices of responding to student mistakes and not immediately giving away answers, and provided examples of student mistakes and example responses to be used. Tutors assigned to the treatment condition received additional training on how to use the Tutor CoPilot tool prior to its launch to minimize the novelty effect. Tutor CoPilot training for treatment tutors is detailed in Supplementary Information section K. While, in theory, the additional professional development for tutors receiving access to Tutor CoPilot may have affected their practice and student learning directly, any effect is likely to have been small given that all tutors were receiving similar information in their initial training.

Students

The school district serves more than 30,000 students, with the majority from ethnic/racial minority groups and economically disadvantaged backgrounds. A total of 1,534 students from nine schools in this school district were initially identified as eligible to participate in our study by the district. Student eligibility to participate in our study was determined by student eligibility for tutoring services according to the state policy for administering accelerated services, which meant performing below grade level on the state test the previous spring. We restrict the sample of students in our study analyses to 1,013 students who attended at least one tutoring session with a tutor enrolled in the study and randomly assigned to the treatment or control condition. The study was conducted with elementary and middle school students in grades 3-6. The majority (79%) of our sample identifies as Hispanic, and 67% of the sample is classified as economically disadvantaged. The full sample description is provided in Supplementary Information section C.

Data

Our study uses three types of data about the tutor, student and session. Supplementary Information section L provides more details on our study’s data.

For tutors, we collected their treatment group assignment, gender, pre-study quality rating, and pre-study tutoring experience. The quality rating is the provider’s internal measure of quality and is based on manual observations of a random sample of the tutor’s sessions, averaged from scores on the provider’s proprietary observation rubric; it takes on continuous values between -1 and 1. The tutor’s tutoring experience is the number of months the tutor had been working with the tutoring provider leading up to the start of the study.

For students, we obtained data about their gender, race, pre-/post-study NWEA MAP Math and Reading Scores, and other relevant covariates. The NWEA MAP assessments are standardized tests administered three times a year, tracking students’ academic growth over time.

We tracked session outcomes such as the number of participation points the tutor awarded the student for their engagement and effort during the tutoring session, whether the student attempted and passed the lesson exit ticket, and student post-session survey responses. Exit tickets are particularly critical as they assess student mastery of each topic and directly influence the student’s progression through the curriculum. Students must pass the exit ticket to advance to the next lesson, meaning students who struggle on exit tickets may encounter delays in their learning progression, limiting their exposure to new material. As a result, the exit ticket pass rate is not just a measure of immediate comprehension—it plays a key role in determining the pace at which students advance through their academic content. Moreover, the number of completed exit tickets by a student is a significant predictor of future MAP test performance – one additional exit ticket completed is associated with a 0.02 standard deviation increase on the test; we report this in Supplementary Information section B.

Additionally, we collected each session’s chat activity and Tutor CoPilot usage. Our final analysis sample includes 4,136 sessions, 2,014 with treatment tutors and 2,122 with control tutors. This translates into 350,000+ chat messages exchanged between tutors and students, more than 170,000 from treatment sessions and 184,000 from control sessions. We observe 2,000+ uses of Tutor CoPilot, defined as the number of clicks on the tool during the session that correspond to either an initial activation of Tutor CoPilot (Figure 1 a) or the tutor clicking on a different strategy from the drop-down options (Figure 1 d). We do not have data distinguishing different types of clicks.

Analysis

This study tested the causal impact of Tutor CoPilot on student learning outcomes and tutor practices. Below, we summarize our preferred methods for analyzing the following research questions. Following our pre-registration, we imputed missing covariates for students and tutors. For categorical variables, we added an additional “missing” category. For continuous variables, we assigned the predicted value based on the other present covariates. Our initial preregistered outcome was EOY MAP Math achievement scores. The implementation resulted in less consistent treatment exposure than expected, however, so we focus on the available session-level outcomes for assessing the impact of tutor copilot on academic outcomes. See Supplementary Information section D for information on the distribution of treatment exposure and the student-level analysis.

RQ1: Impact on Student Outcomes

We employed an intent-to-treat (ITT) regression analysis to determine how offering the Tutor CoPilot to tutors predicts student outcomes. Equation describes our primary model:

where Yist is the session i outcome of interest (e.g., exit ticket passed) for student s working with tutor t in a given class (combination of school and grade) k, Treatmentt is the indicator for whether the session tutor t is in the treatment group, Xs is a vector of student-level covariates, ωk⁡(s) is the fixed effect for class (school and grade), and ϵist is the residual clustered at the student-tutor pair level to account for correlated observations when students are matched to the same tutor more than once. The student covariates include categorical indicators for the student’s gender, race, free and reduced lunch, special education, and limited English proficiency, as well as a continuous variable of the student’s pre-study MAP math score. To account for the multiple session-level outcomes we examine, we also report Romano-Wolf adjusted p-values for our treatment estimates. See Supplementary Information section G for alternative model specifications.

While the ITT analysis captures the overall impact of the intervention, it ignores whether the tutor used Tutor CoPilot during a session. To disentangle the effect of merely having access to the tool from the effect of actually using it, we extended our analysis using a two-stage least squares (2SLS) regression and report those results in the Supplementary Information section H.

RQ2: Heterogeneity by Tutor Initial Effectiveness

We performed a heterogeneity analysis using the exit ticket passing rate as the primary outcome from our primary model (Equation). Specifically, we categorized the tutor’s quality rating and tutoring experience into tercile indicators and interacted these indicators with the treatment indicator to examine how the effect of Tutor CoPilot on students’ exit ticket passing rate varies based on the tutor’s initial effectiveness. This approach allowed us to assess whether more or less effective tutors benefit differently from using Tutor CoPilot in terms of improving student performance.

RQ3: Impact on Tutoring Strategies

We first defined the taxonomy of strategies based on Wang et al. (2024a) and salient response patterns in our data. We distinguish between higher-quality strategies that seek to probe and scaffold students’ problem-solving from lower-quality strategies that lead students directly to the solution or provide generic feedback. Although it is worth noting that the quality of strategies may vary by context, such as tutors sometimes giving away answers in pedagogically valuable ways (e.g., by prompting the student afterward to self-explain why that answer was correct), we do not observe tutors giving away answers in such pedagogically meaningful ways in our tutoring context, as they tend to employ scripted, pedagogically poor techniques.

Second, we trained machine learning classifiers to identify these strategies at scale. We did so by (i) randomly sampling 3,000 tutor messages, blind to study condition, (ii) prompting GPT-4 with our taxonomy of strategies to perform a first pass of labeling these examples, (iii) manually verifying and fixing labels, (iv) fine-tuning RoBERTa models on the labeled data with a class-balancing loss. Our classifiers achieved high performance, with F1-scores ranging between 0.65 and 0.90. Supplementary Information section M provides more details on this setup.

Finally, we used these classifiers to identify the strategies in our entire data of 350,000+ messages. We then estimated the impact of Tutor CoPilot access on the likelihood of utilizing each strategy by analyzing classifications of messages from treatment relative to control tutors. We used the following logit regression, where Cmst is a binary indicator equal to 1 if message m, sent by tutor t to student s, was identified as the strategy of interest, and Treatmentt is the indicator of tutor t treatment status. We clustered standard errors at the student-tutor pair to account for repeated interactions:

RQ4: Tutors’ Perception of Tutor CoPilot

To understand the treatment tutors’ experiences and gather feedback on Tutor CoPilot, we conducted interviews with 18 treatment tutors a week after the study concluded. These interviews explored their usage of Tutor CoPilot and how their use evolved over time. We report the qualitative findings and themes that emerged from these interviews. Additional details on the interview setup are available in the Supplementary Information section I.