On the Opportunities and Risks of Foundation Models

In the year 2000, the largest gathering of world leaders convened at the United Nations Millennial Summit to reflect on an ideal vision for the future. Delegates concluded that a primary focus should be education, declaring it “a foundation for human fulfillment, peace, sustainable development, economic growth, decent work, gender equality and responsible global citizenship." This discussion was ultimately recodified into the United Nations Sustainable Development goal to “ensure inclusive and quality education for all and promote lifelong learning" (United Nations General Assembly, 2015). However, providing high quality, inclusive education at a large scale poses difficult societal and economic challenges. The price of education per student is growing faster than economy-wide costs (Bowen, 2012), limiting the resources available to support student learning. In the United States, one symptom is that private education debt held by students has reached $1.6 trillion, surpassing total credit card debt (Friedman, 2020). Considering the rising need to provide adult retraining, the gap between the demand for education and our ability to provide it is alarmingly large and has concerning achievement disparities across protected demographics.

With the advent of the digital age and the rapid growth in digital learning, computational approaches to education have shown promise in increasing the effectiveness of learners and teachers. Several core directions have emerged as potentially impactful applications of AI for education (Woolf et al., 2013), such as systems that can provide meaningful feedback to students (Malik et al., 2021), help teachers improve (Jensen et al., 2020; Demszky et al., 2021; Suresh et al., 2021), or even create personalised and adaptive learning experiences that tailor the learning process to individual students’ needs and dispositions (Connor, 2019).

Despite this potential, building technical solutions to effectively scale inclusively and quality of education has proven to be exceptionally difficult. One particular challenge is that existing work has focused on custom solutions to highly specific tasks for which large amounts of training data has to be collected from scratch. Due to the difficulty and cost of creating large datasets, using this approach to solve every educational task independently is fundamentally limited. Instead, is it possible to create general-purpose approaches that are reusable across various tasks and subjects?

Foundation models have already started to boost the performance of some specific flagship tasks in education. Recent examples include using MathBERT (Shen et al., 2021b) to power “knowledge tracing" — the challenge of tracking a student’s understanding over time given their past responses — and the “feedback challenge", where an algorithm has to interpret a student’s answer to a structured open-ended task, such as a coding question (Wu et al., 2021e). Can foundation models lead to even more transformative changes in this domain? And what are the known and imagined risks of foundation models applied to education? In this section, we first frame the conversation around the ethical considerations. We then ground our discussion in two concrete tasks: (1) understanding student misconceptions, and (2) improving student understanding through instruction.

Important concerns for centering foundation models in education research

The future of AI for education is exciting, especially in the context of foundation models. However, we caution the reader to be especially thoughtful about the impact of any AI research applied to education. The goal of education are deeply interwoven with complex, long term social impact. While we actively work to improve digital education, it is imperative that we put in substantial thought to try and imagine the complexities of any disruption in this space (Piech and Einstein, 2020). Ethical challenges range from issues such as data bias, legal constraints, and the impact of digital socialization. These issues are not unique to foundation models, but they are worth reflecting on regularly as research makes substantial progress in AI for education. Reflection on impact is especially important when research starts by asking “what can new AI technology afford?"

Many of the issues in §5.6: ethics apply to education. For example, as in many other domains, small biases in foundation model training data could be hard to track down (Dixon et al., 2018; Bolukbasi et al., 2016), but have important implications for equity of educational access. Moreover, these systems may experience a high degree of “feedback", where the collected data continually reinforces the model’s decisions. This issue of bias goes beyond what data is collected and includes concerns over the applications that researchers choose to work on. Below, we discuss other education-specific issues. Many of the issues revolve around the question: “who benefits?" and for whom is new technology created?

Removing teachers from the loop

One of the goals of digital education, especially based on AI, is to increase the productivity of the learning experience so that more learning happens per unit time or unit cost. One can imagine that decision makers could use this increased productivity to remove human teachers from the loop. The long term implications of such decisions are hard to know a priori. Could interacting with an education system optimized to maximize “learning” have adverse effects on socioemotional skill development? Could it create fewer opportunities for interacting with others? Loneliness is on the rise in younger generations (Cigna, 2018), and teachers are a modulating force for pressures that AI researchers might not envision.

Was this work done by the learner or a foundation model?

Another challenge is how to effectively teach students who have access to foundation-model-based tools. For example, it will be much more complex for teachers to understand the extent of a student’s contribution if the student worked together with a powerful generative model, or to regulate ineffective collaborations and detect plagiarism. Visual Studio has recently released GitHub CoPilot, an AI pair-programmer built upon GPT-3 (Chen et al., 2021f). How will this change computer science education? Many challenges for beginner programmers might be trivial to CoPilot or its technical successors, which could undermine the learning experience for novices. It would be instructive to study other examples of technological advances that disrupted education for certain subjects, such as calculators in math classrooms and Google Translate in language courses, both of which now coexist with traditional instruction.

Privacy and security

One important ethical issue in the use of AI in education is highlighted by the strict legal guidelines concerning privacy in student work. For example, in the United States, student information is protected by the Family Education Rights and Privacy Act (FERPA). These laws and regulations are especially important for children under 13, who have their data privacy and security additionally protected by the Children’s Online Privacy Protection Act. Among other things, FERPA limits teachers from sharing personally identifiable student work. This could directly impact initiatives to share data used both for training and for evaluating foundation models. Moreover, there is an open question as to whether the weights of a foundation model could somehow leak the (possibly private) data it was trained upon (Nasr et al., 2018; Song et al., 2017). These issues, and their corresponding approaches, are similar to the challenges described in §3.1: healthcare.

This list is not exhaustive and the ethical considerations will vary from project to project.

Foundation models of student thought

When building AI tools for inclusive, and joyful education, there are many tasks where foundation models could be useful. Many of those tasks require us to first understand the learners whom we are trying to help, especially in the context of open ended work.

What would it take for a foundation model to be able to reason about student understanding? It is easy to imagine a foundation model which has been adapted to answer a math question correctly, but it is less clear how to build a model that can diagnose mistakes in student understanding based on the student’s answers. To explore this theme, we consider the case study of providing feedback to students who are working on open-ended tasks such as writing a short paragraph, drawing a physics diagram, or writing code. This “feedback challenge” exemplifies how foundation models can be helpful off-the-shelf for learners, and also demonstrates open areas for foundation model research.

To effectively provide feedback to students, two central capabilities are required: (1) understanding the subject matter of the task (e.g., physics or coding), and (2) the diagnostic ability to “notice": a technical term in education for inferring why a student made a mistake. For typical student interactions in a typical classroom, there is not enough data for an AI model to learn, from scratch, both of these central capabilities. Even for massive courses with millions of students, supervised algorithms barely understand the complex student reasoning behind even short, four-line programs (Malik et al., 2021). As such, the feedback task inherently requires a transfer of understanding from external data and experience.

Foundation models, as they currently exist, are directly helpful for the first of these capabilities: understanding a specific subject matter. For example, when learning to provide feedback on short programming questions, a foundation model such as GPT-3 can efficiently understand what fluent code looks like with a few examples. Some research in this direction has already started exploring foundation models that can quickly adapt to questions in new subject matter domains (Wu et al., 2021e; Condor et al., 2021). Similarly, foundation models could also integrate multiple modes of information such as the text of a task’s prompt, diagrams in the question, or even the content of a grading rubric provided to teaching assistants. This unified representational ability can help foundation models comprehend a subject matter through richer sources of information. As a concrete case study, many of these insights were leveraged as core components of an algorithm which was able to grade an introductory Computer Science midterm at Stanford University, with the same effectiveness as human teaching assistants (Wu et al., 2021e). In this case, subject matter encoding was built on a foundation model that had been adapted on GitHub code and a corresponding small dataset for each question’s subject matter. In general, we can imagine leveraging various sources of data to adapt foundation models to different subject matter. For example, math adaptation could use mathematical websites or textbooks (Shen et al., 2021b) or historical student answers on platforms such as Gradescope; spoken language understanding could leverage radio archives or podcasts; and domains like creative writing could look to large digital archives like Project Gutenberg.

In contrast to subject matter, adapting a foundation model to the task of mapping observed mistakes to flaws in a student’s thought processes is much less well-explored. The ability for an instructor to ‘‘notice’’ the reasons behind why a student makes a specific mistake is a critical component of the feedback challenge. Imagine, for example, a student learning two digit addition who answers the question ‘‘what is 26 + 19?" with the response ‘‘315." Take a moment and try to guess why they gave that answer and what misconceptions they have.. This ability to notice could be posed as an adaptation task for foundation models (§4.3: adaptation) or perhaps even as a reasoning task (§2.4: reasoning).

While difficult, training an AI system to notice is an achievable goal. Across classrooms, and across learning tasks in a given domain, there are generalizable patterns in how students arrive at their answers. The labeled data that can directly be used for this adaptation task, such as instructor-written feedback to student work in (Wu et al., 2021e), are often held privately by instructors in disparate datasets. However, publicly accessible data, such as StackOverflow interactions, might also be creatively used to adapt a foundation model to notice. Some research has also explored effective ways of extracting, from instructors, generative descriptions of how students make mistakes (Malik et al., 2021; Gulwani and Singh, 2013) — these hand-written generative models could also be used to generate adaptation data to help foundation models diagnose student mistakes.

Foundation models for instruction

Reasoning about student understanding is an essential step towards a second objective: provide inclusive, high quality instruction. Computational approaches to instruction focus on different tasks like content personalization (Connor, 2019), question generation (Guo et al., 2016; Willis et al., 2019; Srivastava and Goodman, 2021), adaptive curriculum design (Mandel et al., 2014; Doroudi et al., 2017), and predicting instructor intervention (Chandrasekaran and Kan, 2019; Alrajhi et al., 2021). In this subsection, we discuss how foundation models could be useful in the act of teaching students.

Since effective teaching requires reasoning about student understanding, the previous discussions on understanding subject matter and “noticing” are extremely relevant. However, providing effective instruction requires an additional capability: that of understanding pedagogy (McKenzie, 2003). This encapsulates an effective understanding of techniques to guide a student, such as asking Socratic questions or providing analogies/contrasting cases; using encouraging or supportive language; tailoring the difficulty of questions to the student; and generating examples that are relevant to a student’s interests and background.

How can foundation models be adapted to understand good pedagogy for instruction? One idea is to consider adaptation using data source where instruction is the primary role. For example, data from question answering forums like StackOverflow could potentially be used to build a tutor which can parrot common Socratic questions. Similarly, a foundation model adapted on encyclopedias such as Wikipedia might be able to give answers to student questions which are (often) factually correct. There are also public data sources like textbooks, lecture videos, lesson plans, and graded feedback that collectively contain important pedagogical behaviours which could be adapted by foundation models (Figure 15).

Another adaptation challenge for instruction based on foundation model is to learn how to speak to students like teachers. The language used by teachers is often different from the language used by the general population. Teachers are ideally trained to speak to students with respect and in a way that intentionally helps them form a positive identity with the subject being learned (Truax, 2018). Cautionary examples like Microsoft’s 2016 Twitter bot “Tay," a chatbot that started generating hate speech within 24 hours of being deployed live, show us the importance of explicitly accounting for this factor in education. To train a language model which is more heavily influenced by professional teachers in classrooms, we could perhaps adapt foundation models to data sources like lecture videos or recorded office hour videos.

The adaptation problem above is compounded by the fact that different education contexts vary significantly in the kind of language that would be appropriate: for example, effective instruction in a 5th-grade science class would look quite different from that in a college physics class, much less a college literature class. This presents technical challenges beyond what would be faced in typical NLP domain shift settings (e.g., question answering based on news articles vs. Reddit posts), as the foundation model would need to be fluidly adaptable in terms of its tone and language, and not just the factual content that it generates.

Beyond sound pedagogical techniques and instructional language, how might foundation models provide even more insightful forms of instruction? §2.1: language of this paper highlights the fact that remarkably complex language can be acquired by babies in a short amount of time. As the authors point out, a salient difference between foundation model training and human language acquisition is that “human language is grounded to the real world: for example, a baby’s caretakers point to objects while they talk about them." This same insight can also inspire ideas as to how foundation models can be used for generative education. Humans seem to learn well when presented with real-world analogies and contrasts which may be cross-cutting between their current context and past experiences. For example, when teaching sign language, an instructor might use an analogy such as "the hand shapes for the word ‘morning’ looks like the sun rising" or note that “the hand shape you just made look very similar to another word, so let us focus on the differences." As another example, when teaching Swahili to a learner who already knows Arabic and English, an instructor could point out that the Swahili word for 8 (pronounced nane) is a “false friend” that is phonetically similar to English word for 9 (pronounced nine). Foundation models that can integrate multi-modal data have the potential to make these kinds of rich analogies and comparisons that are typical in childhood language learning (Figure 16).