On the Opportunities and Risks of Foundation Models

Healthcare tasks (e.g., patient care via disease treatment) and biomedical research (e.g., scientific discovery of new therapies) require expert knowledge that is limited and expensive. Foundation models present clear opportunities in these domains due to the abundance of data across many modalities (e.g., images, text, molecules) to train foundation models, as well as the value of improved sample efficiency in adaptation due to the cost of expert time and knowledge. Further, foundation models may allow for improved interface design (§2.5: interaction) for both healthcare providers and patients to interact with AI systems, and their generative capabilities suggest potential for open-ended research problems like drug discovery. Simultaneously, they come with clear risks (e.g., exacerbating historical biases in medical datasets and trials). To responsibly unlock this potential requires engaging deeply with the sociotechnical matters of data sources and privacy as well as model interpretability and explainability, alongside effective regulation of the use of foundation models for both healthcare and biomedicine.

§3.2: Law

Legal applications require that attorneys read and produce long coherent narratives that incorporate shifting contexts and decipher ambiguous legal standards. Foundation models may provide benefits in this domain: ample data exists in the form of legal documents and their generative capabilities are well-suited to the many generative tasks required in law, but significant improvements are required for foundation models to be able to reliably reason over various sources of information to generate truthful long-form documents. As is the care in healthcare (§3.1: healthcare), the sample efficiency of adaptation for foundation models is of heightened value given the costs of expert time and knowledge in the legal domain, which may allow for the re-allocation of expertise towards pressing problems of justice and government service. The responsible development of foundation models for law will require specific consideration of privacy, and highlights core limitations of existing foundation models that will require fundamental advances with respect to provenance for their behavior and guarantees for the factuality of their generation.

§3.3: Education

Education is a complex and subtle domain; effective teaching involves reasoning about student cognition and should reflect the learning goals of students. The nature of foundation models presents promise here that has yet to be realized in the sphere of AI for education: while certain many streams of data in education are individually too limited to train foundation models, the ability to leverage relevant data from outside the domain (e.g., the Internet) and make use of data across multiple modalities (e.g., textbooks, mathematical formula, diagrams, video-based tutorials) jointly offers hope for foundation models that are broadly applicable to educational tasks. If foundation models lead to a significant improvement in education-relevant capabilities, there is clear potential for new applications that align with the open-ended generative (e.g., problem generation) and interactive (e.g., feedback to teachers) aspects of foundation models; the sample efficient adaptation of foundation models suggests greater ability for adaptive and personalized learning. In this event, renewed consideration is required of hallmarks of applying technology to education (e.g., student privacy), along with certain concerns becoming more critical (e.g., inequity in access to technology in education, technology-aided plagiarism).

Overview of technology

Now we discuss the technology behind building better model architectures, training and adaptation procedures, and of course scaling up the systems. One crucial but often overlooked topic is data — where does it come from and what is its composition? In addition, we want foundation models to be robust to distribution shifts and secure against attackers. Finally, we wish to understand why foundation models work from both a mathematical perspective as well as an empirical perspective.

§4.1: Modeling

What structural properties give rise to a foundation model? In the modeling section, we explore the underlying architectures behind foundation models and identify 5 key attributes. First, we start by discussing expressivity of the computational model — to capture and assimilate real-world information, and scalability — to adeptly handle large quantities of high-dimensional data. These properties are successfully realized by existing architectures such as the transformer network (Vaswani et al., 2017) that underpins most foundation models to date. We then proceed to attributes may be essential for the next generation of models, including: multimodallity — to consume, process and potentially produce content from different sources and domains, memory capacity — to effectively store and retrieve the acquired knowledge, and finally, compositionality, to foster successful generalization to novel settings and environments. We believe that realizing the full potential envisioned for foundation models will hinge on modelling advances to fulfill these desiderata.

§4.2: Training

Training objectives mathematically specify how models should learn and acquire capabilities from their training data. The current status quo for training foundation models involves modality-specific objectives (e.g., masked language modeling (Devlin et al., 2019) for text and SimCLR (Chen et al., 2020c) for images) that are often chosen heuristically. We envision that future training objectives for foundation models will reflect two changes: principled selection derived from systematic evidence and evaluation (§4.4: evaluation), and domain-generality to provide rich, scalable, and unified training signal across data sources and modalities. We also discuss important design trade-offs, including generative vs discriminative training, the choice of input data representation, and the potential of future training objectives that involve explicit representations of goals.

§4.3: Adaptation

Foundation models are intermediary assets; they are unfinished and generally should not be used directly, instead requiring adaptation for specific downstream tasks. The de facto approach for adaptation has been fine-tuning, with recent work suggesting that lightweight fine-tuning alternatives and prompting-based methods may achieve favorable accuracy-efficiency tradeoffs. Moving forward, we envision a more expansive view of adaptation that goes beyond just specializing foundation models to perform the task of interest: adaptation will alleviate deficiencies of stand-alone foundation models (e.g., temporal adaptation to reflect changes over time in the world) or introduce constraints (e.g., GDPR compliance relating to the right to be forgotten; §4.7: security); this broader perspective on adaptation coincides with a need for new evaluation protocols (§4.4: evaluation) that systematically evaluate adaptation methods while controlling for resources (e.g., runtime, memory) and access requirements involved in adaptation.

§4.4: Evaluation

Evaluation offers context to foundation models by providing a means to track progress, understand models, and document their capabilities and biases. Foundation models challenge the ability of standard evaluation paradigms in machine learning to achieve these goals since they are one step removed from specific tasks. To envision new paradigms in evaluation that suit foundation models, we discuss (a) evaluating foundation models directly to measure their inherent capabilities and inform how foundation models are trained, (b) evaluating task-specific models by controlling for adaptation resources and access, and (c) broader evaluation design to provide richer context beyond measures of accuracy (e.g., robustness (§4.8: robustness), fairness (§5.1: fairness), efficiency (§4.5: systems), environmental impact (§5.3: environment)). Reform of evaluation practices will allow for evaluation that adequately serves both the diverse goals and stakeholders involved in the foundation model paradigm.

§4.5: Systems

While the training data (§4.6: data) determines the theoretical information available for foundation models, and model architectures (§4.1: modeling) and training objectives (§4.2: training) determine how much of this information can be extracted, computer systems determine what is practically achievable. Systems are a key bottleneck for scaling in terms of data and model size, both of which appear to reliably track with improvements in capabilities. To ensure that we can train the next generation of foundation models efficiently (with respect to time and cost), we will require the co-design of algorithms, models, software, and hardware. This co-design is already starting to happen to in various forms, from carefully tuned parallelism strategies to new architectures such as retrieval-based and mixture-of-expert models. Beyond training, we consider what will be required to deploy applications on top of foundation models (e.g., efficient inference).