On the Opportunities and Risks of Foundation Models

This report investigates an emerging paradigm for building artificial intelligence (AI) systems based on a general class of models which we term foundation models. A foundation model is any model that is trained on broad data (generally using self-supervision at scale) that can be adapted (e.g., fine-tuned) to a wide range of downstream tasks; current examples include BERT (Devlin et al., 2019), GPT-3 (Brown et al., 2020), and CLIP (Radford et al., 2021). From a technological point of view, foundation models are not new — they are based on deep neural networks and self-supervised learning, both of which have existed for decades. However, the sheer scale and scope of foundation models from the last few years have stretched our imagination of what is possible; for example, GPT-3 has 175 billion parameters and can be adapted via natural language prompts to do a passable job on a wide range of tasks despite not being trained explicitly to do many of those tasks (Brown et al., 2020). At the same time, existing foundation models have the potential to accentuate harms, and their characteristics are in general poorly understood. Given their impending widespread deployment, they have become a topic of intense scrutiny (Bender et al., 2021).

Emergence and homogenization

The significance of foundation models can be summarized by two words: emergence and homogenization. Emergence means that the behavior of a system is implicitly induced rather than explicitly constructed; it is both the source of scientific excitement and anxiety about unanticipated consequences. Homogenization indicates the consolidation of methodologies for building machine learning systems across a wide range of applications; it provides strong leverage towards many tasks but also creates single points of failure. To better appreciate emergence and homogenization, let us reflect on their rise in AI research over the last 30 years.

Machine learning

Most AI systems today are powered by machine learning, where predictive models are trained on historical data and used to make future predictions. The rise of machine learning within AI started in the 1990s, representing a marked shift from the way AI systems were built previously: rather than specifying how to solve a task, a learning algorithm would induce it based on data — i.e., the how emerges from the dynamics of learning. Machine learning also represented a step towards homogenization: a wide range of applications could now be powered by a single generic learning algorithm such as logistic regression.

Despite the ubiquity of machine learning within AI, semantically complex tasks in natural language processing (NLP) and computer vision such as question answering or object recognition, where the inputs are sentences or images, still required domain experts to perform “feature engineering” — that is, writing domain-specific logic to convert raw data into higher-level features (e.g., SIFT (Lowe, 1999) in computer vision) that were more suitable for popular machine learning methods.

Deep learning

Around 2010, a revival of deep neural networks under the moniker of deep learning (LeCun et al., 2015) started gaining traction in the field of machine learning. Deep learning was fueled by larger datasets, more computation (notably, the availability of GPUs), and greater audacity. Deep neural networks would be trained on the raw inputs (e.g., pixels), and higher-level features would emerge through training (a process dubbed “representation learning”). This led to massive performance gains on standard benchmarks, for example, in the seminal work of AlexNet (Krizhevsky et al., 2012) on the ImageNet dataset (Deng et al., 2009). Deep learning also reflected a further shift towards homogenization: rather than having bespoke feature engineering pipelines for each application, the same deep neural network architecture could be used for many applications.

Foundation models

Foundation models have taken shape most strongly in NLP, so we focus our story there for the moment. That said, much as deep learning was popularized in computer vision but exists beyond it, we understand foundation models as a general paradigm of AI, rather than specific to NLP in any way. By the end of 2018, the field of NLP was about to undergo another seismic change, marking the beginning of the era of foundation models. On a technical level, foundation models are enabled by transfer learning (Thrun, 1998) and scale. The idea of transfer learning is to take the “knowledge” learned from one task (e.g., object recognition in images) and apply it to another task (e.g., activity recognition in videos). Within deep learning, pretraining is the dominant approach to transfer learning: a model is trained on a surrogate task (often just as a means to an end) and then adapted to the downstream task of interest via fine-tuning.

Transfer learning is what makes foundation models possible, but scale is what makes them powerful. Scale required three ingredients: (i) improvements in computer hardware — e.g., GPU throughput and memory have increased 10× over the last four years (§4.5: systems); (ii) the development of the Transformer model architecture (Vaswani et al., 2017) that leverages the parallelism of the hardware to train much more expressive models than before (§4.1: modeling); and (iii) the availability of much more training data.

The importance of the availability of data and the ability to harness it cannot be underestimated. Transfer learning with annotated datasets has been common practice for at least a decade, for example, pretraining on the ImageNet dataset (Deng et al., 2009) for image classification in the computer vision community. However, the non-trivial cost of annotation imposes a practical limit on the benefits of pretraining.

In self-supervised learning on the other hand, the pretraining task is derived automatically from unannotated data. For example, the masked language modeling task used to train BERT (Devlin et al., 2019) is to predict a missing word in a sentence given its surrounding context (e.g., I like sprouts). Self-supervised tasks are not only more scalable, only depending on unlabeled data, but they are designed to force the model to predict parts of the inputs, making them richer and potentially more useful than models trained on a more limited label space.

There had been considerable progress in self-supervised learning dating back to word embeddings (Turian et al., 2010; Mikolov et al., 2013; Pennington et al., 2014), which associated each word with a context-independent vector, provided the basis for a wide range of NLP models. Shortly thereafter, self-supervised learning based on autoregressive language modeling (predict the next word given the previous words) (Dai and Le, 2015) became popular. This produced models that represented words in context, such as GPT (Radford et al., 2018), ELMo (Peters et al., 2018), and ULMFiT (Howard and Ruder, 2018).

The next wave of developments in self-supervised learning — BERT (Devlin et al., 2019) GPT-2 (Radford et al., 2019), RoBERTa (Liu et al., 2019), T5 (Raffel et al., 2019), BART (Lewis et al., 2020a) — quickly followed, embracing the Transformer architecture, incorporating more powerful deep bidirectional encoders of sentences, and scaling up to larger models and datasets.

While one can view this last wave of technical developments purely through the lens of self-supervised learning, there was a sociological inflection point around the introduction of BERT. Before 2019, self-supervised learning with language models was essentially a subarea in NLP, which progressed in parallel to other developments in NLP. After 2019, self-supervised learning with language models became more of a substrate of NLP, as using BERT has become the norm. The acceptance that a single model could be useful for such a wide range of tasks marks the beginning of the era of foundation models.

Foundation models have led to an unprecedented level of homogenization: Almost all state-of-the-art NLP models are now adapted from one of a few foundation models, such as BERT, RoBERTa, BART, T5, etc. While this homogenization produces extremely high leverage (any improvements in the foundation models can lead to immediate benefits across all of NLP), it is also a liability; all AI systems might inherit the same problematic biases of a few foundation models (Bolukbasi et al., 2016; Caliskan et al., 2017; Abid et al., 2021, inter alia)) — see §5.1: fairness, §5.6: ethics for further discussion.

We are also beginning to see a homogenization across research communities. For example, similar Transformer-based sequence modeling approaches are now applied to text (Devlin et al., 2019; Radford et al., 2019; Raffel et al., 2019), images (Dosovitskiy et al., 2020; Chen et al., 2020d), speech (Liu et al., 2020d), tabular data (Yin et al., 2020), protein sequences (Rives et al., 2021), organic molecules (Rothchild et al., 2021), and reinforcement learning (Chen et al., 2021b; Janner et al., 2021). These examples point to a possible future where we have a unified set of tools for developing foundation models across a wide range of modalities (Tamkin et al., 2021b).

Besides the homogenization of approaches, we also see the homogenization of actual models across research communities in the form of multimodal models — e.g., foundation models trained on language and vision data (Luo et al., 2020; Kim et al., 2021a; Cho et al., 2021; Ramesh et al., 2021; Radford et al., 2021). Data is naturally multimodal in some domains—e.g., medical images, structured data, clinical text in healthcare (§3.1: healthcare). Thus, multimodal foundation models are a natural way of fusing all the relevant information about a domain, and adapting to tasks that also span multiple modes (Figure 2).

Foundation models have also led to surprising emergence which results from scale. For example, GPT-3 (Brown et al., 2020), with 175 billion parameters compared to GPT-2’s 1.5 billion, permits in-context learning, in which the language model can be adapted to a downstream task simply by providing it with a prompt (a natural language description of the task), an emergent property that was neither specifically trained for nor anticipated to arise.

Homogenization and emergence interact in a potentially unsettling way. Homogenization could potentially provide enormous gains for many domains where task-specific data is quite limited — see the opportunities presented in several such domains (e.g., §3.1: healthcare, §3.2: law, §3.3: education); on the other hand, any flaws in the model are blindly inherited by all adapted models (§5.1: fairness, §5.6: ethics). Since the power of foundation models comes from their emergent qualities rather than their explicit construction, existing foundation models are hard to understand (§4.4: evaluation, §4.10: theory, §4.11: interpretability) and they have unexpected failure modes (§4.7: security, §4.8: robustness). Since emergence generates substantial uncertainty over the capabilities and flaws of foundation models, aggressive homogenization through these models is risky business. Derisking is the central challenge in the further development of foundation models from an ethical (§5.6: ethics) and AI safety (§4.9: ai-safety) perspective.

Naming

We introduce the term foundation models to fill a void in describing the paradigm shift we are witnessing; we briefly recount some of our reasoning for this decision. Existing terms (e.g., pretrained model, self-supervised model) partially capture the technical dimension of these models, but fail to capture the significance of the paradigm shift in an accessible manner for those beyond machine learning. In particular, foundation model designates a model class that are distinctive in their sociological impact and how they have conferred a broad shift in AI research and deployment. In contrast, forms of pretraining and self-supervision that technically foreshadowed foundation models fail to clarify the shift in practices we hope to highlight.

Additionally, while many of the iconic foundation models at the time of writing are language models, the term language model is simply too narrow for our purpose: as we describe, the scope of foundation models goes well beyond language. We also considered terms such as general-purpose model and multi-purpose model that capture the important aspect that these models can serve multiple downstream tasks, but both fail to capture their unfinished character and the need for adaptation. Terms such as task-agnostic model would capture the manner of training, but fail to capture the significant implication to downstream applications.

We chose the new term foundation models to identify the models and the emerging paradigm that are the subject of this report. In particular, the word “foundation” specifies the role these models play: a foundation model is itself incomplete but serves as the common basis from which many task-specific models are built via adaptation. We also chose the term “foundation" to connote the significance of architectural stability, safety, and security: poorly-constructed foundations are a recipe for disaster and well-executed foundations are a reliable bedrock for future applications. At present, we emphasize that we do not fully understand the nature or quality of the foundation that foundation models provide; we cannot characterize whether the foundation is trustworthy or not. Thus, this is a critical problem for researchers, foundation model providers, application developers who rely on foundation models, policymakers, and society at large to address.