On the Opportunities and Risks of Foundation Models

Training objectives are mathematical functions describing how to transform a model architecture and large amount of broad data into a foundation model. For example, GPT-3 was trained with a language modeling objective, which rewards the model for predicting the next word correctly (Shannon, 1948). We begin by laying out some goals of these training approaches, describe important design trade-offs in current approaches, and outline important goals for the path ahead.

Goals of training objectives

Here we outline some key goals for training algorithms in light of the recent rapid progress in these methods and models.

Leveraging broad data

The rise of self-supervised learning algorithms has unlocked the power of internet-scale datasets which would be intractable to annotate by hand. This kind of broad data comes in many forms, including images, audio recordings, and video (§2.2: vision); robotic and sensor data (§2.3: robotics); and text, either in isolation or paired with other modalities like images (§2.1: language). Because this data lacks external annotations, a major focus for researchers is designing bespoke self-supervised algorithms that leverage the unique structure within each kind of data to produce a training signal for a foundation model.

Domain completeness

An important goal for foundation model training algorithms is to be domain complete, in the sense that solving the training task requires capabilities that are broadly useful for downstream tasks in the domain (see §2.1: language, §2.2: vision, §2.3: robotics). This property is crucial for the generality of a foundation model. For example, language modeling may require models to acquire capabilities as wide-ranging as coreference, sentiment and translation as the model learns to predict the next word in a document. In contrast, a supervised learning task like sentiment classification may lead to a more narrow set of capabilities (see §2.1: language). As important as this quality is, it is not obvious a priori what tasks will result in a domain complete capabilities, or even how to evaluate the full breadth of a model’s capabilities (see §4.4: evaluation and §4.10: theory).

Scaling and compute efficiency

Procedures for training foundation models must reliably convert data, a model architecture, and compute into a broadly capable model. To maximize the capability of a foundation model, we can identify the bottlenecks to this process and propose new training algorithms which remove them. The rise of self-supervised algorithms has made model size and compute resources increasingly salient bottlenecks (Kaplan et al., 2020; Henighan et al., 2020), leading to a shift where models are evaluated not solely on their capabilities but rather on the amount and kind of compute needed to reach those capabilities (§4.4: evaluation). The efficiency of training objectives can vary tremendously, laying in sharp relief how important the design of a training approach is to the emergence of powerful capabilities given a fixed compute budget. Thus, a major goal for training researchers is to design training objectives with a richer training signal, resulting in models which learn faster and attain stronger capabilities. One force aiding this development is the surprising predictability of how capabilities scale with different kinds of architectures, data sizes, and compute (Hestness et al., 2017; Kaplan et al., 2020), a striking phenomenon which enables model developers to make choices based on clearer trends instead of more costly random searches.

Design trade-offs in current SSL methods

Current self-supervised learning (SSL) methods for training foundation models are diverse, but what unites them is that they produce prediction problems from unlabeled data without the need for human annotators. SSL objectives manufacture a rich training signal from this data through carefully-designed constraints, either on the data itself (e.g., redacting or noising) or on the way the model is able to represent or process the data (e.g., latent bottlenecks). At some level, these constraints “bake in" the kinds of capabilities desired when adapting models to downstream tasks (§4.3: adaptation).

Here, we describe three important design choices that current models explore, along with their respective tradeoffs in terms of their resulting capabilities.

At what level of abstraction should we model?

A fundamental question is what the input representation of a foundation model should be. One option is to model the input at the level of raw bytes. However, this high dimensionality may cause the model to focus on predicting less semantic aspects of the input, slowing the rate at which it acquires more generally-useful capabilities. These approaches also become intractable when training models like transformers (Vaswani et al., 2017) whose compute costs grow quadratically with the input size. Another option is to use domain knowledge to reduce the input space of a model — such strategies include patch embeddings (Dosovitskiy et al., 2020) as well as fixed or learned tokenization (Schuster and Nakajima, 2012; Sennrich et al., 2016; Kudo and Richardson, 2018; van den Oord et al., 2017; Ramesh et al., 2021). These methods may alleviate some challenges facing generative approaches, but have the trade-off that they may jettison possibly-useful information in the input. The choice of a continuous vs discrete input also has trade-offs for adaptation (§4.3: adaptation); more work is needed to capture the benefits of both approaches.

Generative vs discriminative models

Generative training approaches are conceptually elegant yet powerful — they train models to learn joint or conditional distributions over training inputs. Two major families of generative foundation models include autoregressive foundation models (van den Oord et al., 2016; Radford and Narasimhan, 2018; Chen et al., 2020d; Yang et al., 2019; Ramesh et al., 2021), which generate inputs piece by piece, and denoising foundation models (Devlin et al., 2019; Raffel et al., 2019) which corrupt and then recover the inputs. The specific kind of generation performed in the training process determines what kind of interactivity is available during adaptation (see §2.5: interaction and §4.3: adaptation), and future models may enable an even richer set of interactions.

While generative training approaches have their benefits, several discriminative approaches have also recently gained traction. These methods do not enable generation-based interaction, yet they may enable more efficient learning for classification- or regression-based tasks in high-dimensional continuous settings like images, audio, and video. Most of these methods output vectors for (parts of) inputs, which are trained to be similar for different “views” of an input (Wu et al., 2018; van den Oord et al., 2018; Chen et al., 2020c; He et al., 2020; Grill et al., 2020; Caron et al., 2021; Zhang et al., 2020a; Radford et al., 2021) or used to predict whether parts of inputs are real or fake (Clark et al., 2020; Iida et al., 2021). Better understanding the trade-offs between generative and discriminative training, as well as capturing the best of both approaches, remain interesting avenues for future study.

Capturing multimodal relationships

Another increasingly important research area is capturing the relationships between multiple kinds of data. What this means may differ based on the context and the goals of a modeler. For example, CLIP (Radford et al., 2021) and ViLBERT (Lu et al., 2019a) are both multimodal vision-language, but differ in the precise way they are multimodal. The former encodes images and text separately into vectors, enabling users who have examples from a single modality to retrieve, score, or classify examples from the other modality. The second processes images and text jointly at an early stage of the model, enabling downstream applications like visual question answering where reasoning over pairs of related images and text (e.g., images and questions about them) are provided. Multimodal foundation models remain a nascent research area; much is still unexplored about the different ways a model can be multimodal as well as better understanding the capabilities these additional modalities bring.

Paths forward

We close with some important goals for the future of foundation model training.

Out-of-the-box SSL

Right now, SSL objectives are highly domain-specific: different methods currently prevail in natural language processing, computer vision, and speech processing. This has two major disadvantages: First, these different techniques make it challenging to grasp the common threads and scientific principles underlying why each of these methods work. Second, this domain-specificity requires developing new foundation model training methods from scratch for each new field, including medical, scientific, and new multimodal settings. A more general objective for efficiently training foundation models on any kind of data would represent a significant milestone for the foundation model training community (Tamkin et al., 2021b).

Obtaining a rich training signal

It is clear that not all training objectives are made equal — some are radically more efficient than others, translating into far more capable foundation models for a given compute budget. Are there training methods orders of magnitude more efficient than those currently known? If so, how can we find them? These investigations will be shaped by many forces, including what future software and hardware advances (§4.5: systems) make possible. We also need not view data (§4.6: data) and training algorithms as independent factors: not only does the quality and availability of the data influence the training signal, but the training algorithm itself could adaptively seek out or construct richer training examples as the model improves to accelerate learning (Tamkin et al., 2021c).

Goal-directed training of foundation models

Adaptation methods such as prompting (§4.3: adaptation) draw on emergent properties that result almost as an afterthought of training. Can we train foundation models where the ability to understand and reliably carry out goals in a complex world is part of the model’s training objective? A focus on developing general capabilities distinguishes this direction from the goal of adapting an existing foundation model to a specific task via reinforcement learning (e.g., Stiennon et al. (2020)). Instead, one might imagine more sophisticated versions of current methods which acquire a diverse range of real-world capabilities from raw online (Klyubin et al., 2005; Singh et al., 2005; Salge et al., 2013; Shakir Mohamed, 2015; Florensa et al., 2017; Pathak et al., 2017; Haber et al., 2018) or offline (Precup et al., 2000; Lange et al., 2012; Ajay et al., 2021; Yang and Nachum, 2021; Schwarzer et al., 2021) interactions, without the need for human annotations or task construction. Such methods might use techniques quite similar to existing SSL algorithms: e.g., training sequence models in goal-directed contexts where they can be directly asked to carry out certain tasks via conditioning (e.g., UDRL (Schmidhuber, 2019; Srivastava et al., 2019) or Decision Transformer (Chen et al., 2021b); also see §2.3: robotics). The complex behaviors that have already emerged in simple interactive environments (Baker et al., 2020) suggest multitask, multiagent, and multimodal goal-directed training of foundation models as an interesting avenue for future study.