On the Opportunities and Risks of Foundation Models

The field of Artificial Intelligence (AI) Safety concerns itself with potential accidents, hazards, and risks of advanced AI models, especially larger-scale risks to communities or societies. Current foundation models may be far from posing such risks; however, the breadth of their capabilities and potential applications is striking, and a clear shift from previous ML paradigms. While AI safety has historically occupied a more marginal position within AI research, the current transition towards foundation models and their corresponding generality offers an opportunity for AI safety researchers to revisit the core questions of the field in a new light and reassess their immediate or near-future relevance.

Traditional problems in AI safety

A major branch of AI safety research concerns the implications of advanced AI systems, including those that might match or exceed human performance across a broad class of cognitive tasks (Everitt et al., 2018). A central goal of safety research in this context is to mitigate large-scale risks posed by the development of advanced AI. These risks may be significantly more speculative than those considered in §5.2: misuse, §4.8: robustness, and §4.7: security; however, they are of far greater magnitude, and could at least in principle result from future, highly-capable systems. Of particular concern are global catastrophic risks: roughly, risks that are global or trans-generational in scope—causing death or otherwise significantly reducing the welfare of those affected (e.g., a nuclear war or rapid ecological collapse) (Bostrom and Cirkovic, 2011). What AI safety research amounts to, then, is a family of projects which aim to characterize what (if any) catastrophic risks are posed by the development of advanced AI, and develop plausible technical solutions for mitigating the probability or the severity of these risks. The best-case scenario from the point of view of AI safety is a solution to the control problem: how to develop an advanced AI system that enables us to reap the computational benefits of that system while at the same time leaving us with sufficient control such that the deployment of the system does not result in a global catastrophe (Bostrom and Cirkovic, 2011). However technical solutions are not sufficient to ensure safety: ensuring that safe algorithms are actually those implemented into real-world systems and that unsafe systems are not deployed may require additional sociotechnical measures and institutions.

Reinforcement Learning (RL), which studies decision-making agents optimized towards rewards, has been a dominant focus in AI safety for the past decade. What is at issue here is the difficulty of specifying and instantiating a reward function for the AI that aligns with human values, in the minimal sense of not posing a global catastrophic threat. While this problem, known as value alignment (Gabriel, 2020; Yudkowsky, 2016), may seem trivial at first glance, human values are diverse, amorphous, and challenging to capture quantitatively. Due to this, a salient concern is reward hacking, where the AI finds an unforeseen policy that maximizes a proxy reward for human wellbeing, but whose misspecification results in a significant harm. Many efforts to combat the value alignment problem have focused on maximizing corrigibility, which is when errors in the design of a system can be corrected once the system is running (Soares et al., 2015). This can be far from straightforward—in the RL context, an agent with a specified goal would be incentivized to prohibit attempts to alter that goal, as any attempt to alter that goal would likely be suboptimal for the goal’s realization (Omohundro, 2008).

However, pure RL is not the only theorized route to advanced AI. Foundation models can also be trained with simple (self-)supervised objectives like next-token prediction, yet can still be used in interactive and goal-directed ways, with or without additional RL training. Moreover, it appears that many of these methods may result in increased capabilities through straightforward scaling of compute, number of parameters, and dataset size (Hestness et al., 2017; Kaplan et al., 2020). What concepts like value alignment and corrigibility amount to in the broader context of foundation models differ in several respects to the pure RL case, and must accordingly be carefully theorized.

Current foundation models and AI safety

Many of these risks in the RL setting result from models optimized to carry out goals. However, a key challenge for AI safety research on recent foundation models is that goal-directed behavior may emerge despite not being explicitly optimized for (see also §4.2: training). As an example, large language models may be trained on corpora where agents use language in goal-directed ways, such as in persuasive text. To predict the next token well, a model may acquire a general capability to reason and produce arguments, which could emerge with suitable contexts. Foundation models trained on other kinds of human data may capture other kinds of goal-directed behavior present in the data; e.g., robotic agents trained to mimic humans in videos may attempt to punch or knock-out their human operators if their training data includes videos of boxing matches. Recent work has also attempted to directly train agents to produce goal-directed behavior; for example, the Decision Transformer trains a sequence model on trajectories prepended with their returns (Srivastava et al., 2019; Schmidhuber, 2019; Chen et al., 2021b). One can then generate high-return trajectories by “prompting” this model with a high return, which raises similar questions of reward hacking from the RL context.

However, a major aim of safety research on goal-directed models is to gain more principled control and explainability over the actions being pursued by the agent, as opposed to relying on inscrutable decisions from a blackbox neural network. This makes current foundation models an exciting avenue of study for AI safety research, as aligning them may be a useful precursor for aligning more advanced models (Christiano, 2016; Cotra, 2021; Kenton et al., 2021). One challenge is the misalignment between the foundation model’s training objective and the desired behavior; for example, a language model may be trained to predict the next word of all documents in the training corpus regardless of veracity, but users may want the model to only output true or helpful text (Tamkin et al., 2021a). One potential way to steer goal-directed agents towards desired behavior may be to train them with natural language descriptions of actions — this may enable steering them with language as well as enabling them to output interpretable language describing the task they "believe" they are performing, similar to methods for controllable generation and source attribution (e.g., Keskar et al., 2019, see also §2.3: robotics, §2.5: interaction, and §4.11: interpretability). However, further advances would be necessary to ensure the reliability and self-consistency of such models in the wild (§4.8: robustness), as well as gaining a more mechanistic understanding of how these models operate (Cammarata et al., 2020, also see §4.11: interpretability). And even if natural language-based control of future foundation models enables better task specification and monitoring, models may acquire deceptive or otherwise undesirable behavior from human data — identifying and neutralizing this behavior is another important direction for future study.

While the self-supervised objectives described in the previous paragraph train models to capture human behavior in the data, new training paradigms may produce goal-directed foundation models capable of carrying out a wide range of tasks in complex environments, and which exhibit capabilities superior to humans in different domains (see §4.2: training). For example, goal-directed foundation models may be trained in an open-ended self-play setting, similar to AlphaGo, or in vast multitask single-agent RL setups. This might lead to emergent capabilities that complicate efforts to get agents to carry out goals, especially if many agents are trained together in a rich world-simulator that encourages the development of skills like deception, misdirection, dissimulation, persuasion, and strategic planning. Aside from countering deceptive behavior, it also remains unclear how to effectively evaluate and control the behavior of very capable models, known as scalable oversight or alignment (Amodei et al., 2016; Leike et al., 2018); e.g., scoring novel reactions proposed by a chemical foundation model (see §4.4: evaluation). New human-in-the-loop approaches for training, steering, monitoring, and understanding these models are thus exciting future directions.

Finally, even before any of these more advanced capabilities emerge, an important research area for AI safety in the near term is characterizing and forecasting the capabilities of current self-supervised foundation models. There are three aspects which make this challenging. First, the generality of foundation models means that they can be applied to countless different kinds of applications in unexpected ways. Enumerating current and planned applications of foundation models is not sufficient to capture the full range of ways they could be used. Second, even within a particular application, model capabilities are emergent: they grow and change in unexpected ways as models scale. For example, the ability to control GPT-3 via “prompting" was an emergent phenomenon of which only the barest glimpses were evident in the smaller GPT-2 model (Radford et al., 2019; Brown et al., 2020). What the emergent properties of future foundation models will look like is unknown. Third, even within a particular application and scale, a model’s capabilities are not easy to characterize. For example, the ability of GPT-3 to perform addition improves dramatically once commas are added to the inputs (Branwen, 2020; Brockman, 2020). Similarly, small rewordings of prompts can have large impacts on task performance. Since the space of prompts is intractable to enumerate, it is challenging to definitely assert that any task is outside the reach of current prompt-based foundation models — this is a major challenge for reasoning about possible catastrophic risks from foundation models.

Potential catastrophic risks from future foundation models

The broad and quickly-growing capabilities of current models suggest the benefit of attempting to characterize possible catastrophic risks from more advanced systems. We see at least two ways in which advanced foundation models might contribute to such outcomes.

Catastrophic robustness failures

§4.8: robustness discusses how models may behave in unexpected or harmful ways when confronted with new kinds of data (Amodei et al., 2016; Yudkowsky et al., 2008). These failures may be especially consequential if foundation models are integrated into important systems that leverage foundation models’ ability to quickly adapt to many different tasks and situations. Failures could be catastrophic if they occur in warfare systems (resulting in unwanted discharge of weapons, possibly igniting a conflict), critical infrastructure (accidental destruction of critical energy or agricultural capabilities), or if they become essential to a large fraction of economic activity (whose unexpected failure could result in a sudden collapse in living standards and political instability; see also §5.5: economics). Indeed, the threat of catastrophic robustness failures is particularly pertinent for foundation models in contrast to other kinds of AI. This is because a foundation model consists of a single model that may be adapted for many different use cases, such that robustness failures derived from the statistical associations learned by the model could in principle manifest in a correlated way across several different domains. If the same foundation model is integrated into multiple critical functions, then lack of robustness in the model could lead to correlated failures that span multiple critical functions or failsafes.

Misspecified goals

The use of foundation models might increase the risks of optimizing misaligned yet easy-to-specify goals, often referred to as Goodhart’s Law (Kenton et al., 2021; Goodhart, 1984). A current-day example of these risks is the negative effects of some recommender systems (e.g., polarization, media addiction) which may optimize simple engagement metrics rather than a harder-to-measure combination of societal and consumer well-being (Burr et al., 2018; Milano et al., 2020). Future institutions may leverage uninterpretable foundation models to maximize simple measures such as profit or GDP, due to these models’ ability to adapt to the many different subproblems each of these metrics is dependent on. However, at larger scales optimizing for these proxy metrics instead of a more holistic goal designed for human welfare could inadvertently lead to environmental or geopolitical harms (Gabriel, 2020; Creel and Hellman, 2021).

Conclusion

In sum, we argue that current and potential future emergent properties of foundation models make them ripe objects of study for the field of AI safety. We encourage future work on characterizing and forecasting the exact capabilities and risks of foundation models; developing new methods to align foundation models to human values and desired goals; and for states, research labs, and businesses to coordinate on proactive measures to mitigate salient risks.