On the Opportunities and Risks of Foundation Models

Data is the lifeblood of foundation models; the training data of these models largely determines what these capabilities these models can acquire. The centrality of data is not unique to foundation models; recent calls for data-centric AI (Press, 2021; Ré, 2021) indicate the pervasive importance of managing, understanding, and documenting data used to train machine learning models. For foundation models specifically, the current modus operandi is for training data to be selected using unspecified or unclear principles with a general lack of transparency regarding the nature of training data. We believe an alternative approach is needed to re-imagine the data ecosystem surrounding foundation models: we draw upon work on data visualization and management to propose a data hub for foundation models. We articulate how this proposal relates to many of the relevant data-centric considerations for foundation models: selection, curation, documentation, access, visualization and inspection, quality assessment, and legal regulation.

§4.7: Security and privacy

Security and privacy for foundation models is largely uncharted at present. Fundamentally, foundation models are a high-leverage single point of failure, making them a prime target for attack: existing work demonstrates a variety of security vulnerabilities (e.g., adversarial triggers to generate undesirable outputs) or privacy risks (e.g., memorization of training data) for these models. Further, the generality of foundation models compounds these concerns, intensifying the risk for function creep or dual use (i.e., use for unintended purposes). For security, we view foundation models as akin to operating systems in traditional software systems; we discuss steps towards secure foundation models which, if achieved, would provide a strong abstraction layer to build upon for reliable ML applications. For privacy, by leveraging knowledge transfer from public data, foundation models may enable more sample efficient adaptation to sensitive data distributions, i.e., privacy-preserving applications may incur less degradation in accuracy when built using foundation models.

§4.8: Robustness to distribution shifts

A major limitation of standard machine learning is that it produces models that are not robust to distribution shifts, where the training distribution does not match the test distribution (for the downstream task). Existing work shows that adapting a foundation model trained on a broad range of unlabeled data improves the robustness of adapted models across a wide variety of shifts. This opens a new set of promising directions for improving training and adaptation of foundation models for robustness. However, we do not believe that foundation models are a panacea for robustness — challenges such as extrapolation across time and spurious correlations are not likely to be fully addressed.

§4.9: AI safety and alignment

Ensuring foundation models are reliable (§4.5: systems), robust (§4.8: robustness), and interpretable (§4.11: interpretability) is increasingly important when considering the potential real-world applications of these models. In addition to critical and immediate considerations, we also consider the relationship between foundation models and larger-scale risks, hazards, and harms that have the potential for increased relevance as model capabilities continue to advance. For example, we consider the importance of aligning foundation models such that they are not deployed with misspecified goals or values. We also discuss the relevance of forecasting the emergent behaviors of foundation models (e.g., the ability to deceive or plan strategically), which may complicate attempts to adapt them to particular tasks, and may require new approaches for interpretability (§4.11: interpretability) or evaluation (§4.4: evaluation).

§4.10: Theory

Learning theory provides a broad foundation for the variety of contexts encountered in applied machine learning; theory offers both understanding, principles, and guarantees to complement empirical findings. At present, the study of foundation models is largely empirical: the theory of standard supervised learning, while relatively mature, is inadequate to fully explain foundation models. Specifically, the discrepancy between the training phase and the adaptation phase within the foundation model regime pinpoints the insufficiency of existing theory, since these phases correspond to (potentially) completely different tasks and data distributions. Nevertheless, we endeavor that advances in theory to address this discrepancy, even in simple, limited settings, will provide useful insights.

§4.11: Interpretability

Interpretability provides clarity to foundation models: the opacity of the deep neural networks that underpin foundation models, alongside the expected ubiquity of foundation models, heightens the need to understand these models and their capabilities. Interpretability methods at present generally are designed for interpreting and explaining the behavior of task-specific models; the nature of foundation models (i.e., the wide array of tasks these models are beneficial for and the unexpected emergent properties they acquire) introduces new challenges for interpretability research. To frame the discussion of interpretability for foundation models, we propose the one model-many models paradigm, which aims to determine the extent to which the one model (the foundation model) and its many models (its adapted derivatives) share decision-making building blocks. In addition to interpreting the decision-making components involved, we further discuss explainability in the context of foundation models (e.g., the validity ofpost hoc explanations generated by models) as well as the mechanisms that drive model behavior (which may clarify the extent to which understanding foundation models can extend to understanding their adapted derivatives). Given the critical role we ascribe interpretability in the study of foundation models, we conclude with an assessment of the societal impact of interpretability and non-interpretability.

Overview of society

We believe the rapid development of foundation models, adapted and deployed to various applications, will have wide-ranging consequences on the health of societies. What makes these models so exciting and also so troubling is their task agnosticity. Societal impact is easier (but still non-trivial) to understand and reason about when we talk about specific systems deployed to users, but how can we take into account the societal impact of all possible systems and use cases when developing foundation models?

§5.1: Inequity and fairness

In many contexts, machine learning has been shown to contribute to, and potentially amplify, societal inequity. Foundation models may extend this trend, i.e., furthering the unjust treatment of people who have been historically discriminated against. However, understanding the relationship between inequity and foundation models requires reckoning with the abstraction of foundation models; foundation models are intermediary assets that are adapted for applications that impact users. Therefore, we delineate intrinsic biases, i.e., properties in foundation models that portend harm, and extrinsic harms, i.e., harms arising in the context of specific applications built using foundation models. We taxonomize various sources (e.g., training data, lack of diversity among foundation model developers, the broader sociotechnical context) that give rise to these biases and harms, emphasizing the importance, and technical difficulty, of source tracing to understand ethical and legal responsibility. We do not view unfairness as inevitable in the foundation model paradigm: to address unfair outcomes that arise from foundation models, we dually consider proactive interventions (e.g., technical methods like counterfactual data augmentation) and reactive recourse (e.g., mechanisms for feedback propagation and attribution of moral/legal responsibility).

§5.2: Misuse

We define foundation model misuse as the use of foundation models as they are technically intended (e.g., to generate language or video), but with the goal of causing societal harm (e.g., to generate disinformation, to develop deepfakes for harassment). We argue that advances in foundation models will result in higher-quality machine-generated content that will be easier to create and personalize for misuse purposes. For example, disinformation actors may use them to quickly generate collections of articles targeted across different demographic groups (e.g., nationality, political party, religion, etc.). While these new capabilities may limit existing human detection methods for harmful content (e.g., tracking similar text across different sources), foundation models may themselves provide promising potential as automated misuse detectors.

§5.3: Environment

Foundation models are the byproducts of computationally expensive training regimes, with the existing trajectory favoring even more intensive models; the energy required for this training coincides with the release of more carbon into the atmosphere and the degradation of the environment. At present, current discussion centers these enormous single-time training costs and the potential to amortize these costs across repeated use. We seek to clarify these discussions by identifying assumptions that shape the calculus of environmental impact for foundation models. Further, we envision that the ecosystem surrounding foundation models requires a multi-faceted approach: (a) more compute-efficient models, hardware, and energy grids all may mitigate the carbon burden of these models, (b) environmental cost should be a clear factor that informs how foundation models are evaluated (§4.4: evaluation), such that foundation models can be more comprehensively juxtaposed with more environment-friendly baselines, and (c) the cost-benefit analysis surrounding environmental impact necessitates greater documentation and measurement across the community.

§5.4: Legality

Foundation models rest on tenuous legal footings at present; how the law bears on both the development and use of these models is largely unclear. Legal and regulatory frameworks for foundation models specifically, alongside those for AI technology more generally, will be needed to influence, constrain, and even foster practices in research, development, and deployment. Centering on the legal landscape of the United States, where existing consideration of algorithmic tools remains broadly uncertain, we highlight the pertinent issues of liability for model predictions and protections from model behavior. With respect to both issues, we describe how legal standards will need to be advanced to address these given the intermediary status of foundation models (as opposed to that of user-facing task-specific models).

§5.5: Economics

Foundation models are likely to have substantial economic impact due to their novel capabilities and potential applications in a wide variety of industries and occupations. We consider the implications of the development and use of foundation models for the future of the US and global economy with a focus on productivity, wage inequality, and concentration of ownership.

§5.6: Ethics of scale

In addition to running the risk of increasing inequity, as discussed in §5.1: fairness, the widespread adoption of foundation models poses other ethical, political and social concerns. We discuss ethical issues related to the scale of application of foundation models, such as homogenization and the concentration of power, as well as the norms and release strategies appropriate to address them.