On the Opportunities and Risks of Foundation Models

Foundation models are scientifically interesting due to their impressive performance and capabilities, but what makes them critical to study is the fact that they are quickly being integrated into real-world deployments of AI systems with far-reaching consequences on people. For example, Google search, which boasts 4 billion users, now depends on foundation models like BERT (Devlin et al., 2019) as one of its signals.

We must thus pause and ask: What is the nature of this social impact? In this report, we address many aspects of this question: the potential exacerbation of social inequities (§5.1: fairness), the economic impact due to increased capabilities (§5.5: economics), the environmental impact due to increased computation demands (§5.3: environment), potential concerns of amplifying disinformation (§5.2: misuse), legal ramifications due to powerful generative capabilities (§5.4: legality), ethical issues resulting from homogenization, and the broader political economy in which foundation models are developed and deployed (§5.6: ethics). Given the protean nature of foundation models and their unmapped capabilities, how can we responsibly anticipate and address the ethical and societal considerations they raise? A recurring theme is that it is easier to reason about the social impact of specific systems deployed to specific users than it is to reason about the social impact of foundation models, which could be adapted to any number of unforeseen downstream systems.

Before attempting to answer these questions, we need to lay some groundwork. First, let us distinguish between research on foundation models and deployment of foundation models. Most of what is publicly known is foundation models research --- through academic papers, demonstrations, and progress on leaderboards. While the production of knowledge can play a vital role in shaping the future, the direct social impact is through the actual deployment of these models, which is governed by proprietary practices on often private data. Sometimes the deployment is through new products --- e.g., GitHub’s Copilot based on OpenAI’s Codex model (Chen et al., 2021f), but often, it is through upgrades to existing products (e.g., Google search using BERT). Research models are often not extensively tested and might have unknown failure modes; warning labels should be placed on research models that are not fit to deploy. On the other hand, deployed foundation models that actually affect people’s lives should be subject to much more rigorous testing and auditing.

To further understand the research and deployment of foundation models, we must zoom out and consider the full ecosystem that these foundation models inhabit, from data creation to actual deployment. It is important to note that the foundation model is only one component (though an increasingly important component) of an AI system. Simplifying, we can think about the ecosystem of a foundation model in terms of sequence of stages, extending the training and adaptation stages from before. Appropriately, as we’re interested in social impact, people occupy both ends of the pipeline. This ecosystem view allows us to see that different questions about foundation models (e.g., whether a foundation model is ethical) should actually be answered with respect to different stages.

• (1) Data creation: Data creation is fundamentally a human-centric process: all data is created by people and most data is at least implicitly about people. Sometimes data is created by people for other people in the form of emails, articles, photos, etc., and sometimes it is a measurement of people (e.g., genomic data) or a measurement of the environment people live in (e.g., satellite images). It is important to note that all data has an owner and is created with a purpose (where that purpose may or may not include training a foundation model).

• (2) Data curation: Data is then curated into datasets. There is no single natural distribution of data; even the most permissive Internet crawl requires some selection and post-filtering. Ensuring data relevance and quality while respecting legal and ethical constraints is critical but challenging. While this is recognized in industry, it is underappreciated in AI research (§4.6: data).

• (3) Training: Training foundation models on these curated datasets is the celebrated centerpiece in AI research, though it is only one of many stages.

• (4) Adaptation: In the context of machine learning research, adaptation is about creating a new model based on the foundation model that performs some task (e.g., document summarization). For deployment, adaptation is about creating a system, which requires potentially many different modules, custom rules (e.g., restrictions on the output space) or classifiers (e.g., for toxicity classification), and combination with other complementary signals (e.g., a question answering model’s generated answers would be validated against relevant documents). For example, a problematic model capable of generating toxic content might be tolerable if appropriate precautions are taken downstream. The extra application-specific logic is crucial for mitigating harms.

• (5) Deployment: The direct social impact of an AI system occurs when it is deployed to people. Though we would not want to deploy potentially harmful foundation models trained on questionable data, there might still be value in permitting them in research to advance scientific understanding, though one must still exercise caution. More generally, it is standard practice in large-scale deployments to conduct gradual releases, where deployment happens to an increasing fraction of users; this can partially mitigate any potential harms.

While this report is about foundation models, it is important to note that many of the impacts come from decisions made in other stages in the pipeline, and thoughtful monitoring and intervention is needed at every stage. While large organizations might own the entire pipeline, each stage could be performed by a different organization, e.g., a company which specializes in creating custom foundation models for various domains that application-developers can use.

Think ecosystem, act model

While the social impact depends on the whole ecosystem, it is still important to be able to reason about the social implications of a foundation model, given that many researchers’ and practitioners’ purview is restricted to the training stage. This is difficult because foundation models are unfinished intermediate objects that can be adapted to many downstream applications, sometimes by an entirely different entity for unforeseen purposes. What we need are two things: (i) surrogate metrics for a representative set of potential downstream evaluation (§4.4: evaluation), and (ii) a commitment to documenting these metrics (Mitchell et al., 2019) similar to data sheets for materials such as metals and plastics, which can be adapted to many downstream use cases.

Characterizing the potential downstream social impact of foundation models is challenging and demands a deep understanding of both the technological ecosystem and of society. One cannot fully assess the harms (§5.1: fairness) of a foundation model without recognizing how it will be deployed, and one cannot just define automatic metrics without considering the rich social and historical context.

The future of foundation models

Foundation models have demonstrated raw potential, but we are still in the early days. Despite their deployment into the real world, these models are very much research prototypes that are poorly understood. Even the professional norms — what Robert Merton calls the ethos of science (Merton, 1979) — around foundation models are underdeveloped. For example, there is lack of agreement on basic questions such as when models are “safe” to release or how the community should react in response to methodological misconduct. Given that the future of foundation models is thus filled with uncertainty, a big question is: who will determine this future?

Disciplinary diversity

The technology behind foundation models is based on decades of research in machine learning, optimization, NLP, computer vision, and other fields. These technical contributions have come from both academia and industrial research labs. However, research on building foundation models themselves has occurred almost exclusively in industry — big tech companies such as Google, Facebook, Microsoft, or Huawei, or startups such as OpenAI or AI21 Labs, though AI2 is a notable exception (Peters et al., 2018; Zellers et al., 2019b).

The furious pace of technological progress and the entrenchment due to centralization raise powerful concerns that demand the attention of humanists and social scientists in addition to technologists. We should not rely on post-hoc audits of ethical and social consequences, conducted only after the technical architecture and deployment decisions have been made. We instead need to infuse social considerations and ethical design deeply into the technological development of foundation models and their surrounding ecosystem from the start. Academic institutions are unique in that they host the widest set of disciplines under one roof, thus bringing together computer scientists, social scientists, economists, ethicists, legal scholars, etc. Given the importance of disciplinary diversity in understanding and solving problems that combine technical, ethical, legal, social, and political dimensions (Hong and Page, 2004; Solomon, 2006; Steel et al., 2018), we therefore see academia as playing a crucial role in developing foundation models in such a way to promote their social benefit and mitigate their social harms, as well as determining the contexts under which actions in each of the stages of the ecosystem (§1.2: ecosystem) ranging from data curation to deployment should be strictly prohibited.

Incentives

The political economy in which foundations models are designed, developed, and deployed provides an inevitable incentive structure for decision-making at every stage. How people and institutions respond to incentives is an elementary lesson of economics. Market-driven commercial incentives can align well with social benefit: making foundation models more accurate, reliable, safe, and efficient while searching for a wide variety of potential use cases can produce a great deal of social utility. However, commercial incentives can also lead to market failures and underinvestment in domains where shareholders are unable to capture the value of innovation. Just as the pharmaceutical industry has little incentive to devote significant resources to the research and development of malaria treatments, because poor people cannot afford medications, the tech industry has little incentive to devote significant resources to technologies designed for improving the condition of poor and marginalized people (Reich et al., 2021). What’s more, commercial incentives can lead companies to ignore social externalities (Acemoglu, 2021; Reich et al., 2021) such as the technological displacement of labor, the health of an informational ecosystem required for democracy, the environmental cost of computing resources, and the profit-driven sale of technologies to non-democratic regimes. Finally, there is little incentive for any given company to create an open, decentralized ecosystem for developing foundation models that encourages broad participation.

In contrast, the long-standing and deeply-seated research mission of universities is the production and dissemination of knowledge and creation of global public goods (Kerr, 2001; Rhoten and Calhoun, 2011; Nussbaum, 2010). We believe that academia is distinctively positioned to shape the development of foundation models to ensure that we capture directions with potentially large social benefit that might not otherwise be prioritized by industry.

Loss in accessibility

Unfortunately, academia has not been able to participate in the fullest way possible due to the loss in accessibility. One of the often overlooked effects of the deep learning revolution was the increase in reproducibility and open science: it increasingly became the norm to publicly release code and datasets, and packages such as TensorFlow (Abadi et al., 2016) and PyTorch (Paszke et al., 2019) made it much easier for people to collaborate and build off of each other’s work. Initiatives like the ML Reproducibility Challenge as well as reproducibility checklists adopted by major conferences (Pineau et al., 2020), alongside platforms like CodaLab Worksheets helped advance community standards for reproducibility. This resulted in a surge in technological innovation and progress.

Foundation models start to roll back this positive trend. Some models (e.g., GPT-3) are not released at all (only API access to a limited pool of people). Even datasets (e.g., for GPT-2) are not released. While trained models may be available (e.g., BERT), the actual training of foundation models is unavailable to the vast majority of AI researchers, due to the much higher computational cost and the complex engineering requirements.

Some meaningful research can still be done by training smaller models within reach of an academic budget, and indeed the surprisingly regularity predicted by scaling laws (Kaplan et al., 2020) make this a viable strategy for cases where the differences due to scale are quantitative (e.g., accuracy goes up). However, due to the emergent nature of these foundation models, some functionalities like in-context learning have only been demonstrated in models of sufficient size, so scale is needed to even ask the right questions.

It is also possible to productively study pre-existing models that have been released; indeed, this has led to a large subcommunity within NLP for probing these models (Rogers et al., 2020; Manning et al., 2020). Having access to existing models can be useful for powering downstream applications or identifying defects (e.g., bias), but this might not be enough for us to design better architectures or training objectives for foundation models that can fix these defects (e.g., mitigate the bias). It is worth reflecting on how much of NLP research today is based on BERT, a particular (and somewhat arbitrary) foundation model. Given the need to infuse social awareness and ethical design into the construction of these models, it is possible that we need to build foundation models that look quite different from what exists today. This will demand intense experimentation at scale.

Community efforts such as EleutherAI and Hugging Face’s BigScience project are attempting to train large foundation models, but the gap between the private models that industry can train and the ones that are open to the community will likely remain large if not grow. Further, today startups (OpenAI, Anthropic, AI21 Labs, etc.) are much more well-resourced than academia and can therefore still afford to train the largest foundation models (e.g., OpenAI’s GPT-3). However, big tech companies are on a completely different level in terms of resources, especially in terms of the infrastructure, users, and data that come from their market position. The fundamental centralizing nature of foundation models means that the barrier to entry for developing them will continue to rise, so that even startups, despite their agility, will find it difficult to compete, a trend that is reflected in the development of search engines (Radinsky, 2015).

One way to close the resource gap is for the government to invest in public infrastructure. We can look to Big Science projects such as the Hubble Space Telescope and the Large Hadron Collider as inspiration, where substantial investment made possible fundamental scientific discoveries which wouldn’t have been possible. One can imagine a similar infrastructure for computing, from which academic research on foundation models would greatly benefit. In the US, the nascent National Research Cloud initiative is a step in this direction.

Another complementary approach is to rely on volunteer computing, in which any of the billions of computing devices (nodes) can connect to a central server and contribute computation. The Folding@home project has successfully implemented this approach for simulating protein dynamics (Beberg et al., 2009). Recently, the Learning@home project is attempting to harness volunteer computing for training foundation models (Ryabinin and Gusev, 2020). The high latency connections between nodes and the high bandwidth requirements for training foundation models make this an open technical challenge.

Summary

There are tremendous economic incentives to push the capabilities and scale of foundation models, so we anticipate steady technological progress over the coming years. But the suitability of a technology relying largely on emergent behavior for widespread deployment to people is unclear. What is clear that we need to be cautious, and that now is the time to establish the professional norms that will enable the responsible research and deployment of foundation models. Academia and industry need to collaborate on this: industry ultimately makes concrete decisions about how foundation models will be deployed, but we should also lean on academia, with its disciplinary diversity and non-commercial incentives around knowledge production and social benefit, to provide distinctive guidance on the development and deployment of foundation models that is both technically and ethically grounded.