On the Opportunities and Risks of Foundation Models

What could a foundation model come to understand about the data it is trained on? An answer to this question would be extremely informative about the overall capacity of foundation models to contribute to intelligent systems. In this section, we focus on the case of natural language, since language use is a hallmark of human intelligence and central to the human experience.

The best foundation models at present can consume and produce language with striking fluency, but they invariably lapse into the sort of incoherence that suggests they are merely “stochastic parrots” (Bender et al., 2021). Are these lapses evidence of inherent limitations, or might future foundation models truly come to understand the symbols they process?

Our aim in this section is to clarify these questions, and to help structure debates around them. We begin by explaining what we mean by foundation model, paying special attention to how foundation models are trained, since the training regime delimits what information the model gets about the world. We then address why it is important to clarify these questions for the further development of such models. Finally, we seek to clarify what we mean by understanding, addressing both what understanding is (metaphysics) and how we might come to reliably determine whether a model has achieved understanding (epistemology).

Ultimately, we conclude that skepticism about the capacity of future models to understand natural language may be premature. It is by no means obvious that foundation models alone could ever achieve understanding, but neither do we know of definitive reasons to think they could not.

What is a foundation model?

There is not a precise technical definition of foundation model. Rather, this is an informal label for a large family of models, and this family of models is likely to grow and change over time in response to new research. This poses challenges to reasoning about their fundamental properties. However, there is arguably one defining characteristic shared by all foundation models: they are self-supervised. Our focus is on the case where self-supervision is the model’s only formal objective.

In self-supervision, the model’s sole objective is to learn abstract co-occurrence patterns in the sequences of symbols it was trained on. This task enables many of these models to generate plausible strings of symbols as well. For example, many foundation models are structured so that one can prompt them with a sequence like “The sandwich contains peanut” and ask them to generate a continuation – say, “butter and jelly”. Other models are structured so that they are better at filling in gaps; you might prompt a model with “The sandwich contains __ and jelly” and expect it to fill in “peanut butter”. Both capabilities derive from these models’ ability to extract co-occurrence patterns from their training data.

There is no obvious sense in which this kind of self-supervision tells the model anything about what the symbols mean. The only information it is given directly is information about which words tend to co-occur with which other words. On the face of it, knowing that “The sandwich contains peanut” is likely to be continued with “butter and jelly” says nothing about what sandwiches are, what jelly is, how these objects will be combined, etc. This might seem to suggest an inherent limitation on what a foundation model could achieve. However, we need not restrict the model to seeing only textual input. A foundation model might be trained on a wide range of different symbols: not just language but also computer code, database files, images, audio, and sensor readings. As long as it is just learning co-occurrence patterns of the sequences it is exposed to, then it counts as a foundation model by our definition. As part of this learning, the model might come to represent strong associations between a given piece of text and a particular sensor reading, or between a sequence of pixel values and a database entry. These associations might reflect important aspects of the world we inhabit and the language we use to talk about it.

What is at stake?

Before considering analyses of what understanding is, it is worth reflecting on why we might care about the question of whether a foundation model could achieve it. These models are poised to be deployed for numerous purposes with various functionalities. Some of our goals in deployment may only be met to the extent that the model is capable of understanding. Here we list a few such goals:

• • Trust: One might argue that we cannot trust a system’s linguistic behavior unless it understands the language it is using. Of course, we currently trust engineered systems to do things (e.g., manufacturing auto parts) without the question of understanding even arising, but language might be special in this regard, since it is uniquely human. In addition, language can be used to deceive and misrepresent, so understanding alone clearly does not imply trust. On the whole, then, understanding might be taken as a necessary condition for trust in the context of language use.

• • Interpretability: If genuine natural language understanding in some way involves maintaining and updating an internal model of the world (including, e.g., the speech context), and if we (as engineers) are able to analyze how linguistic input and output interface with this internal model, that could afford substantial gains in interpretability, predictability, and control of these systems.

• • Accountability: Not unrelated to the previous points, in the future we may find it desirable to hold artificial agents in some way accountable for the language they produce (The HAI Adaptive Agents Group, 2021). Depending on how we think about concepts like accountability, responsibility, agency, and the like, language understanding may emerge as a prerequisite.

The mere possibility that understanding will play an indispensable role in any of these matters provides strong motivation to develop a framework for theorizing about it.

What is understanding?

Our central question is whether a foundation model could come to understand a natural language. With the above, we can now sharpen it: is self-supervision sufficient for understanding, keeping in mind that there are no constraints on the data used for this supervision? In order to address this question, we first need to define what we mean by understanding.

As a start, we find it helpful to make explicit a distinction that is sometimes conflated in discussions of the topic. The distinction is between the metaphysics and the epistemology of understanding. Metaphysics concerns what it would mean (“in principle”) for an agent to achieve understanding. Epistemology, by contrast, concerns how (“in practice”) we could ever come to know that an agent has achieved the relevant type of understanding. In short, metaphysics is more about our ultimate target, whereas epistemology is more about how (if at all) we could know when we have reached it. Our epistemology thus depends to some extent on our metaphysics.

Metaphysics of understanding

Philosophy of language offers a number of alternatives for what it is to understand natural language. Simplifying the landscape for the sake of brevity, the following three broad classes of views all have connections with research lines in AI and NLP:

• • Internalism: Language understanding amounts to retrieval of the right internal representational structures in response to linguistic input. Thus, language understanding is not even a possibility without a rich internal conceptual repertoire of the right kind.

• • Referentialism: Roughly, an agent understands language when they are in a position to know what it would take for different sentences in that language to be true (relative to a context). That is, words have referents and (declarative) utterances are truth-evaluable, and understanding involves a capacity to evaluate them relative to presentation of a situation or scenario.

• • Pragmatism: Understanding requires nothing in the way of internal representations or computations, and truth and reference are not fundamental. Rather, what matters is that the agent be disposed to use language in the right way. This might include dispositions toward inference or reasoning patterns, appropriate conversational moves, and so on. Crucially, the relevant verbal abilities constitute understanding.

While this is a simplified picture of the space of possibilities, we already see how they relate in quite different ways to the goals mentioned above. On the pragmatist view, for instance, achieving language understanding does not imply anything about our ability to trust or interpret the system, insofar as it guarantees nothing about the agent’s internal structure or its relation to the (non-linguistic) world. On the internalist view, by contrast, a fairly robust kind of internal/causal interpretability is at least strongly suggested. The question of whether or not a foundation model could understand language in principle takes on a very different character depending on which of these metaphysical characterizations we adopt.

Internalism and referentialism can both be cast as defining a mapping problem: to associate a linguistic sign with a “meaning” or a “semantic value”. For internalism this will be a representation or concept, a program for computing a value, or some other type of internal object. For referentialism, it might be a mapping from a word to an external referent, or a mapping from a situation to a truth value (all relative to a context). Could self-supervision suffice for achieving the desired mapping in a foundation model? Here, the nature of the training examples might be relevant. If the model receives only linguistic inputs, then its capacity to learn this mapping might be fundamentally limited in ways that prevent it from learning to refer in the relevant sense. (Indeed, Merrill et al. (2021) identify some theoretical limits, albeit under very strong assumptions about what it means to learn the meaning of a symbol.) However, if the input symbol streams include diverse digital traces of things in the world – images, audio, sensors, etc. – then the co-occurrence patterns might contain enough information for the model to induce high-fidelity proxies for the required mapping. For referentialism, there is still a further question of how these proxies relate to the actual world, but the same question arises for human language users as well.

Bender and Koller (2020) give an interesting argument that combines referentialism with pragmatism. They imagine an agent O that intercepts communications between two humans speaking a natural language L. O inhabits a very different world from the humans and so does not have the sort of experiences needed to ground the humans’ utterances in the ways that referentialism demands. Nonetheless, O learns from the patterns in the humans’ utterances, to the point where O can even successfully pretend to be one of the humans. Bender and Koller then seek to motivate the intuition that we can easily imagine situations in which O’s inability to ground L in the humans’ world will reveal itself, and that this will in turn reveal that O does not understand L. The guiding assumption seems to be that the complexity of the world is so great that no amount of textual exchange can fully cover it, and the gaps will eventually reveal themselves. In the terms we have defined, the inability to refer is taken to entail that the agent is not in the right dispositional state for understanding.

Fundamentally, the scenario Bender and Koller describe is one in which some crucial information for understanding is taken to be missing, and a simple behavioral test reveals this. We can agree with this assessment without concluding that foundation models are in general incapable of understanding. This again brings us back to the details of the training data involved. If we modify Bender and Koller’s scenario so that the transmissions include digitally encoded images, audio, and sensor readings from the humans’ world, and O is capable of learning associations between these digital traces and linguistic units, then we might be more optimistic – there might be a practical issue concerning O’s ability to get enough data to generalize, but perhaps not an in principle limitation on what O can achieve.

We tentatively conclude that there is no easy a priori reason to think that varieties of understanding falling under any of our three positions could not be learned in the relevant way. With this possibility thus still open, we face the difficult epistemological challenge of clarifying how we could hope to evaluate potential success.

Epistemology of understanding

A positive feature of pragmatism is that, by identifying success with the manifestation of concrete behaviors, there is no great conceptual puzzle about how to test for it. We simply have to convince ourselves that our limited observations of the system’s behavior so far indicate a reliable disposition toward the more general class of behaviors that we took as our target. Of course, agreeing on appropriate targets is very difficult. When concrete proposals are made, they are invariably met with objections, often after putative success is demonstrated.

The history of the Turing Test is instructive here: although numerous artificial agents have passed actual Turing Tests, none of them has been widely accepted as intelligent as a result. Similarly, in recent years, a number of benchmark tasks within NLP have been proposed to evaluate specific aspects of understanding (e.g., answering simple questions, performing commonsense reasoning). When systems surpass our estimates of human performance, the community’s response is generally that the test was flawed, not that the target was reached. There may be some suite of behaviors that is our real target, but it is just hard to circumscribe or turn into a practical test. Then again, this might reveal that internalism or referentialsm are what we had in mind all along.

If we take internalism or referentialism as the ultimate target – our gold standard for what understanding is – then behavioral tests will always be at best imperfect as a means of assessing whether understanding has been achieved. The imperfections are two-fold. First, behavioral tests will always have gaps that could allow unsophisticated models to slip through. Second, a system might have achieved the mapping that these views require, but we may be unable to show this with behavioral testing. Recent experiences with the model GPT-3 show how challenging this might become: depending on the prompt one uses, one can see surprisingly coherent outputs or utter nonsense, and so prompt engineering requires deep expertise (Rong, 2021).

Thus, both internalism and referentialism call for structural evaluation methods that allow us to study their internal representations, probing them for information (Tenney et al., 2019; Manning et al., 2020), studying their internal dynamics (Sundararajan et al., 2017), and perhaps actively manipulating them according to specific experimental protocols supporting causal inference (Vig et al., 2020; Geiger et al., 2020). There may be fundamental limitations on what we can learn from practical experiments about the inner workings of a complex foundation model, but it is clear that these methods will be useful whenever our target aligns with internalism or referentialism.

Moving the discussion forward

It seems clear that there are no easy answers to the question of whether foundation models will ever understand language. To even begin to address the question, one must resolve a difficult metaphysical question about which there are a number of substantively distinct views. The metaphysical question then feeds into an epistemological question that poses many practical challenges. Nonetheless, the above discussion does invite one practical conclusion: if foundation models are pursued as a path to language understanding in artificial agents, then multimodal training regimes may well be the most viable strategy, as they would seem the most likely to provide the model with the requisite information. Whether self-supervision then suffices is a completely open question.