On the Opportunities and Risks of Foundation Models

Reasoning and search have been a central theme throughout the history of AI. Classic tests of intellect, from strategy games to abstract mathematical discovery, served as inspirational goal posts that pushed the limits of “machine intelligence” through a need to devise ever smarter ways of searching for winning solutions. In the early days, symbolic methods were the dominant approach for reasoning (Russell and Norvig, 2020), but the involved engineering effort and the need to formalize heuristics to tackle intractable search spaces quickly proved cumbersome. More recently, data-driven methods using neural networks have shown encouraging results — e.g., defeating the best humans in Go (Silver et al., 2016), a board game with a much larger space of actions than the classic challenge of chess — by exploiting statistical structures and learning useful heuristics. This section outlines existing reasoning tasks, ones that require scaling to ever-larger search spaces and understanding the world broadly (§2.4.1: reasoning-tasks). We then argue in §2.4.2: reasoning-role that foundation models should play a central role towards general reasoning as vehicles for capturing the statistical regularities of unbounded search spaces (generativity), allowing positive transfer across tasks and scenarios (universality), and exploiting the grounding of knowledge in multi-modal environments (grounding).

What are the current tasks?

Many reasoning problems pose unbounded search spaces, where systems must deal with numerous kinds of open-ended alternatives. Consider trying to prove that the angles ∠B and ∠C are equal for an isosceles triangle △ABC with AB=AC (Figure 9). A system can perform any number of actions at each step of reasoning. For instance, the system could add a new auxiliary point with an arbitrary construction, say a perpendicular line, a parallel line, or a tangent circle, and the search space only grows larger as the diagram grows more complicated. One way to prove this theorem is to draw a line AD that is the angle bisector of A, and use the congruence of the two triangles △ABD and △ACD to show ∠B=∠C, but how can systems find this without extensive search?

More generally, a mathematician is not confined with searching in diagram constructions and Euclidean theorems: mathematicians can apply a vast number of theorems from various branches of mathematics, make high-level conjectures, formalize new mathematical concepts, or find counterexamples. This contrasts with more structured AI challenges such as the game of Go, whose search space is considered much smaller.

Besides theorem proving, many real-world problems deal with unbounded search spaces, such as program synthesis (Gulwani et al., 2017), drug discovery (Drews, 2000), chemical synthesis (Segler et al., 2018), computer-aided design (Haigh, 1985), combinatorial optimization (Bengio et al., 2021), and more. These reasoning problems tend to exhibit similar structure, like the bijection between retrosynthesis in drug discovery and theorem proving in propositional logic, illustrated in Figure 10: in both problems, one is building a tree of synthesis, whose nodes are chemical products on the one side and propositions on the other, and the leaf nodes are the products on the one side, and end axioms on the other. In these problems, a simulated environment is often provided, which allows a solver to run several search threads towards building the solution tree. The simulator often provides intermediate feedback, say, informing the solver with the remaining propositions to establish before the proof is considered complete. The solver in turn needs to select the most promising search thread and proceed based on the intermediate feedback.

Recently, there has been a surge of interest in applying learning-based approaches to tackle reasoning problems. To overcome the unbounded search space challenge, researchers first started with a constrained search space to make the problem tractable (Huang et al., 2018; Bansal et al., 2019). But such approaches suffered from the limited kinds of actions the solver could issue. For example, the solver could only apply theorems from a known database to prove the target theorem, instead of synthesizing novel theorems and lemmas. Because large language models offered a generic way of modeling the output space as a sequence, they quickly became a more favorable choice, allowing the generation of arbitrary kinds of actions. Researchers have applied these language model-based approaches to various applications, such as predicting protein structures (Senior et al., 2020), proving formal theorems (Polu and Sutskever, 2020; Han et al., 2021), conjecturing theorems (Urban and Jakubuv, 2020; Rabe et al., 2021; Li et al., 2021b), synthesizing programs from natural language (Chen et al., 2021f; Ling et al., 2016), repairing, generating and understanding code (Yasunaga and Liang, 2021; Lu et al., 2021b; Guo et al., 2020; Svyatkovskiy et al., 2020; Kim et al., 2021b; Zügner et al., 2021). It has also been shown that scaling model size significantly improves reasoning capabilities (Polu and Sutskever, 2020), and furthermore standard techniques from language modelling, such as pretraining, can also greatly improve performance on these tasks (Rabe et al., 2021; Polu and Sutskever, 2020).

What’s the role of foundation models?

Generativity

We believe that the generative capabilities of foundation models are essential for effective reasoning. Due to the unbounded search space, it becomes intractable to enumerate all kinds of possibilities. Instead, with foundation models, one can model the distribution of the optimal decisions, and generate suitable candidates to proceed to the next step. In particular, as foundation models offer a generic way of modeling the output space as a sequence, the next decision generation is entirely unconstrained and hence universal. Such flexibility is essential for many of the reasoning challenges we discussed, to allow creative generation in domains such as mathematical conjecturing (Li et al., 2021b) and synthesizing novel programs (Chen et al., 2021f). As one scales up foundation models, the capabilities of capturing such statistical structures also grow immensely (Polu and Sutskever, 2020).

Universality

As we mentioned in the last section, many reasoning problems exhibit similar latent structures. We believe that the unifying framework imposed by a foundation model can transfer and share significant heuristics across tasks, ranging from generalizing low-level techniques that work well for one task to new scenarios all the way to directly finding meta-techniques that work well across numerous kinds of problems. In addition, since a foundation model is trained across many domains, it can positively transfer meta-knowledge encoded in the foundation models’ weights across tasks and domains (Papadimitriou and Jurafsky, 2020; Wu et al., 2021f; Lu et al., 2021a). The foundation model training and adaptation framework encourage a separation of concerns, where foundation model training learns meta-knowledge such as the shared search tree structure between drug retrosynthesis and propositional logic proofs, and the adaptation phase can focus on learning the task specific vocabulary. Thus, foundation models can reduce the complexity of the learning problem in the adaptation phase, improving sample complexity and generalization.

Grounding

Reasoning problems are often easily expressed in symbolic languages (e.g., mathematics, code, SMILE representation of molecules). However, these symbols have deep underlying semantic meanings — saying “isosceles triangle” paints a vivid image in the human mind. Foundation models can enable deep groundings and semantic meanings. First, grounding representations in other modalities, such as visual or physical, are essential to grasp abstract concepts in reasoning tasks and endow them with concrete meaning (Larkin and Simon, 1987; Jamnik, 2001). Since the models may be trained on multiple modalities, foundation models can assist in understanding a range of data sources (e.g., images, texts). Hence, in the geometry example case, with its understanding of geometrical shapes learned in natural images, a foundation model could effectively utilize the diagrammatic representation of the problem. However, aligned multi-modal data in reasoning is scarce, and it remains an open question whether foundation models can discover connections between different modalities in an unsupervised manner (e.g., discovering of commutative diagram with the corresponding algebraic equations). Furthermore, even within the symbolic domain, symbols can have various levels of interpretation. For example, high-level programming languages can be translated to low-level assembly codes. Foundation models can learn a shared representation that encompasses these various views. Past works have shown that self-supervised tasks (Han et al., 2021; Peng et al., 2021; Li et al., 2021a) allow the model to understand the inner workings behind the high-level code scripts, and further assist downstream tasks.

Future challenges in reasoning

Due to the intrinsic difficulty of these problems, high-quality annotated data is scarce and harder to collect compared to raw images and text. There have been several attempts towards alleviating this issue. In mathematics, researchers proposed to generate synthetic theorems in the hope of generalizing to realistic theorems (Wang and Deng, 2020; Wu et al., 2021a; Firoiu et al., 2021; Zhou et al., 2021c). Another approach is to design self-supervised tasks to augment datasets (Yasunaga and Liang, 2020; Ren et al., 2020; Han et al., 2021; Rozière et al., 2021; Yasunaga and Liang, 2021), or better pretraining objectives (Wu et al., 2021f). However, we still lack general principled approaches in designing self-supervised tasks, as most of the existing works are tailored to specific problem setups (Yasunaga and Liang, 2020; Ren and Leskovec, 2020; Han et al., 2021). Building a foundation model will encourage a unifying framework of constructing a suite of self-supervised tasks that can be applied to all reasoning problems. In addition, interactivity (§2.5: interaction) could, with enough scalability, alleviate the data scarcity problem by bringing humans into the loop to minimally guide the learning curriculum or data augmentation process, for example, in selecting axioms to add or conjectures to explore, while interactive tools themselves are a motivating use of foundation models for reasoning (Han et al., 2021; Chen et al., 2021f) in assisting people with the most cognitively demanding or laborious aspects. Interpretation-friendly interactive tools could find further applications in education by assisting humans in learning with the help of highly capable foundation models (§3.3: education).

Improving the high-level reasoning capabilities is a core challenge for existing foundation models. Humans perform abstract reasoning and high-level planning in tackling difficult problem-solving tasks (Miller et al., 1960). For example, when building a software tool or proving a theorem, we often start with a high-level sketch before delving into the low-level details (Koedinger and Anderson, 1990). Existing foundation models are not trained to generate such high-level plans. Instead, they often focus solely on predicting the next low-level steps (Polu and Sutskever, 2020; Han et al., 2021; Chen et al., 2021f). Unfortunately, to train foundation models to emulate human-like reasoning, we again face a data collection challenge. Although such data does exist in limited settings (Li et al., 2021b), in general, data for high-level reasoning is scarce and difficult to collect. One line of research is to let abstract and modular hierarchy to emerge by itself during learning (Ellis et al., 2021; Hong et al., 2021), but it still remains an open question how to scale these approaches to more general and realistic settings.

Aside from these challenges, there exist many open questions that are also essential to topics discussed in other sections. What constitutes a good architecture for reasoning reliably (§4.1: modeling)? How can we understand and interpret these models theoretically (§4.10: theory and practically §4.11: interpretability)? Can we train robust reasoning models that could generalize to out-of-domain problems (§4.8: robustness and §4.3: adaptation)? We believe research about foundation models on each of these fronts can greatly broaden their impact for the field of reasoning.