On the Opportunities and Risks of Foundation Models

A longstanding challenge of robotics research is to endow robots with the ability to handle the myriad conditions they will encounter in real-world settings. In this section, we discuss how the ideas underlying foundation models can potentially help bring about “generalist” robots that can, for example, cook a new meal in a new house, with a new kitchen. To make progress towards this goal, existing foundation models will not suffice. We need new types of models trained on a multitude of data sources, spanning grounded robotic interaction data to videos of humans performing tasks, amongst others. We focus on how such foundation models can apply to the problem of a robot controlling its own physical embodiment to successfully perform different tasks. This is a high-dimensional and closed-loop decision-making problem: the actions that a robot takes directly influence what it perceives next, which in turn influences the next robot action. This closed-loop aspect is not traditionally studied in language and computer vision, where large offline datasets are dominant and foundation models have already seen success. We focus on how the demonstrated benefits of foundation models — large-scale, self-supervised learning — can be leveraged in this new closed-loop data regime. The promise of a new type of robotic foundation model is in its ability to amplify the potential of robots to improve key facets of daily life ranging from manufacturing (Nof, 1999; Sanneman et al., 2020), construction (Khoshnevis, 2004; Bock, 2007), autonomous driving (Thorpe et al., 1988; Badue et al., 2020), to household aid (Thrun and Mitchell, 1995; Brooks, 2002; Dillmann, 2004; Goodrich and Schultz, 2007; Gupta et al., 2018; Shridhar et al., 2020) and personal assistance (Dragan and Srinivasa, 2013; Javdani et al., 2018), amongst others. Our discussion in this section primarily focuses on mobile manipulation robots for household tasks, but we expect its essence to be broadly applicable to the other use-cases of robotics listed above.

On the critical path towards building new types of foundation models for robotics is embracing opportunities in task specification and task learning, coupled with tackling challenges in data acquisition and safety and robustness. Consider the following robot learning paradigm: starting with a description of a task capturing what a user might like the robot to do (e.g., “make breakfast”) — learn a corresponding policy to generate the desired robot actions. While policies can be parameterized in different ways, a common choice is that of a function that maps the task representation and environment observation (e.g., a scene image from a fixed or egocentric camera, or inputs from alternative sensors like LIDAR) to robot actions (Andrychowicz et al., 2017; Nair et al., 2018). As the robot acts in a task-conditioned manner, the subsequent states are fed back to the policy, generating more actions until the task has been satisfied.

Yet, implementing such a paradigm in practice is difficult. To begin, what is the right interface for describing one’s goals? For a given user in one context, “make breakfast” carries an implication of a full breakfast that consists of fried eggs, toast, and a glass of orange juice; for another user, “make breakfast” may imply idlis with sambar and a tumbler of filter coffee. In general, high-level context-dependent goals like these do not stand alone and can introduce a multitude of ambiguities. How does one specify a goal (and corresponding subgoals) with enough clarity to both resolve these ambiguities, and in so doing, allow a robot to make progress on the given task? Additionally, how might we craft general task representations that might aid generalization to similar objectives (e.g., fetching a glass of milk instead of orange juice). Going a step further, how do we build methods that aid robots in learning policies for new tasks and new environments (in this case, a brand new kitchen with new utensils, appliances, layouts, etc.)?

Recent breakthroughs in applying foundation models for language and vision (§2.1: language and §2.2: vision) suggest several potential benefits of large-scale, self-supervised pretraining for improving generalization. The ability to tap into diverse streams of data to learn meaningful representational priors (akin to those learned by models such as BERT and GPT-3) holds promise for learning powerful robotic foundation models for task specification. Diverse robotic interaction data can be used for learning action-conditional dynamics models or policies indexing general and semantically meaningful skills thereby holding promise for task learning. Yet while these opportunities exist, the key stumbling block is collecting the right data. Unlike language and vision data, robotics data is neither plentiful nor representative of a sufficiently diverse array of embodiments, tasks, and environments — we (as a field) still have not converged on the kinds of data that would be maximally useful for enabling generalist robotics (e.g., offline demonstrations, third-person recordings of humans, egocentric videos, autonomous experience, etc.) Coupled with issues in obtaining the right scale and diversity of data are questions of ensuring safety and robustness: how do we behave in a new environment without causing damage?

Building new types of foundation models for robotics thus consists of a dichotomy of opportunities and challenges: opportunities for task specification and learning balanced against challenges of data collection and safe deployment. This section explores both by presenting a picture of how robotic foundation models might help us develop generalist robots, in a way that not only meaningfully addresses the challenges associated with building such systems, but that also embraces the potential of multi-modality — incorporating perception, actuation, and language — as well as human-robot interaction for specification and learning.

Opportunities

Robotic foundation models could take a variety of forms: problems in robotics do not easily conform to a one-size-fits-all model, since different problems have different input-output signatures — a contrast to domains like NLP where many problems can be cast into a general “text-in, text-out” signature. We focus on opportunities in generalizable task specification and learning across tasks, environments, and robot embodiments.

Foundation models for task specification

Before robots can learn how to solve tasks in a general purpose way, they must understand what the desired task is: for example, to be useful in a new kitchen, a robot needs to know what we would like it to cook, as well as behaviors we would like it to avoid. Therefore, a necessary first step towards developing generalist robots is building a new type of foundation models for reliable task specification, i.e., the intuitive and effective communication of task objectives, preferences, and constraints. We formalize task specification as a process that transforms a human-provided task description into a quantitative metric that measures a robot’s task completion and progress — e.g., a reward function. This signal is crucial for optimizing robot behavior, diagnosing failures, and prompting human feedback. As the most natural way to describe a task can vary depending on the user, environment, or task, robotic foundation models for task specification should accept a variety of description modalities, such as goal states (Fu et al., 2018; Singh et al., 2019), natural language (MacGlashan et al., 2015; Karamcheti et al., 2017; Misra et al., 2017b; Co-Reyes et al., 2019; Shao et al., 2020), videos of humans (Shao et al., 2020; Chen et al., 2021c; Liu et al., 2018), pairwise or ranking comparisons (Biyik and Sadigh, 2018), interactive corrections (Co-Reyes et al., 2019; Karamcheti et al., 2020) and physical feedback (Ross et al., 2011; Bajcsy et al., 2017).

An important requirement of general purpose models for task specification is the ability to transfer to new environments and tasks. Reliably transforming task descriptions into generalizable reward signals for robot learning remains an open problem (Taylor et al., 2016) — one that robotic foundation models will arguably be well suited for. When applied to task specification, such models should provide more robust (§4.8: robustness) reward signals by learning from large and broad datasets — even leveraging multiple of the description modalities listed above. One possible instantiation of a new foundation model for task specification might be to learn a mapping from arbitrary (language, current observation) pairs to reward signals by training on diverse language and vision datasets (Bahdanau et al., 2019; Fu et al., 2019; Chen et al., 2021c). By learning informative priors from these broad, diverse datasets, such a model may be able to generalize to unseen language instructions and observations in unseen environments. In general, the potential for new foundation models to be able to deftly bridge modalities and generalize broadly make them appealing for general purpose task specification.

Foundation models for task learning

In addition to enabling more general task specification, robotic foundation models could make learning to solve new tasks more efficient and reliable. In this context, these new types of foundation models might take the form of a joint distribution over actions, sensor observations, rewards, and other properties of interest. Conditioning on different dimensions of this joint distribution recovers different inference problems, each corresponding to a different signature:

• • Dynamics modeling: p(future observations ∣ actions, past observations) (Finn and Levine, 2017; Hafner et al., 2019; Wu et al., 2021d).

• • Policy learning: p(actions ∣ observations, goal) (Kaelbling, 1993; Schaul et al., 2015; Ding et al., 2019).

• • Inverse reinforcement learning: p(reward function ∣ observations, actions) (Ng and Russell, 2000; Ziebart et al., 2008; Finn et al., 2016a).

A plausible training objective for a robotic foundation model is to predict the different elements of the joint distribution described above in an autoregressive fashion (Janner et al., 2021; Chen et al., 2021b, §4.1: modeling). However, these are not the only options. In particular, robot datasets contain large amounts of unlabeled data consisting of synchronized observations from many different sensor modalities (e.g., RGB and depth cameras, haptic sensors, microphones, etc.) and a sequence of actions that the robot has performed to generate these observations. Beyond the objectives above, a robotic foundation model could be trained to predict observations of one sensor modality from another or to predict whether two streams of sensory observations are from the same segment of time. These kinds of self-supervised objectives can leverage multi-modal correspondences to produce low-dimensional representations of high-dimensional data, and can even be combined with the above objectives to yield models, policies, and rewards on top of those representations.These objectives may facilitate the training of powerful robotic foundation models from unlabeled data — as long as the data exhibits diverse, meaningful behavior. §2.3.2: robotics-challenges discusses the challenges of collecting such data further.

In language and vision, foundation models have demonstrated the capability to learn broadly applicable priors from large, diverse datasets, that can be subsequently adapted to downstream tasks (§2.1: language, §2.2: vision). Robotic foundation models have the potential to similarly enable few-shot adaptation of perception and control to new environments, tasks, and embodiments though leveraging different data, self-supervised objectives, and modalities than have been studied with existing language and vision models. Consider our running kitchen example. To cook in a new kitchen, a robot needs to adapt to the specific environment — its spatial layout, the available equipment, etc. Priors learned from offline videos of humans, robotic interaction, text, and/or simulation might encode general aspects of kitchens, such as the fact that stoves are usually against walls and must be turned on in order to produce heat. Such commonsense knowledge, physical priors, and visual priors could make adaptation to new environments more sample efficient. Similarly, developing a new foundation model for robot task learning might enable the use of a large number of cooking videos in its training dataset to adapt a policy for a common skill, such as “fry an egg,” to a specific user’s preferences from a low number of demonstrations — allowing for sample efficient adaptation. Finally, with their potential to learn the cross-modal representations described earlier, robotic foundation models could help enable adaptation to new embodiments. This aspect of adaptation is crucial to make these models widely useful.

Challenges and risks

Despite this exciting vision, multiple challenges need to be overcome. To enable the generalization discussed above, we must collect robotic datasets of sufficient size and diversity. Additionally, we need mechanisms to ensure that we can deploy learned behaviors safely in the real world.

Data needs & challenges

Learning a policy for a robot that perceives the state of its environment via sensors and takes actions to accomplish tasks traditionally requires large datasets of the robot interacting in the real world. On the other hand, many learning tasks in computer vision and natural language processing rely on large and diverse offline datasets that can easily be scraped from the web. Motivated by the advances of existing foundation models in language and vision, we are excited by the possibility of leveraging large offline data sources for training new foundation models for robotics.

One path towards this goal is collecting large datasets for offline learning, for example using teleoperation (Mandlekar et al., 2019), kinesthetic teaching (Sharma et al., 2018), or autonomous methods (Pinto and Gupta, 2016; Gupta et al., 2018; Levine et al., 2018; Dasari et al., 2019; Kalashnikov et al., 2021; Chen et al., 2021d), which have shown some promising indications on generalization. While scaling up robot data collection to the size of vision and language datasets (Deng et al., 2009; Krishna et al., 2017; Raffel et al., 2019; Gao et al., 2020a) remains an open challenge, the increasing scale and quality of robotic datasets suggests they can play an important role in learning foundation models for robotics. Moreover, because robots have the ability to actively and autonomously shape their environment, they should be able to generate targeted unlabeled data at scale.

Given the challenging closed-loop nature of learning control, it is possible that collecting datasets of size comparable to those used in vision and language is insufficient for robotics. One exciting option is to additionally leverage external, non-robotic sources of data such as videos of humans or existing vision and natural language datasets. Such data is diverse and exists in large quantities on the web (Deng et al., 2009; Lee et al., 2012; Heilbron et al., 2015; Goyal et al., 2017a; Damen et al., 2018; Gao et al., 2020a; Grauman et al., 2021), affording the possibility of broad generalization if properly leveraged. Elegantly addressing the gap between the robot’s domain and those found in videos or language on the web remains an open challenge; however, recent progress in domain adaptation (Smith et al., 2019; Schmeckpeper et al., 2020) and using pretrained video and language models in robotics (Lynch and Sermanet, 2020; Shao et al., 2020; Chen et al., 2021c) present promising directions towards closing this gap.

Finally, simulation presents a boundless source of rich interactive data that robots can learn from, with a range of sensor modalities like rendered visuals, point-clouds, and simulated touch/audio. However, a major challenge lies in bridging the gap between simulation and the real world, both in the underlying physics and in the semantic distribution of environments and tasks. Recent work has shown that by using extensive domain randomization, tasks ranging from flight (Sadeghi and Levine, 2017) to contact-rich manipulation (Mahler et al., 2017; OpenAI et al., 2019) and locomotion (Peng et al., 2020; Hwangbo et al., 2019) skills learned in simulation can be transferred to real robots with some success, and that the semantic and visual distribution of the real world can be simulated by scanning the real world into a simulation (Chang et al., 2017; Kolve et al., 2017b; Savva et al., 2019a; Szot et al., 2021; Shen et al., 2021a). While these are promising steps towards closing the sim-to-real gap, effective and general sim-to-real learning of manipulation and locomotion skills remains an open challenge. Simulation data, real robot data, videos of humans, and natural language data could all be essential to learning robotic foundation models.

Safety & robustness

Further complicating the development of new foundation models for robotics is ensuring their safety and robustness when training or deploying them in the real world. We can expect the safety risks from these models for robotics to be different from their language counterparts given that embodied agents are empowered to manipulate and interact with their surroundings directly in the physical world. One core safety challenge for learning-based systems is the chicken-and-egg problem of needing to specify system constraints for safety prior to collecting data, after which unforeseen unsafe behaviors requiring additional constraints may emerge. For instance, an agent adapting to a new kitchen outside of the training distribution requires sufficient safety guarantees to ensure safe data collection, which may either adversely affect task performance or cause the agent to fail in novel ways. One way to resolve this is restricting the complexity of the environment or increasing the complexity of the robot such that irrecoverable states or unsafe actions are avoided by construction. The robot can also be tasked with autonomously resetting the environment to facilitate uninterrupted learning (or adaptation) from large-scale data collection (Eysenbach et al., 2017; Gupta et al., 2021b). This would either mean ensuring that nothing in the kitchen is breakable, or ensuring and replacing the items the agent may break while it attempts to collect data.

To address risks posed by robotic foundation models that fail to generalize or produce unexpected behaviors to new stimuli, potential future directions include developing a causal analysis of agents (Déletang et al., 2021), new formal safety evaluation tools, and realistic simulation environments (Corso et al., 2020; Dreossi et al., 2017; Julian and Kochenderfer, 2019). Finally, deriving formal safety guarantees for robotic foundation models, e.g., Hamilton-Jacobi reachability of safe-sets (Chow et al., 2018; Fisac et al., 2019; Herbert et al., 2021) or developing safety boundaries for learning that are interpretable (§4.11: interpretability) to human operators, could help reduce risks posed by such models (Berkenkamp et al., 2017). As the development and study of these new types of foundation models progresses, solutions to these challenges will be crucial.

Conclusion

While the promise of robotic foundation models are many — spanning multiple levels of the robotics pipeline from task specification to task learning — the challenges are significant. Collecting data in the physical world that covers diverse environments and embodiments at scale is a sizable hurdle, and ensuring the safety and robustness of such systems is equally exigent. Despite this, our optimism prevails; tackling these challenges now, before developing models offers us the chance to identify ways to collect the right data, from the right sources, at the right scale to build safe and reliable robotic foundation models with the capabilities we desire.

Underpinning this section has been a theme of multimodality. Robotic foundation models — in all possible instantiations — have and will continue to benefit from work in other subfields of AI such as language and vision (§2.1: language, §2.2: vision). Yet as we consider incorporating these extensions from other fields, there are interdisciplinary challenges on the horizon that touch other aspects of foundation models: systems innovation for training and deploying such models for real-time robotics (§4.5: systems), innovation in interfaces for robust human-robot interaction (§2.5: interaction), and lessons to incorporate as we better grasp the safety and robustness of such models (§4.9: ai-safety, §4.8: robustness). Building a reliable ecosystem and thoughtful research practices around foundation models in general, and robotic foundation models in particular, is key to realizing these goals.