On the Opportunities and Risks of Foundation Models

Foundation models signal a paradigm shift where increasingly massive quantities of data are being “fed” to these models for improved adaptation performance (Devlin et al., 2019; Radford et al., 2021; Tolstikhin et al., 2021) with the overarching rule-of-thumb being "the more data the better" (Kaplan et al., 2020). As previous sections have mentioned, this focus on data curation has raised concerns around the foundation model data lifecycle including (1) managing the data at such a large scale (§1: introduction), (2) integrating data across new modalities (§2.3: robotics, §3.1: healthcare), (3) reasoning over licensing and governance regulations — especially when considering the massing web-crawls used in foundation models training — (§3.1: healthcare, §5.4: legality), and (4) understanding the data quality (§4.4: evaluation).

While foundation models add new and difficult facets to these challenges, we see parallels between these issues and core challenges in communities such as data management and data analytics as well as industrial ML pipelines. For example, data management has long studied scalable declarative systems for data analysis, versioning, provenance, and integration—addressing challenges (1) and (2) (Zaharia et al., 2012; Cudré-Mauroux et al., 2009; Stonebraker and Weisberg, 2013; Stonebraker and Ilyas, 2018; Hellerstein and Stonebraker, 2005). Industry has pipelines dealing with challenge (3) to manage diverse data licenses and help mitigate data violations. There is an entire ecosystem of research and systems tackling challenge (4) to support interactive data analytics and visualization (Hohman et al., 2020). While these solutions are not necessarily "foundation model-ready", we believe a path to better management of the foundation model data lifecycle should take inspiration from these existing systems.

In this section, we address managing the foundation model data lifecycle. We first outline four desiderata including data management at scale, support for heterogenous data sources, data governance, and data quality monitoring. We then envision how all of these requirements can be integrated into a holistic data management solution called a data hub. The data hub is simply a data management toolkit that can be used by the private or public sectors to better support the interactive management of the foundation model data lifecycle.

Data Management Desiderata

Current practices in foundation model development are generally ad-hoc across the entire lifecycle from data curation and data documentation to model monitoring and patching (Gebru et al., 2018; Bandy and Vincent, 2021; Bender and Friedman, 2018). Research in the data management community has shown that well-defined data management platforms facilitate ML model development at scale through data ingestion, data versioning, data provenance, efficient analysis, and model monitoring (Hellerstein and Stonebraker, 2005; Agrawal et al., 2019; Vartak et al., 2016; Ikeda and Widom, 2010). Taking inspiration from the data management community, we consider core desiderata when building a holistic data management platform for foundation models.

• (1) Scalability. Foundation models are being trained on increasingly massive quantities of data (Kaplan et al., 2020) with the WuDao 2.0 model being trained on 4.9 TB of multi-modal data. This scale is expected to increase as most recent models are trained largely on public facing datasets. Public data represents an extremely small fraction of data compared to the petabytes of business and personal data collected every day and used in industrial foundation model pipelines (Marr, 2017). There is therefore a growing need for highly scalable techniques that can handle multi-modal foundation model datasets.

• (2) Data integration. Recent work using foundation models demonstrates that leveraging integrated structured and unstructured data can help models better generalize to rare concepts (Orr et al., 2020) and improve factual knowledge recall (Orr et al., 2020; Logeswaran et al., 2019; Zhang et al., 2019a; Peters et al., 2019; Poerner et al., 2020). Despite these recent successes, integrating datasets for foundation models remains a challenge. Many works use unstructured text data with structured entity knowledge or image data (Antol et al., 2015). There is a growing need to integrate datasets across diverse modalities such as text, video, eye-tracking (Hollenstein et al., 2020), and robotic simulations (Lynch and Sermanet, 2021) (see §2.3: robotics). We need data-integration solutions that can be applied at an industrial scale to multiple modalities and to multiple domains, such as government, business, and science.

• (3) Privacy and governance controls. The training data used for foundation models may risk the violation of the privacy of data subjects; their data may be disclosed, collected, or used without their consent (Jo and Gebru, 2020) or outside the context for which consent was originally given. The issue of consent and use is especially relevant for foundation models where downstream applications cannot always be anticipated. As explained in §5.4: legality, these issues are compounded with the prevalence of web scraped datasets for foundation model training. As there are still open legal questions about how web-crawled data will be governed and copyrighted, the consequences of using web data remain unclear to foundation model providers in the public and private sector. We need tooling to help foundation model providers adapt to emerging regulations and guidelines to ensure safe and responsible data management.

• (4) Understanding data quality. Data quality impacts model performance (Lee et al., 2021b); however, toolkits or methods to systematically and scalably understand the training data and relevant data subsets are still in their infancy. The data creation process can be messy, and the data can contain different types of biases (Blodgett et al., 2020; Bender et al., 2021) (see §5.1: fairness) and consist of poisoned, false, or duplicated information (Chang et al., 2020; Carlini and Terzis, 2021; Buchanan et al., 2021; Lee et al., 2021b). Data is also continuously updated and refined (Kiela et al., 2021) and may have emergent entities (Fetahu et al., 2015), distribution shift (Chen et al., 2021a), and concept meaning shift (Kenter et al., 2015; Lazaridou et al., 2021). Further, once deployed, foundation models may present undesirable behavior on critical, fine-grained sub-populations of data that foundation model providers need to detect and mitigate (Goel et al., 2021; Hohman et al., 2018; Ré et al., 2019; Oakden-Rayner et al., 2019). We need toolkits that can detect and potentially mitigate different types of undesirable data to improve model performance in an interactive and iterative fashion. Such toolkits also need to adapt to the dynamical nature of training data.

Data Hub Solution

Pulling on years of work from data management, data science, and data analytics, we envision a foundation model lifecycle data management solution, which we term a data hub. While examples of ML-focused data hubs as well as more traditional data management systems exist, they either (1) do not treat data integration as a first class primitive, (2) do not natively support the end-to-end lifecycle with model predictions, or (3) do not allow for interaction-driven data curation and refinement, where foundation model providers can dynamically explore and update possible datasets subject to access control guidelines. We now discuss how the data hub addresses the four desiderata.

Data scale

To address the management at scale challenge, the data hub will need standard data management solutions (Armbrust et al., 2009) such as infrastructure to store and maintain large-scale datasets as they change over time and scalable interfaces to query, select, and filter datasets. The hub should support heterogenous compute as well as cloud infrastructure to support scalable solutions in different environments.

Data integration

The hub should incorporate data integration as a first class citizen. It will need advanced data integration solutions (Stonebraker and Ilyas, 2018; Abiteboul, 1997; Dong et al., 2020; Rekatsinas et al., 2017a) to allow for the merging of structured and unstructured knowledge across modalities and domains. Further, this implies the hub will need to support storing and querying over heterogeneous datasets and sources.

Access control

Considering the access controls of the hub, the hub will need to support diverse documentation, e.g., dataset sheets (Gebru et al., 2018) or data statements (Bender and Friedman, 2018), to allow data curators to reflect on their processes and be transparent about the intended use cases, potential biases, and limitations of their dataset. The data hub will need to decide which documentation is required for data to be uploaded (e.g., the data source and data description) and which information is recommended (e.g., what tasks the data could be used for). Furthermore, documentation may need to be updated as datasets evolve (Goel et al., 2021).

Data sources are often associated with licenses, and the hub will need to integrate different sources with different legal concerns and conditions (Masur, 2018). Further, certain datasets have legal guidelines to protect the privacy of the data subjects. The hub will need methods to ensure a dataset does not release personally identifiable information (PII), that the aggregation of anonymized or de-identified data does not release PII, and that the data subjects have given informed consent for their data to be disseminated.

Pulling on ideas from data integration (Rekatsinas et al., 2017b), the hub should support mechanisms to enable efficient and safe maintenance and sharing of data resources. Especially as the legality of certain public datasets (e.g., web dumps) are still being decided (§5.4: legality), the hub critically needs tooling to help identify licensing violations and mitigate the impact of any governance violation. As certain violations will likely relate to model behavior, we need systems to support better understanding of model behavior, as we describe next.

Data quality tooling

Drawing on the field of data analysis and exploration, as users interactively select, filter, and refine the data to use for training or adaptation, the hub will need tools to quickly understand a user’s current dataset and its impact on model behavior (Hohman et al., 2020). Furthermore, these systems can allow end-to-end foundation model monitoring by incorporating model performance through recent work on slice (sub-population) finding (Chung et al., 2019), model validation on relevant subsets (Goel et al., 2021; Ribeiro et al., 2020), and data valuation (Ghorbani and Zou, 2019). Recent works also present methods that use the model to detect which subpopulations of data contribute the most to a given output to further aid model debugging (Keskar et al., 2019).

Once users can monitor model behavior—especially on rare, yet critical sub-populations—, the hub should provide methods and guidance for users to maintain models by correcting model errors. Although “model patching” (Goel et al., 2020a) is still an open problem, the work of (Orr et al., 2020) provides a first description of using data engineering to maintain a production self-supervised system that corrected for undesirable behavior through changes to the data, not model. We believe the data hub will need to support interfaces for users to inject targeted data modifications for model maintenance.

We also acknowledge that data curation and exploration are not performed in isolation, and believe the data hub should support a community around sharing useful metrics and analysis pipelines. Inspired by similar community sharing platforms like Hugging Face’s ModelHub or Tableau Public’s visualization sharing platform, we want users to share insights about foundation model training data.

Open questions

Although our described data hub is inspired by existing toolkits and solutions, we do not believe they are all ready for the challenges of foundation models. In particular, some open questions revolving around designing a data hub are:

• • How should we support data versioning so datasets can be updated while maintaining old versions for reproducibility (Agrawal et al., 2019)? Once models are deployed and error buckets are identified, datasets may need to be updated to include more examples from these error buckets. How should these new, targeted examples be collected?

• • As described in §4.2: training, we imagine fewer models will be trained from scratch and more will be fine-tuned. How do we support provenance or lineage information to understand where the original data came from, while maintaining subject privacy (Chen et al., 2015a)?

• • In the public sector, a data hub may be organized and run by an open-source community of individuals consisting of data curators and foundation model providers. In this setting, answers to questions such as who stores the data? who pays for any compute? who is liable if licensing is violated? are particularly murky. How can the data hub provide that right tooling so that once answers to such questions are resolved, they can be operationalized with ease?

• • What is the right set of statistics over the data to provide adequate documentation, without being too costly or difficult to obtain?

• • How can a data hub support targeted data modifications such as augmentation (Ma, 2019; Shorten and Khoshgoftaar, 2019) or data programming (Ratner et al., 2017)?

• • How can monitoring toolkits better detect when a foundation model needs to be updated due to poor performance on dynamically changing evaluation data?

Our vision for a data hub is not complete or fully detailed. However, we present initial thoughts on data challenges, and one solution to prompt thinking for how to improve data management for the foundation model lifecycle.