On the Opportunities and Risks of Foundation Models

The societal impact of foundation models, referring both to the construction of the models themselves and their role in developing applications, requires careful examination. Specifically, we anticipate that foundation models will have wide-ranging societal consequences that are challenging to understand: foundation models are intermediary assets that are not directly deployed, but rather serve as a foundation that is further adapted. As a result, traditional approaches to reasoning about the societal impact of technology are likely complicated; societal impact is easier (but still difficult) to grasp for systems with well-specified purposes. In this chapter, we discuss how we may grapple with and beginning to understand the complexity of the societal impact of models foundation models. Specifically, we discuss (i) the harms with respect to inequity (§5.1: fairness) and misuse (§5.2: misuse), (ii) the impact with respect to the economy (§5.5: economics) and environment (§5.3: environment), and (iii) the broader considerations with respect to the law (§5.4: legality) and ethics (§5.6: ethics).

Inequity and fairness

Introduction

Foundation models have the potential to yield inequitable outcomes: the treatment of people that is unjust, especially due to unequal distribution along lines that compound historical discrimination (Hellman, 2021). Like any AI system, foundation models can compound existing inequities by producing unfair outcomes, entrenching systems of power, and disproportionately distributing negative consequences of technology to those already marginalized (Sweeney, 2013; Kay et al., 2015; Buolamwini and Gebru, 2018; Benjamin, 2019; Ajunwa, 2019; D’Ignazio and Klein, 2020; Crawford, 2021). Here we ask what fairness-related harms relate to foundation models, what sources are responsible for these harms, and how we can intervene to address them. The issues we discuss here are related to broader questions of algorithmic fairness and AI ethics (Corbett-Davies and Goel, 2018; Chouldechova and Roth, 2020; Hellman, 2020; Johnson, 2020; Fazelpour and Danks, 2021), race and technology (Benjamin, 2019; Hanna et al., 2020; Gebru, 2021; Field et al., 2021), and the coexistence of society and technology (Abebe et al., 2020).

Harms

Foundation models are intermediary assets with no specified purpose before they are adapted; understanding their harms requires reasoning about both their properties and the role they play in building task-specific models. We delineate intrinsic biases, i.e., properties of the foundation model that indirectly but pervasively affect downstream applications, and extrinsic harms, i.e., harms that arise in the context of specific downstream applications (Galliers and Spärck Jones, 1993).

Intrinsic biases

Properties of the foundation model can lead to harm in downstream systems. As a result, these intrinsic biases can be measured directly within the foundation model, though the harm itself is only realized when the foundation model is adapted, and thereafter applied, i.e., these are latent biases or harms (DeCamp and Lindvall, 2020). We focus on the most widely studied form of intrinsic bias, representational bias, specifically considering misrepresentation, underrepresentation and overrepresentation. People can be misrepresented by pernicious stereotypes (Bolukbasi et al., 2016; Caliskan et al., 2017; Abid et al., 2021; Nadeem et al., 2021; Gehman et al., 2020) or negative attitudes (Hutchinson et al., 2020), which can propagate through downstream models to reinforce this misrepresentation in society (Noble, 2018; Benjamin, 2019). People can be underrepresented or entirely erased, e.g., when LGBTQ+ identity terms (Strengers et al., 2020; Oliva et al., 2021; Tomasev et al., 2021) or data describing African Americans (Buolamwini and Gebru, 2018; Koenecke et al., 2020; Blodgett and O’Connor, 2017) is excluded in training data, downstream models will struggle with similar data at test-time. People can be overrepresented, e.g., BERT appears to encode an Anglocentric perspective (Zhou et al., 2021a) by default, which can amplify majority voices and contribute to homogenization of perspectives (Creel and Hellman, 2021) or monoculture (Kleinberg and Raghavan, 2021) (§5.6: ethics). These representational biases pertain to all AI systems, but their significance is greatly heightened in the foundation model paradigm. Since the same foundation model serves as the basis for myriad applications, biases in the representation of people propagate to many applications and settings. Further, since the foundation model does much of the heavy-lifting (compared to adaptation, which is generally intended to be lightweight), we anticipate that many of the experienced harms will be significantly determined by the internal properties of the foundation model.

Extrinsic harms

Users can experience specific harms from the downstream applications that are created by adapting a foundation model. These harms can be representational (Barocas et al., 2017; Crawford, 2017; Blodgett et al., 2020), such as the sexualized depictions of black women produced by information retrieval systems (Noble, 2018), the misgendering of persons by machine translation systems that default to male pronouns (Schiebinger, 2013; Schiebinger, 2014), or the generation of pernicious stereotypes (Nozza et al., 2021; Sheng et al., 2019; Abid et al., 2021). They can consist of abuse, such as when dialogue agents based on foundation models attack users with toxic content (Dinan et al., 2021; Gehman et al., 2020) or microaggressions (Breitfeller et al., 2019; Jurgens et al., 2019). All of these user-facing behaviors can lead to psychological harms or the reinforcement of pernicious stereotypes (Spencer et al., 2016; Williams, 2020).

In addition to harms experienced by individuals, groups or sub-populations may also be subject to harms such as group-level performance disparities. For example, systems may perform poorly on text or speech in African American English (Blodgett and O’Connor, 2017; Koenecke et al., 2020), incorrectly detect medical conditions from clinical notes for racial, gender, and insurance-status minority groups (Zhang et al., 2020b), or fail to detect the faces of people with darker skin tones (Wilson et al., 2019; Buolamwini and Gebru, 2018). As foundation models are more pervasively applied, including in high-stakes domains, these disparities can spiral into further, and more severe, harms. Koenecke et al. (2020) discuss how if African American English speakers cannot reliably use speech recognition technologies (e.g., due to inequities in underlying foundation models), this may mean they cannot benefit from certain derivative products (e.g., voice assistants, assistive technologies) and will be disadvantaged if these technologies are used to conduct interviews for employment or transcribe courtroom proceedings. More generally, characterizing these group-level harms (and working towards justice for those harmed) also requires the AI community to improve its understanding of group-based prejudice (Allport, 1954) and social groups: we point to relevant work in the social sciences and other communities on moving beyond binary treatments of gender (Lindsey, 2015; Westbrook and Saperstein, 2015; Richards et al., 2017; Darwin, 2017; Keyes, 2018; Hyde et al., 2019; Cao and Daumé III, 2020; Dinan et al., 2020), more nuanced treatments of race (e.g. Penner and Saperstein, 2008; Freeman et al., 2011; Saperstein and Penner, 2012; Saperstein et al., 2013; Penner and Saperstein, 2015; Field et al., 2021), better addressing intersectional identities (e.g. Crenshaw, 1989; Nash, 2008; Gines, 2011; Penner and Saperstein, 2013; Ghavami and Peplau, 2013; Bright et al., 2016; Buolamwini and Gebru, 2018; May et al., 2019; O’Connor et al., 2019; Guo and Caliskan, 2021), and more modern treatments of disability (e.g. Batterbury, 2012; Spiel et al., 2019; Hutchinson et al., 2020).

Additional considerations

To more completely understand the harms of foundation models, further documentation is required of both the intrinsic biases and extrinsic harms; future work should articulate the relationship between intrinsic biases and extrinsic harms (Blodgett et al., 2020; Blodgett et al., 2021; Goldfarb-Tarrant et al., 2021). This documentation requires centering stakeholders beyond academics and industry practitioners: the inequitable impact of foundation models will be experienced largely by minority populations, which are underrepresented in both academia and industry. For foundation models specifically, their creation and study likely will be conducted by those with the access and resources required, further emphasizing the importance of venues that center marginalized voices (D’Ignazio and Klein, 2020, §5.6: ethics). In particular, user studies of specific adapted models, when aggregated across applications, can provide compelling and individualized documentation of the harms that derive from the intrinsic biases of foundation models, all while centering individual users. In this way, we imagine the methodologies in human-computer interaction (HCI), with some adjustment to accommodate the abstraction involved in foundation models, will help center the voices of marginalized communities (further discussion in §2.5: interaction).

Sources

In order to fully characterize and properly intervene on the harms of foundation models, we must be able to trace their source to the properties of the foundation model and the adaptation process, and further decompose to the roles of individual sources of biases (Friedman and Nissenbaum, 1996). Source tracing is vital for attributing ethical and legal responsibility for experienced harm, though attribution will require novel technical research that foregrounds matters such as causality (Pearl, 2000) and influence (Koh and Liang, 2017).

Data

Data of several types shapes the behavior of applications, and the associated extrinsic harms, based on foundation models: the training data used to train the foundation model, the adaptation data used to adapt the foundation model, and test-time user data/interaction. For all of these data sources, the properties of the data (e.g., toxicity and hate speech (Henderson et al., 2017), abusive language (Waseem et al., 2017), microaggressions (Breitfeller et al., 2019), stereotypes (Voigt et al., 2018)) will manifest in the biases of the foundation model (and its adapted derivatives). Since the training data is the key data source that determines the foundation model and the associated intrinsic biases, we focus on the training data here. At present, the relationship between the training data, along with associated data practices (e.g., data curation, data selection, and data weighting (Paullada et al., 2020; Bender et al., 2021; Rogers, 2021)) and the intrinsic biases acquired by the foundation model remains unclear; future work is critically needed to clarify this relationship. Since foundation models generally require training data of immense scale, which poses clear challenges not only to its documentation (Bender et al., 2021) but also comprehensive scientific exploration to articulate the relationship of data biases and model biases, we anticipate new protocols are required to address this scale. Establishing scaling laws for bias, akin to those for accuracy metrics (Kaplan et al., 2020; Henighan et al., 2020), may enable systematic study at smaller scales to inform data practices at larger scales.

Modeling

Modeling decisions (e.g., training objective (§4.2: training), model architecture (§4.1: modeling), adaptation method (§4.3: adaptation)) influence the biases in foundation models and their derivatives, thereby affecting the experienced extrinsic harms. Existing work demonstrates that foundation models amplify training data biases, extending trends seen for machine learning and deep learning models (Zhao et al., 2017; Wang et al., 2019d; Jia et al., 2020; Hashimoto et al., 2018), though much still remains unclear about what and how model properties are responsible for this bias amplification. Further, given that applying foundation models directly may be infeasible (due to their scale), efforts to compress these models or make them more efficient also appear to amplify bias (Hooker et al., 2020; Renduchintala et al., 2021). Amplification may also be exacerbated by feedback loops, in which foundation models modify societal behavior and induce sociological changes, which modifies subsequent training data; feedback effects of this form tend to exacerbate inequity in other ML applications (Lum and Isaac, 2016; Ensign et al., 2018; Hashimoto et al., 2018). Beyond the explicit decisions made in training and applying foundation models, community values (Birhane et al., 2020) and norms (§5.6: ethics) both indirectly and implicitly (Liu et al., 2021b) shape decision-making in building models. As a result, measuring biases in conjunction with work introducing foundation models (e.g. Brown et al., 2020) and in standard benchmarks (Friedman and Nissenbaum, 1996, §4.4: evaluation), as well as conducting user studies with diverse user groups to document experienced harm, are steps towards ensuring that best practices actively emphasize the consideration of bias and inequity.

Modelers

As with all algorithmic systems, poor representation and diversity of stakeholders and marginalized communities in decision-making bodies that develop or apply foundation models is inherently problematic, and may contribute to greater experienced harm for these communities. While difficult to document, existing efforts to develop foundation models suggest this as a possibility: Caswell et al. (2021) demonstrate the flawed data handling of less-represented languages in the multilingual datasets used to train multilingual models and Hutchinson et al. (2020) show that models often contain undesirable biases towards disabled persons. In both instances, these biases and harms may have been noticed earlier by better representation of these parties in developer teams. Further, since end-users are likely more diverse than developers and may notice these concerns earlier, allowing for user feedback to contribute to foundation model design (§2.5: interaction) is an important direction forward.

Interventions and recourse

Addressing, mitigating, and rectifying the inequities associated with technology requires integrating social and technical methodologies (Abebe et al., 2020). For foundation models specifically, we consider both proactive methods, which change how models are developed and deployed to prophylactically reduce harm, as well as reactive methods, which respond to harm and make changes for the future. At its core, the abstraction of foundation models complicates both aspects: knowing if interventions at the level of the foundation level are successful in reducing harm requires downstream observations at the level of specific deployed applications and recourse in the event of harm requires upstream propagation of both feedback and accountability to foundation model providers.

Intervention

General principles that govern intervention on technological systems apply to the foundation model setting: identifying which sources are most responsible for bias or harm provides the evidence required for targeted action. For example, the urgency of calls for improved diversity in the teams that design, produce, and control technology (e.g., foundation models) and their applications (Longino, 1990; Harding, 2015; Nielsen et al., 2017; O’Connor et al., 2019; Hofstra et al., 2020; Katell et al., 2020) is further intensified if the lack of diversity is shown to relate to harm (Caswell et al., 2021). In addition, transparent documentation (e.g. Gebru et al., 2018; Bender and Friedman, 2018; Mitchell et al., 2019) and auditing (e.g. Raji and Buolamwini, 2019) are similarly critical in providing the impetus for intervention and change (Burrell, 2016; Lipton, 2018; Creel, 2020; Raji et al., 2020; Wilson et al., 2021). The scale of foundation models, as well as the specifics of their accessibility, introduce new challenges for existing protocols for documentation and auditing that we discuss further in §5.6: ethics.

To date, many of the interventions considered for reducing the inequitable impact of technology, including in the foundation model regime, are methods for technical mitigation that center the data (to obviate reflecting inequities or biases) and modelling decisions (to avoid amplifying data biases) involved. Of specific importance in the foundation model regime is recognizing that these mitigation approaches may target different steps in the pipeline such as the training data (e.g. Lu et al., 2020), modelling objectives (e.g. Zhao et al., 2018)), and adaptation methods and test-time use (e.g. Park et al., 2018; Zhao et al., 2019). As a result, different approaches may not only be more or less effective, but require action from different entities (e.g., foundation model providers vs. application developers) and more or less intensively affect the expensive training process for these models (e.g., changing the process of creating a foundation model vs. altering it post hoc). Technical intervention of this form may also target different goals: some interventions, such as changing the training data, aims to reduce intrinsic bias. On the other hand, most work on mitigation in algorithmic/ML fairness instead considers reducing outcome disparities in terms of model behavior, i.e., the outputs of downstream systems that more directly relate to extrinsic harm. Technical mitigation of all forms at present is severely limited: methods that measure or combat intrinsic bias are brittle or ineffectual (Gonen and Goldberg, 2019; Ethayarajh et al., 2019; Bommasani et al., 2020; Zhou et al., 2021b; Antoniak and Mimno, 2021), methods that measure or combat extrinsic outcome disparities may not align with stakeholder goals (Saha et al., 2020), and there is some evidence to suggest certain types of technical intervention may be simultaneously unsatisfiable (Corbett-Davies and Goel, 2018; Kleinberg et al., 2017), impossible (Lechner et al., 2021), or may even exacerbate inequity (Xu et al., 2021). In spite of this state of affairs, we continue to believe technical methods will still play an instrumental role in addressing the harms that arise in the foundation model regime; in general, we advocate for transparency, especially given that technical mitigation methods may not be able to achieve the intended goals. More broadly, claims of bias and bias mitigation must be made carefully to clearly communicate the status quo to various stakeholders with differing expertise (e.g., application developers building on top of foundation models and policymakers regulating the technology; (Nissim et al., 2020)).

Recourse

Unfortunately, proactive intervention is unlikely to fully resolve all potential harm or inequity that may arise due to foundation models. When harm arises, there is currently no widely-adopted (or legally required) framework for resolving the appropriate recourse for the harmed parties. While certain protocols may exist for specific applications, the abstraction of foundation models again introduces a disconnect: harms likely are partially attributable to both the foundation model providers and the downstream application developers, but allocating this responsibility to either party remains challenging. More simply, mechanisms are not in place to even communicate these harms to foundation model providers (even if feedback or complaints are raised to application developers). As a result, new norms and standards are needed on how feedback from application developers and end-users should reach upstream to the foundation model providers, how to determine the entities (e.g., foundation model providers, application developers) responsible for these harms, and the relationship to legal responsibility (§5.4: legality). To make progress on this matter, we encourage future work to consult the practices used in other domains (especially those with similar abstractions and multi-entity structures), and we anticipate any standards introduced will likely need to be reasonably dynamic, so that they can be synchronized with the rapidly changing status quo for these models and their applications.

Takeaways

Machine learning has an established trackrecord of inequitable impact, with much of the burden of its harms borne by marginalized communities. Foundation models introduce new challenges to this calculus but, ultimately, for their societal impact to be equitable, significant research and change is required to understand the harms they cause and to meaningfully address and rectify these harms:

• (1) The one-to-many nature of foundation models, i.e., the same few foundation models being used across many applications, means the intrinsic properties of foundation models pervade to many downstream applications. Pernicious biases in these models therefore have out-sized effect on the experienced harms.

• (2) Biases and harms in the foundation model regime originate from many sources (e.g., training and adaptation data, modelling and adaptation decisions, modeler diversity and community values). Attributing the sources for bias and harm is fundamental for questions of intervention and responsibility; attribution requires new technical research to be done reliably.

• (3) The inequities of foundation models are not inevitable, but addressing them requires a multi-pronged approach comprised of both proactive intervention (e.g., data-centric and model-centric changes) and reactive recourse (e.g., mechanisms for feedback and accountability).