On the Opportunities and Risks of Foundation Models

In this section, we consider misuse of foundation models — situations where people use foundation models as they are intended to be used (e.g., to generate language), but where their capabilities are intentionally leveraged to cause harm to populations or individuals. This definition positions misuse concerns between those of inequity (where models can cause harm without bad intentions; §5.1: fairness) and security (where bad actors exploit unintentional abilities or vulnerabilities in models to cause harm; §4.7: security). Below, we outline how foundation models both enable new forms of misuse and support new tools for misuse detection and mitigation.

Foundation models will be misused for harmful purposes

Advances in the scale (§4.2: training), multimodality (§4.1: modeling), and adaptivity (§4.3: adaptation) of generative foundation models will allow them to be misused to generate high-quality, cheap, and personalized content for harmful purposes. In this section, we discuss these three dimensions within the context of two examples of malicious activity: manipulative content creation and harassment.

Content quality

Foundation models are capable of automatically generating much higher-quality, human-looking content than prior AI methods. They may empower disinformation actors, where states, for example, create content to deceive foreign populations without being transparent that the content is linked to a state. Currently, creating this content often requires hiring people who speak the language of the population being targeted. Governments may outsource content production to native speakers in the country they are targeting,, but this decision causes real risks for operational security. Foundation models will allow for the creation of content that is often indistinguishable from content created by humans (Kreps et al., 2020; Clark et al., 2021) — and indeed it will be able to do this for a wide variety of languages — enabling both goals of creating content that resonates and maintaining operational security.

In addition to deceiving foreign populations, foundation models’ ability to generate high quality synthetic images (deepfakes) or text may be abused to harass individuals. Deepfakes have already been used for the purpose of harassment. For example, Rana Ayyub, an Indian investigative journalist, was targeted by a high-quality deepfake that superimposed her face onto a pornographic video, leading her to leave public life for months. Because foundation models are often multimodal (§4.1: modeling), they could similarly impersonate speech, motions, or writing, and potentially be misused to embarrass, intimidate, and extort victims.

Cost of content creation

Foundation models will substantially decrease the costs of content creation, further lowering the barrier to entry for malicious actors to carry out harmful attacks (Brundage et al., 2018). The budget for one 2017 influence operation that originated in Russia and targeted Americans was $12.2 million (DiResta et al., 2018). More recently, individuals in Russia paid $75-$200 per article to American freelancers as part of a disinformation campaign. Foundation models will lower these marginal costs. While foundation models, such as GPT-3, may make mistakes when generating content (Buchanan et al., 2021), it will be more feasible to hire a small number of editors to fix them than to hire content creators directly. Initial costs to train foundation models are more significant (§4.5: systems), but these expenses should be manageable for most state actors (Buchanan et al., 2021).

In addition to monetary cost, foundation models require fewer technical skills to achieve high-quality results. Current tools, such as video editing software, can enable credible photo or video deepfakes, but require several hours of a skilled user’s time to yield quality content. Foundation models lower this barrier to use: their few-shot adaptation capabilities (§4.3: adaptation) enable new modes of interaction for application users (§2.5: interaction) that will allow users to rapidly iterate for content creation.

Personalization

Foundation models will reduce obstacles to creating personalized content. For example, disinformation from Russian individuals that targeted the US in 2016 included highly customized content. Social media posts were crafted to push narratives about Syria (e.g., the U.S. should get out of Syria) that resonated with Black Lives Matter activists (DiResta et al., 2018) (e.g., by suggesting that the U.S. should focus on issues facing the Black community in America, and not on issues in Syria). The same narratives were repackaged to resonate with Texas secessionists (DiResta et al., 2021). Such a content creation endeavor is costly and time consuming. Foundation models will allow for similar activity, but at scale due to the low cost of adaptation (§4.3: adaptation).

In addition to foundation models allowing an actor to personalize content for niche audiences, they also allow an actor to personalize content to target a single individual --- a capability that can be abused by harassers. Foundation models that condition their generations on personal attributes or information can create realistic personalized content, which could be more embarrassing, place victims in more danger, and lead to more successful extortion attempts.

Foundation models will be powerful detectors of harmful content

While the generative capabilities of foundation models will provide ample misuse opportunities, these same abilities may make them strong detectors of harmful content. While these capabilities are equally relevant for detecting human- and model-generated content, we focus on the detection of model-generated content in this section. First, we outline the challenges that current manual detection approaches will face in discovering harmful misuses of foundation model. Then, we propose how the interactive and multimodal representation capabilities of foundation models may make them powerful tools for automatic detection of harmful content. Finally, we discuss the risks associated with deploying automatic detection models in online settings to combat potential foundation model misuse.

Rethinking human interventions

Currently, malicious practices are frequently uncovered (and on social media, sometimes removed) by humans searching the internet to uncover content origination. For example, fake social media profiles commonly steal profile photos from dating sites, which are discoverable through reverse image searches. Similarly, disinformation websites frequently use plagiarized content to mask deceptive content (DiResta and Grossman, 2019), which is easily identified by conducting internet phrase searches. Foundation models will limit the efficacy of these detection strategies. Already, relatively unsophisticated disinformation campaigns have leveraged AI-generated photos to remove the possibility of discovery through reverse image search. Tools for assessing whether these photos are AI-generated are available, but foundation models will complicate this work — for text and video as well — challenging manual human discovery techniques (Ippolito et al., 2020; Clark et al., 2021).

Foundation models as detectors

The same abilities of foundation models that make them strong generators of creative content may make them strong detectors of model-generated content. Existing works demonstrate that foundation models can be adapted to detect disinformation from text generators (Zellers et al., 2019b) — which generate statistical textual artifacts (Holtzman et al., 2020) — and that they can be used to evaluate the toxicity levels of their own generations using prompt questions (Schick et al., 2021). Below, we describe how future foundation models will enable more powerful detection systems of machine-generated, harmful content.

Improvements in the interactive and multimodal interfaces of foundation models will provide new opportunities to improve detection of foundation model misuse for harmful content generation. Current statistical detectors must be retrained and re-deployed to integrate new knowledge about the textual content of misuse strategies (Dinan et al., 2019). The rapid learning capabilities of foundation models (§4.3: adaptation) may allow them to adapt from human feedback to new misuse strategies that the foundation model was not initially trained to recognize (Lee et al., 2021a).

Simultaneously, the multimodal abilities of foundation models will enable more expressive representation of misuse ecosystems. Prior work has explored how misinformation spreads more rapidly across social networks than authentic content (Starbird et al., 2018; Vosoughi et al., 2018), yielding recognizable signatures when analyzed retrospectively. The multimodal capabilities of foundation models could allow them to jointly learn representations of harmful content and its typical dissemination signature on social networks. These joint representations could provide powerful tools for predicting whether certain types of automatically-generated content are indicative of misuse behavior.

Risks of foundation models as automatic detectors

Improvements in automatic detection systems for both model-generated and human-generated harmful content will make these systems more prevalent online, yielding potential negative consequences. Any detection system will have false positive cases where human-generated fair content will be flagged as harmful (Sap et al., 2019; Xu et al., 2021). The rate at which algorithmic false positives affect users (or groups of users) may cause downstream harm (§5.1: fairness). The adaptive capabilities of foundation models should make systemic false positives easier to address as the model can be locally edited to re-classify those examples (§4.3: adaptation). However, corner cases will likely not be prioritized and recourse will be challenging in these situations.

More broadly, wide-scale deployment of misuse detection systems may engender an “arms race” between harmful content generators and detectors. Most content generators that use foundation models will lack the resources to develop them individually, and will use systems deployed by larger entities. While terms of use policies should outline acceptable uses of these systems (§5.6: ethics), deployers of foundation models will also need internal detection systems to identify misuse of their products and mitigate them (§5.4: legality). However, there will be fewer controls for misuse actors with the resources to develop their own foundation model-based content generators, putting pressure on platforms to curate the content shared through their distribution channels. Optimistically, content platforms encompass some of the most well-capitalized firms in the world. Their resources may enable the development of detectors beyond the capabilities of most individual misuse agents. This resource advantage could disincentivize individual foundation model development due to the high costs of repeatedly training these systems at scale. However, many instances of foundation model misuse could still be successful even without the largest foundation models to power them, particularly as attackers may leverage the interactive capabilities of foundation models to rapidly generate content that can evade detection.