On the Opportunities and Risks of Foundation Models

As central components in critical data-driven decision-making systems, machine learning models must address a variety of security and privacy threats. These threats can be characterized using the traditional “CIA triad” of computer security. ML systems should protect the Confidentiality of user data against inference and reconstruction attacks (Fredrikson et al., 2015; Shokri et al., 2017; Carlini et al., 2019; Carlini et al., 2021). Moreover, the secrecy of trained models themselves can be at risk of model stealing attacks (Tramèr et al., 2016; Papernot et al., 2017). The Integrity of ML systems can be compromised by adversarial examples (Biggio et al., 2013; Szegedy et al., 2014) and data poisoning attacks (Biggio et al., 2012; Chen et al., 2017). Finally, resource-depletion attacks (Shumailov et al., 2020; Hong et al., 2020a) can threaten the Availability of ML systems.

In regard to these threats, we posit that the security role of foundation models in future machine learning systems will be akin to the role played by the operating system in traditional software systems. Due to its generality and ubiquity, a foundation model may become a single point of failure and thus a prime target for attacks against applications derived from this model. In turn however, a foundation model imbued with strong security and privacy properties could form the backbone for the design of a variety of secure and reliable ML applications. Of course, these applications may still have to be designed to enforce specific security and privacy guarantees (in the same way that software designers cannot rely on a secure operating system to protect against all security risks).

Risks

Single points of failure

A foundation model that is adapted to a variety of applications represents a single point of failure for these applications. For example, data poisoning attacks on a foundation model, where an adversary inserts malicious examples into the training data, might impact all adapted applications as well. Similarly, adversarial examples against a foundation model (i.e., small input perturbations that cause the model to output very different features) could more easily transfer to adapted applications. Wallace et al. (2019) even find that a single adversarial trigger added to any input can cause language models such as GPT-2 to output a predefined piece of text. A foundation model can also become a single point of failure for data privacy. If a foundation model is pretrained on a company’s private data and the model memorizes part of this data, all downstream applications could run the risk of exposing this data (Carlini et al., 2021). The provider of a foundation model may also be a single point of trust for the privacy of application data. For example, the current API for GPT-3 requires that all (potentially sensitive) data used for fine-tuning or inference be uploaded to OpenAI’s servers. Designing a foundation model service that avoids this centralization of trust is an interesting problem.

If the parameters of a foundation model are public, model stealing attacks on adapted applications could be facilitated, as the attacker only needs to reverse-engineer the “delta” with respect to the public foundation model (Krishna et al., 2019) (e.g., a linear model trained on features extracted from a public frozen model).

Finally, denial-of-service attacks on the foundation model provider could also be a concern and might be exacerbated by querying the model with special high-cost inputs (Shumailov et al., 2020).

Data poisoning

Successful foundation models have so far been trained on large and often uncurated datasets scraped from the Web (Radford et al., 2021; Radford et al., 2019). This permissive data collection — coupled with a lack of direct training supervision — facilitates poisoning attacks on a foundation model’s training data (e.g., injecting hateful speech targeted at a specific individual or company into a few outbound pages from Reddit). Worse, the power of poisoning attacks may be exacerbated by the growing size and accuracy of today’s models (Carlini, 2021).

To illustrate, Schuster et al. (2021) show that a code auto-completion system trained with GPT-2 on Github data can be poisoned into suggesting insecure code snippets with the injection of only a few malicious files. Carlini and Terzis (2021) further show that targeted attacks against CLIP-style (Radford et al., 2021) models require modifying as little as two out of 3 million training examples.

Function creep & dual use

Foundation models learn general features that enable them to be easily adapted to a variety of tasks. This flexibility, however, raises concerns that foundation models could be used beyond their originally foreseen purposes — a risk commonly referred to as function creep or dual use. Examples of function creep in machine learning include overlearning (Song and Shmatikov, 2019) and adversarial reprogramming (Elsayed et al., 2018).

To illustrate, CLIP was originally trained to solve the generic task of predicting image-text pairs, but in doing so also learned to capture rich facial features (Goh et al., 2021). While CLIP’s ‘‘model card’’ explicitly places facial recognition and other surveillance technologies as out-of-scope, CLIP can certainly be re-purposed for such tasks (Radiya-Dixit and Tramèr, 2021). This example illustrates that it may be challenging to constrain (or even foresee) the possible nefarious uses of a foundation model when it is designed. §5.2: misuse provides further discussions on dual (mis)use of foundation models.

Multimodal inconsistencies

Multimodality may increase the attack surface of foundation models, by enabling adversaries to exploit inconsistencies across modalities. The possibility of such attacks was demonstrated in an (in)famous example of CLIP classifying an apple with the word “iPod” stuck to it as an iPod (Goh et al., 2021). More generally, whenever a concept can be expressed using different modalities, inconsistencies across these modalities may be exploitable.

Such inconsistencies are particularly concerning when a foundation model is adapted to a task that primarily relies on only one of the learned modalities. For example, consider using features extracted from CLIP for facial recognition. This is a purely visual task, yet the adapted model’s features will still be sensitive to textual signals (thus, an attacker might be able to evade facial recognition by wearing clothes with imprinted text). Alternatively, consider an autonomous driving system (an application that also relies primarily on vision) that sees a billboard with the word “green” on it, and mistakenly interprets this as a green light.

Opportunities

Security choke points

If adapted applications can inherit vulnerabilities from a foundation model, they can also inherit desirable security characteristics — such as robustness to adversarial examples or poisoning attacks. Foundation models could thus serve as security choke points. For example, a model robust to adversarial examples can retain its robustness when it is adapted to other tasks (Shafahi et al., 2019). Similarly, a foundation model provider that can (somehow) defend against poisoning, model-stealing or resource-depletion attacks could then provide such security guarantees for its customers’ applications.

The tradeoff between a foundation model’s role as a single point of failure or as a security choke point is reminiscent of similar security tradeoffs in other abstraction layers in the software stack (e.g., an operating system, database system, or a Web browser). By virtue of serving many different applications, an abstraction layer is a prime target for attack, but can typically also leverage far greater resources to enhance its security compared to any single application.

Cheaper private learning

Current foundation models are often trained by amassing vast amounts of data from publicly available sources (e.g., from the open Web). This practice may raise concerns about privacy — in the broad sense of taking user data out of its intended context (Nissenbaum, 2004; Carlini et al., 2021). While some existing works aim to mitigate a model’s propensity to memorize training data (e.g., by de-duplicating training data (Lee et al., 2021b), or by pretraining under differential privacy (Anil et al., 2021)), such solutions are unlikely to meet the broad privacy expectations that users could associate with text data (Brown et al., 2022). On the other hand, public pretraining could also end up being a win for user privacy in applications that handle scarce and sensitive data (e.g., in healthcare).

As an example, consider the problem of training a differentially private model (Dwork et al., 2006) for a healthcare task. Training such a model “end-to-end” (i.e., without leveraging any pretraining) to a decent privacy-utility tradeoff currently requires vast amounts of privacy-sensitive data (McMahan et al., 2018; Basu et al., 2021). In contrast, a foundation model pretrained on public data in many cases could be adapted to perform specific tasks with significantly less confidential data (Bommasani et al., 2019; Tramèr and Boneh, 2021; Li et al., 2022; Yu et al., 2022).

Robustness to adversarial examples at scale

There is evidence suggesting that training a model that is robust to adversarial examples requires vastly more data compared to standard training (Schmidt et al., 2018), but that unlabeled data may suffice to bridge this gap (Carmon et al., 2019; Uesato et al., 2019). Moreover, increasing model size and capacity (i.e., over-parameterization) has also been shown to be necessary for achieving adversarial robustness in some settings (Madry et al., 2018; Bubeck and Sellke, 2021). Understanding how best to leverage over-parameterization and unlabeled data to achieve adversarial robustness is an important direction for future research. Given their unprecedented scale (both in terms of model size and training set size), foundation models are uniquely positioned to benefit from this line of inquiry.

Despite their unprecedented scale, current foundation models unfortunately see little gains in robustness to worst-case adversarial perturbations (Fort, 2021; Wallace et al., 2019). However, multimodal models such as CLIP are surprisingly robust to (non-adversarial) distributional shifts (see §4.8: robustness). Whether these gains in distributional robustness can translate to increased resilience against real-world attacks is another exciting open question. Particularly in settings where adversaries are subject to various constraints (e.g., limited query access or computational budget), there is reason to be optimistic that enhanced distributional robustness could lead to concomitant gains in overall security — even if the foundation model remains vulnerable to worst-case “white-box” attacks.