The Past, Present, and Future of the Cognitive Theory of Multimedia Learning

I did not plan to spend a substantial portion of my academic career on developing the cognitive theory of multimedia learning. In this section, I briefly describe how it happened, including how I stumbled upon an appropriate name and a concise visual representation, how I searched for a conceptual framework, and how CTML has grown in terms of research base and design principles.

Stumbling Upon a Name for the Theory

An important challenge in theory development is to find a name that highlights the key concepts in the theory. Table 1 summarizes the names leading up to the cognitive theory of multimedia learning, which highlighted an evolving collection of inter-related concepts. I began with “model of meaningful learning” (Mayer, 1989), which focused on the external conditions for instruction for meaningful learning—namely, having potentially meaningful material, having learners who need help, having illustrations that provide help, and having a test that can detect meaningful learning outcomes. I changed the name to “model of conditions for effective illustrations” in Mayer & Gallini (1990) but retained the same set of conditions albeit in a different order.

How did I progress to the next levels? In continually discussing our theoretical account with students and colleagues, it became clear that this initial model captured the external factors involved in meaningful learning but did not adequately address the internal cognitive processes involved. If I wanted to know how meaningful learning works, I would have to consider the cognitive processes during learning. I was heavily influenced by three threads of scholarship I had been reading about—dual-coding such as articulated by Paivio (1986), limited working memory capacity such as articulated by Miller (1956) and Baddeley (1986) and Sweller (1999), and active cognitive processing during learning such as articulated by Wittrock (1974, 1989). First, I focused on dual-coding.

As a result of my dissatisfaction with a focus solely on external conditions of meaningful learning, the names shifted to focus on the internal conditions for meaningful learning by considering cognitive processing in the learner’s information processing system. Different names emphasized different aspects of cognitive processing such as having dual-channels for verbal and visual material, having limited capacity for cognitive processing, and engaging in generative processing during learning. First, I emphasized the concept of dual-coding (i.e., separate information processing channels for auditory and visual material) with names like “dual-coding model” (Mayer & Anderson, 1991, 1992) and “dual-processing model of multimedia learning” or “dual-processing theory of working memory” (Mayer & Moreno, 1998). The idea of separate channels for processing visual and verbal material was to become a central feature of CTML, as represented by the two rows in the current model.

Then, the name was broadened to differentiate among three cognitive processes during learning—selecting relevant material for further processing, organizing it into a coherent cognitive representation, and integrating it with relevant prior knowledge. Mayer (1996) referred to this idea as the “SOI model,” and other papers used the term “generative theory” (Mayer, 1997; Mayer et al., 1995) or “generative theory of multimedia learning” (Mayer, 1997; Plass et al., 1998) or “generative learning theory” (Fiorella & Mayer, 2015, 2016) or “generative theory of learning” (Mayer, 2010). The SOI model was a major conceptual breakthrough for me, and has remained at the core of CTML ever since, as represented by the arrows in the current model. In short, the SOI model represents the core cognitive processes that drive CTML.

Finally, we began using the name, “cognitive theory of multimedia learning” in Mayer et al., (1996, 1999), Mayer (1997), and Moreno & Mayer (2000). We also elaborated on the underlying ideas of the cognitive theory of multimedia learning in Mayer & Moreno (2003) and all editions of Multimedia Learning (Mayer, 2001, 2009, 2021) and all editions of The Cambridge Handbook of Multimedia Learning (Mayer, 2005, 2014; Mayer & Fiorella, 2022). This approach included the idea of limited capacity and the distinction among extraneous, essential, and generative processing in which cognitive capacity directed at extraneous processing reduced the capacity available for essential and generative processing.

How am I progressing beyond the current model of CTML? Research evidence came pouring in that alerted me to the idea that there may be more to multimedia learning than cognitive processing. We began to find evidence for the role of social process (e.g., how using conversational language can build social rapport) and affective processing (e.g., how the instructor’s gestures and tone of voice can affect learning), and evidence for the role of motivational factors (e.g., benefits of training for self-efficacy in multimedia lessons) and metacognitive factors (e.g., role individual differences in executive function in learning from distracting lessons). We are now grappling with how to represent these additions to CTML.

As we move to expand CTML, as summarized in Table 2, we supplemented CTML with “social agency theory” (Atkinson et al., 2005; Mayer et al., 2003), which incorporates social processes during learning, and with the “cognitive-affective theory of learning with media” (Moreno & Mayer, 2007) and the “cognitive-affective model of e-learning” (Lawson & Mayer, 2022; Lawson et al., 2021), which incorporates affective processes during learning.

The name changes reflect a shift from a focus on external conditions to internal processes ranging from dual-coding processing to generative processing to social and affective processing. Although it took us more than a decade to get there, throughout the twenty-first century, we have landed on the “cognitive theory of multimedia learning” as the name of our theory.

Inching Towards a Visual Representation of the Theory

Although it took many iterations to find a suitable name, we also struggled with finding an appropriate visual representation of the theory. I have found that visual representations help me better understand and improve on the theory, so I generally start with a visual representation and then express my ideas in words. In the case of CTML, it took many tries at building a flowchart that could represent the theory concisely and accurately. Our earliest attempts are shown in Figs. 2 and 3, which depict the external conditions for effective multimedia instruction—having meaningful text, having complimentary illustrations, having learners who need help, and having a test that taps meaningful learning (Mayer, 1989; Mayer & Gallini, 1990).

Next, we shifted from a vertical flowchart depicting external conditions of meaningful learning to a vertical flowchart depicting steps in a dual-coding model (in Fig. 4; Mayer & Anderson, 1992). In a further refinement of the dual-coding model, we flipped to a horizontal flowchart involving a visual channel and an acoustic channel (in Fig. 5; Mayer & Moreno, 1998). Around the same time, we developed a more inclusive flowchart based on a generative theory that broadened the cognitive processes to include selecting, organizing, and integrating, but without the dual channels (in Fig. 6; Mayer, 1996). Figure 7 shows a flowchart version of generative theory (with selecting, organizing, and integrating) that also begins to incorporate dual channels involving text and illustrations (Mayer, 1997; Mayer et al., 1995).

Finally, in Fig. 8 (Mayer, 2001; Mayer et al., 2001), we refined those previous flowcharts to include all three features of the theory: dual-channels as represented by an auditory row across the top and a visual row across the bottom; limited capacity as represented by boxes for sensory memory, working memory, and long-term memory; and generative processing as indicated by arrows for selecting, organizing, and integrating. This has become the stable flowchart representation we have used to depict the cognitive theory of multimedia learning throughout the twenty-first century. It is the single most important representation of CTML, and it continually helps me think about how multimedia learning works and the implications for instructional design. On reflection, it seems fitting that the most important statement of CTLM is itself a multimedia representation consisting of words and graphics.

Going beyond the standard flowchart for CTML in Fig. 8, we also have added some complementary flowcharts for social agency theory (Mayer, 2009), which adds social processing, and for the cognitive-affective model (Lawson & Mayer, 2022; Moreno & Mayer, 2007), which adds affective processing. The changes in our flowchart reflect refinements and additions in the theory, as we grappled with how to integrate an inter-related set of concepts about dual channels, limited capacity, and active processing.

Adding to the Research Base

Across four decades, the research base for the cognitive theory of multimedia learning has grown substantially, which enabled further theory development. Table 3 shows the number of experimental tests conducted by my colleagues and me as well as the number of multimedia design principles we have proposed based on those studies across the three editions of Multimedia Learning (Mayer, 2001, 2009, 2021). Starting with our first multimedia learning studies in 1989, we have been able to generate 15 evidence-based principles based on more than 200 experiments conducted by my colleagues and me.

Table 4 lists the principles that were included in each of the three editions of Multimedia Learning. As can be seen, we began mainly with principles aimed at minimizing extraneous processing—cognitive processing that does not support the instructional goal—such as eliminating unneeded material (i.e., coherence principle). Then, we added principles aimed at managing essential processing—cognitive processing aimed at representing the material in working memory—such as pausing a continuous video to create self-paced segments (i.e., segmenting principle). Lastly, we added principles aimed at fostering generative processing—cognitive processing aimed at making sense of the material—mainly with new technologies such as asking learners to summarize or explain what they are learning (i.e., generative activity principle).

Of course, the growth of the research base supporting CTML goes far beyond what our lab produces and includes an ever expanding network of researchers around the world. As summarized in Table 5, some of this work is described in the three editions of The Cambridge Handbook of Multimedia Learning (Mayer, 2005, 2014; Mayer & Fiorella, 2022).

This growth in the research base is supported by improvements in assessment of learning outcomes—especially the development of appropriate transfer tests. I am a strong proponent of replication and for searching for boundary conditions under which the various design principles apply. Overall, theory development depends on a storehouse of research evidence generated by labs around the world.