How effective is second language incidental vocabulary learning? A meta-analysis

Literature search

As suggested by literature search guidelines (In'nami & Koizumi, Reference In'nami and Koizumi2010; Plonsky & Oswald, Reference Plonsky, Oswald and Plonsky2015), we identified articles relevant to the current meta-analysis through searching the following databases: the Education Resources Information Center (ERIC), Linguistics and Language Behavior Abstracts (LLBA), ProQuest Dissertations and Theses, PsycINFO, Google, and Google Scholar. We also searched 19 journals in second language acquisition and language teaching and learning, where vocabulary studies are widely published, using the search function on each journal's website: Annual Review of Applied Linguistics, Applied Linguistics, Applied Psycholinguistics, Canadian Modern Language Review, ELT Journal, Foreign Language Annals, International Journal of Applied Linguistics, International Review of Applied Linguistics in Language Teaching, Language Learning, Language Teaching, Language Teaching Research, Modern Language Journal, Reading in a Foreign Language, Reading Research Quarterly, RELC Journal, Second Language Research, Studies in Second Language Acquisition, System, and TESOL Quarterly. Abstracts published up to October 2022 were targeted using various combinations of the following key words: “incidental (OR contextualized) vocabulary (OR word OR lexical) learning (OR acquisition),” AND “reading OR listening OR reading while listening OR viewing.” A total of 2,853 reports, including both published or unpublished articles (i.e., dissertations and conference presentations), appeared initially qualified for the meta-analysis. The second author read each abstract to remove duplicates and papers other than empirical studies (e.g., review papers, meta-analysis studies, commentary). Full manuscripts for the remaining reports were retrieved and then screened according to the following selection criteria.

Inclusion criteria

Eight selection criteria were created to assess the identified studies:

• 1. The study measured incidental vocabulary learning gains from comprehension-based meaning-focused learning conditions in which target words were not subject to explicit manners of instruction. In other words, studies using language-focused activities such as word card learning (see Webb et al. (Reference Webb and Webb2020) for a meta-analysis of intentional vocabulary learning) were excluded (see Webb & Nation (Reference Webb and Nation2017) for a list of language-focused activities).

• 2. Adopting a methodological operationalization of incidental learning (Hulstijn, Reference Hulstijn and Robinson2001; Uchihara et al., Reference Uchihara, Webb and Yanagisawa2019), the current meta-analysis included studies in which the participating learners were not told of an upcoming vocabulary posttest subsequent to the treatment.

• 3. The study focused on vocabulary learning through meaning-focused input activities such as reading, listening, reading while listening, and viewing. This meta-analysis focused on studies in which the perceived goal of task completion among students was to comprehend the content of reading, listening, or viewing materials rather than to learn new vocabulary.

• 4. The study did not use glossing or captioned videos in the treatment condition. Meta-analytic reviews examining glossing and captioning effects on vocabulary learning are available elsewhere (see Yanagisawa et al. (Reference Yanagisawa, Webb and Uchihara2020) and Ramezanali et al. (Reference Ramezanali, Uchihara and Faez2020) for glossed reading; Montero Perez et al. (Reference Perez, Van Den Noortgate and Desmet2013) and Reynolds et al. (Reference Reynolds, Cui, Kao and Thomas2022) for L1 and L2 subtitling).

• 5. The study ensured that prior knowledge of target words was little to none by using one of the following design features: using pseudowords, pilot testing target words, or administering pretests.Footnote 1

• 6. The study concerned L2 vocabulary acquisition, not L1 acquisition.

• 7. The study adopted a between-participants design and included a control condition in which participants were not expected to be exposed to target words. The comparison between experimental and control conditions is essential in order to control for practice effects (i.e., an increase in learning gains owing to multiple exposures to the same target items through multiple test-taking opportunities).

• 8. The study reported effect sizes or sufficient information to calculate effect sizes (e.g., means, standard deviations, mean differences, and sample sizes).

It should be noted that we excluded within-comparisons studies that did not use control groups. Including pretest-posttest designs without control groups might involve testing effects on results, making it difficult to attribute the findings to the learning conditions. Moreover, to minimize publication bias—the fact that studies finding large effect sizes tend to be published—this meta-analysis included unpublished studies such as M.A. and Ph.D. theses and conference presentations.

Studies satisfying all eight criteria were included in the current meta-analysis. When additional information was necessary to complete the analysis, we contacted authors and received information from five researchers that allowed inclusion of their data in the meta-analysis (i.e., Ana Pellicer-Sánchez, Marije Michel, Nina Daskalovska, Niousha Pavia, and Yanxue Feng). In total, 24 studies were identified and submitted for the subsequent coding procedures (see Supplementary Appendix 1 for the included studies).

Coding

Table 1 displays the coding scheme specifying publication information, moderator variables related to learner, material, activity, and methodology, and dependent measures for both first and follow-up posttests. We first coded all 24 studies according to our coding scheme, which produced a total of 53 effect sizes. However, some of these effect sizes came from performance based on the same participants in experimental and/or control groups, causing the issue of multiple effect sizes—that is, meta-analysis based on multiple effect sizes from the same participants violates independence of observations (Plonsky & Oswald, Reference Plonsky, Oswald and Plonsky2015). To address this problem, we averaged multiple effect sizes prior to the meta-analysis (In'nami et al., Reference In'nami, Koizumi, Tomita, McKinley and Rose2020). This averaging method enables the meta-analytic outcomes to be comprehensive without any unnecessary loss of valuable data. As a result, for the first posttest, 29 independent effect sizes with 2,771 participants (1,254 in control and 1,517 in treatment groups) were selected and meta-analyzed in the current study. For the follow-up (delayed) posttest (the average retention interval = 34.1 days), nine out of the 29 effect sizes (N = 741; 325 in control and 416 in treatment groups) were available and included for analysis to compare the data with the results based on the first posttest scores. The completed coding sheet and raw data files are available via OSF at https://osf.io/92r3t/.

In order to establish the reliability of the coding procedures, all 24 studies were coded independently by the two researchers whose expertise lies in L2 vocabulary studies. Following Boulton and Cobb's (Reference Boulton and Cobb2017) approach, the number of discrepancies between the two researchers’ coding was counted, and the agreement was rated at 98%. All disagreements and ambiguities were resolved through discussion.

Coding of moderator variables

Eight moderator variables categorized into three groups (learner, material and activity, and methodology) were coded in reference to the following criteria.

L2 proficiency

Following Jeon and Yamashita's (Reference Jeon and Yamashita2014) suggestion and recent meta-analyses on L2 vocabulary learning (Webb et al., Reference Webb, Yanagisawa and Uchihara2020), this study adopted criteria defining L2 proficiency dichotomously (Basic or Beyond basic) in order to avoid inconsistency of proficiency judgements owing to the considerable variation of researchers’ reporting methods of L2 proficiency such as vocabulary test scores, standardized proficiency test scores (e.g., TOEFL), and teachers’ intuitive judgements.

Institutional level

This variable was coded as primary school, secondary school, or university. Other types of institutions such as language school were not found in the current data set.

Text type

Following research suggesting differences in characteristics between narrative and academic texts (Gardner, Reference Gardner2004), we categorized the type of texts used for learning materials in two ways: narrative or expository texts.

Text audience

This variable was coded in terms of whether the text was created for L1 users or L2 learners. Texts made for L2 learners include graded readers, texts written by authors, and modified texts. Texts without any modification or with a minor modification by embedding target words or replacing real words with pseudowords were considered to be written for L1 speakers.

Spacing

The spacing effect occurs when the same target words are encountered multiple times with a certain gap between the encounters. Following Uchihara et al.'s (Reference Uchihara, Webb and Yanagisawa2019) definition of spaced and massed learning conditions, studies in which participants could encounter target words repeatedly over several treatment sessions were identified as spaced-treatment conditions. The current definition of spacing applies to the following scenarios: (a) participants are asked to complete certain tasks (e.g., reading a book) on their own time over an extended period of time (e.g., Daskalovska, Reference Daskalovska2016) and (b) participants are exposed to multiple texts in a series of classroom sessions (e.g., Webb & Chang, Reference Webb and Chang2015a, Reference Webb and Chang2015b). In contrast, studies in which exposure to L2 input was limited to a single-day/one-time treatment were coded as massed-treatment conditions.

Mode of input

This variable consists of four categories: reading, listening, reading while listening, and viewing. For viewing studies, conditions with L2 captions or L1 subtitles were not included in this category.

Pre-knowledge control

It is important to control for prior knowledge of target words in incidental vocabulary learning research to ensure that any gains can be attributed to the learning conditions. To examine this variable, we coded pre-knowledge control in three ways: using pseudowords, pilot testing with other learners of similar L1 and educational backgrounds, or administering pretests, and compared them with studies not administering pretests nor using pseudowords.

Test format

We initially coded test format in six ways: form recognition (e.g., lexical decision task, multiple choice task), meaning recognition (e.g., word matching task), form recall (e.g., L1 to L2 translation), meaning recall (e.g., L2 to L1 translation task), developmental scale (i.e., a vocabulary knowledge scale or VKS), and productive knowledge of spelling (i.e., dictation task). After coding, form recall (k = 0), developmental scales (k = 2) and productive knowledge of spelling (k = 2) were removed owing to an insufficient number of studies using these measures.

Effect size calculation

The first and follow-up posttest effect sizes were calculated according to Hedges’ g (Hedges & Olkin, Reference Hedges and Olkin1985), which offers slightly more conservative calculations compared with Cohen's d, especially for small samples (Borenstein et al., Reference Borenstein, Hedges, Higgins and Rothstein2009). First, the standardized mean difference or d was calculated using the means for treatment and control groups and pooled standard deviations (SDs) according to the following formulae:

Following Morris's (Reference Morris2008) suggestion, we calculated pooled SDs using SDs based on pretest scores when available. Lastly, we used a bias correction factor or J, which was calculated according to the formulae: 1–[3/{4 × df–1}] (Hedges & Olkin, Reference Hedges and Olkin1985) in order to obtain an unbiased effect size, Hedges’ g (=d × J). In order to interpret effect size values for this study, we referred to Plonsky and Oswald's (Reference Plonsky and Oswald2014) benchmarks defining the magnitude of effect sizes: small = 0.40, medium = 0.70, and large = 1.00.

Data analysis

The current study used the Comprehensive Meta-Analysis (version 3.3) (Borenstein et al., Reference Borenstein, Hedges, Higgins and Rothstein2006) software to calculate the mean effect size and conduct moderator analysis. Prior to the effect size aggregation and moderator analysis, we conducted preliminary inspection of the data, revealing that one study (Vidal, Reference Vidal2011) was an outlier in first and second posttests (i.e., more than three SDs above the mean effect sizes); therefore, the study was excluded before proceeding to the subsequent statistical analysis. In addition, we conducted two measures to assess the extent to which publication bias influences the current data sets: fail-safe N and the trim-and-fill method based on the examination of funnel plots (Borenstein et al., Reference Borenstein, Hedges, Higgins and Rothstein2009). These two measures indicated that there was little concern regarding the effect of publication bias on the current meta-analysis findings (see Supplementary Appendix 2 for detailed information regarding the results of the two measures).

A random-effects model was employed to compute the inverse-variance weighted mean effect for the first posttests as well as follow-up posttests, along with a mixed-effects model for subsequent moderator analysis. In effect size aggregation, the homogeneity test was conducted using a within-group Q statistic for the purpose of examining whether there would be a significant variation in true effect sizes across studies. For moderator analysis, between-group Q value was calculated for a total of eight categorical variables. In response to the second research question, we followed previous meta-analysis on L1 incidental vocabulary learning (Swanborn & De Glopper, Reference Swanborn and De Glopper1999) to determine rates of learning by calculating the proportion of learned target items in relation to the total number of target words. To this end, we first revisited the full version of the completed coding sheet and sorted all effect sizes according to test task format (k = 38: form recognition, meaning recognition, and meaning recall) and mode of input (k = 34: reading, listening, reading while listening, and viewing). Next, the mean differences (i.e., learning gain scores in experimental groups – learning gain scores in control groups) were calculated per test format and used to provide an indication of learning gains minus practice effect. Finally, the resulting gain scores were divided by the total number (or maximum scores) of target words to produce rates of learning, which were then averaged for each of the three test formats and four modes of input.