Uncertainty in Estimating Vaccine Impact

Under-ascertainment of COVID-19 mortality is a known issue that has hindered our understanding of the pandemic. In this analysis, we consequently focused on fitting to all-cause excess mortality, which provides a more complete description of the pandemic. However, even when relying on model fits based on reported COVID-19 deaths, we estimated that more than 14 million deaths were averted by COVID-19 vaccination. The discrepancy between vaccine impact estimates based on excess mortality and COVID-19 deaths was concentrated in settings with lower death registration and certification. This substantial discrepancy underpins the crucial need for continued investment in civil registration and vital statistics to prevent biases in mortality reporting further minimising the perceived impact and necessity of vaccination in settings with lower reporting of deaths. In countries with more complete reporting systems, our estimates were broadly comparable to other endeavours focused on officially reported COVID-19 deaths and on understanding the direct impact of vaccination on people older than 60 years in Europe. We identified one study that estimated both the indirect and direct impact of vaccination, which again yielded estimates for vaccine impact in the USA that were similar to our impact estimates based on reported COVID-19 deaths.

In our effort to provide impact estimates globally, we introduced various assumptions into our model. We were hindered by the global disparities in SARS-CoV-2 genomic surveillance and the absence of detailed vaccination data for the majority of countries. Consequently, key model inputs had to be created from working assumptions on which vaccines were delivered, how they were delivered, and when new variants of concern spread worldwide. We also assumed that the relationship between age and IFR was the same for each country. These assumptions would have affected our estimates of deaths averted, with sensitivity analyses showing that higher overall IFRs will increase the number of deaths that could be averted by vaccination. Our impact estimates were also limited by the inherent uncertainty in model-based estimates of excess mortality. These estimates are likely to have underestimated or overestimated COVID-19 death tolls in many countries. Notably, our model fits were unable to recreate excess mortality death tolls in recent epidemic waves in Iraq and Sudan because of the depletion of the susceptible population. These discrepancies could have been due to multiple reasons, including overestimated excess mortality, proportions of excess mortality not due to COVID-19, higher infection fatality rates by age in low-income settings than those estimated from high-income countries, and lower vaccine effectiveness than assumed in our framework. Last, our impact estimates were dependent on the assumed degree of immune escape that each variant of concern exhibits. If immune escape was higher than we assumed, more of the population would have been susceptible to re-infection and consequently more deaths from COVID-19 could have been averted by vaccination.

More broadly, our estimates should be considered in light of the considerable uncertainty inherent in estimating vaccine impact. Uncertainty in the true death toll of the pandemic, the circulating variants of concern and their immunological phenotypes, and the vaccines themselves administered in many countries vastly complicate efforts to derive accurate estimates of the impact of COVID-19 vaccines. However, the results of this analysis still provide a comprehensive and thorough assessment of the impact of COVID-19 vaccination, revealing the substantial impact that vaccines have had and the millions of lives that are likely to have been saved during the first year of vaccination. Despite this, more lives could have been saved if vaccines had been distributed more rapidly to many parts of the world and if vaccine uptake could have been strengthened worldwide. Reaching vaccination coverage targets and improving vaccine coverage globally is dependent on multiple factors and not solely dependent on improving vaccine donations. Vaccine intellectual property needs to be shared more quickly in the future, with more open technology and knowledge transfer surrounding vaccine production and allocation. Vaccine distribution and delivery infrastructure also needs to be scaled up worldwide and misinformation combatted to improve vaccine demand. Improvements must be made in all these areas to reach current vaccine targets and help ensure that vaccines are more equitably distributed in the future.