When meta-analysis misleads: the need for methodological integrity in e-cigarette research
- Alan Gor
- Jun 17
- 3 min read
17 June 2025
Introduction
Tobacco harm reduction (THR), especially in the context of electronic cigarette (EC) research, remains highly contested within public health, often due to studies compromised by methodological flaws [1,2,3]. These shortcomings in study design, analysis, terminology, or interpretation can distort scientific record, erode public trust, and generate misleading or uncritical media coverage and policy reports [4,5,6]. These challenges become especially consequential when flawed evidence is aggregated and amplified through meta-analyses, which play a central role in shaping policy and clinical guidance.
In this issue of Internal and Emergency Medicine, Rodu and colleagues [7] critically assess a widely cited 2024 meta-analysis by Glantz et al. [8], published in NEJM Evidence. Glantz and coauthors concluded that e-cigarette use is associated with disease odds similar to those of cigarette smoking for cardiovascular conditions, and still substantial, though lower, for asthma, COPD, and oral diseases. Rodu et al. identify major methodological flaws that call these conclusions into question. This underlines the urgent need for rigorous and transparent evidence synthesis in tobacco harm reduction science.
Dangers of garbage in, garbage out
The credibility of any meta-analysis is directly dependent on the quality, comparability, and methodological rigor of the studies it includes [9]. When foundational studies are flawed or inconsistent, the meta-analysis built upon them is likely to be compromised, regardless of statistical sophistication [9]. This principle, often summarized by the phrase “garbage in, garbage out”, is particularly relevant in the case examined by Rodu and colleagues. Their critique of the meta-analysis by Glantz et al. exposes a series of methodological issues that raise serious concerns about the reliability of the conclusions.
One of the most problematic aspects highlighted by Rodu et al. is the indiscriminate aggregation of disease outcomes into excessively broad diagnostic categories. For example, Glantz et al. [8] grouped vastly different conditions, such as erectile dysfunction and myocardial infarction, under the umbrella of “cardiovascular disease.” Similarly, Glantz et al. grouped influenza (classified as “respiratory symptoms”) and chronic obstructive pulmonary disease (COPD) within a single category for pooled analysis, despite having distinct clinical profiles. As a result, the pooled risk estimates become difficult to interpret and may give a distorted view of the potential health risks associated with e-cigarette use. This lack of clinical coherence introduces systematic bias and reduces the validity of any generalized conclusions drawn from the analysis.
Adding to this concern is the heavy reliance on cross-sectional studies, which accounted for the majority, 76 percent (94 out of 124), of the odds ratios included in the Glantz et al. meta-analysis. Cross-sectional designs assess both exposure and outcome at a single point in time [10], which means they are inherently incapable of establishing whether vaping preceded the onset of disease. Without a clear temporal sequence, the evidence cannot support causal inferences.
Many of the studies included in the Glantz et al. meta-analysis relied on data from large-scale surveys, such as the National Health Interview Survey (NHIS) and the Behavioral Risk Factor Surveillance System (BRFSS). While these datasets are valuable for descriptive epidemiology, they typically lack crucial temporal information, such as the age of smoking or vaping initiation and the timing of disease diagnosis. Without this temporal resolution, it becomes impossible to establish whether the exposure could plausibly have contributed to the health outcome. We have previously highlighted the gravity of this issue [11], stressing that the persistent repetition of such methodological shortcomings has now reached a scale that risks undermining the credibility of public health science itself. A further concern relates to the possibility of double-counting results derived from the same data sources (e.g., NHIS, BRFSS, or PATH), particularly for outcomes like COPD, which were reported in multiple studies using overlapping samples. Although Glantz et al. report inflating standard errors to account for potential correlation between estimates, this statistical correction does not eliminate the risk that individual disease cases were counted more than once across pooled estimates, which could artificially enhance the perceived consistency or precision of the findings.
A related limitation is the near-total absence of cumulative exposure metrics across the studies included in Glantz et al.’s meta-analysis. As highlighted in recent methodological guidance [3], observational studies should report dose–response exposure histories to avoid misclassification and residual confounding. When meta-analyses include studies lacking this level of granularity, they risk pooling data from participants with vastly different exposure intensities. This can obscure true dose–response relationships and conflate light, short-term use with heavy, chronic use. In the context of comparing exclusive and dual use, such exposure misclassification may seriously distort conclusions about relative harm.
