James P. Scanlan, Attorney at Law

Flaws of Meta-Analysis

(Sept. 2, 2010)

Note: This page discusses the problems of standard meta analysis arising from the patterns described on the Measuring Health Disparities (MHD) and Scanlan’s Rule (SR) pages.  The discussion here is somewhat perfunctory mainly because the same points have been made in so many places with regard to other issue, such as on the Subgroup Effects sub-page of SR.    

Meta-Analysis is an increasingly used procedure for appraising the effect of a factor across a range of studies.  The effect size in each study, however measured, is weighted by the inverse of the variance.  Other things being equal that means the results are weighted by sample size.

When the effect on a dichotomous outcome is issue – for example, in the study of the extent to which a factor increases mortality  – common measures of effect size are the relative risk and the odds ratio.   But for reasons discussed generally on the Scanlan’s Rule page, such studies are problematic because of the way relative risks and odds ratios are affected by the overall prevalence of an outcome.

The point can be illustrated with data from Table 1 of the 2006 British Society for Population Studies.  A better illustration would involve base rates that are in ranges that one actually observes in different studies of the same issue.  But for present purposes, the data in Table 1 suffice.

Table A reflects the situations where that a treatment moves the risk distribution of each set of controls half a standard deviation to the right.  Thus, the effect is the same in each of the 15 studies.

But assuming that the risk ratio for mortality is the measure employed in the analysis, one will get different results depending on which studies are included.  Even if all studies are included, the analysis would be fundamentally flawed.  The same of course holds is the measure employed is the risk ratio for survival or the odds ratio. 

As discussed in many places with regard to other issues, the only useful measure is that set out on the Solutions  subpage of the MHD, which is to say the probit.

Users of meta analysis commonly attempt to determine whether there exists heterogeneity among the studies and if so to take that into account in interpreting the results (as discussed here, for example).  But as I explain the Subgroup Effects sub-page of SR, risk ratios tend to differ solely because of the different base rates, creating a systematic tendency for perceived heterogeneity regardless of whether there is any real heterogeneity (which would be identified by the Solutions/probit approach).  See also the Reporting Heterogeneity sub-page of MHD.