James P. Scanlan, Attorney at Law

Institutional Correspondence

(Apr. 9, 2010; rev. June 1, 2010)

Many thousands of institutions in the United States and around the world engage in activities that involve the appraisal of differences between the rates at which two groups experience an outcome and the bearing of such appraisal on a range of issues in the law and the social and medical sciences. Such institutions include various governmental entities, universities and research institutes, and a variety of scientific and other scholarly journals. With very minor exception, however, the manner in which these institutions appraise differences between outcome rates is fundamentally flawed as a result of the failure to recognize the way that standard measures of differences between outcome rates are affected by the overall prevalence of an outcome, as discussed in the Measuring Health Disparities (MHD), Scanlan’s Rule (SR), and Mortality and Survival pages, among other pages, on this site and in the references made available by those pages, including Can We Actually Measure Health Disparities? (Chance, Spring 2006), Race and Mortality (Society, Jan/Feb 2000), Divining Difference (Chance, Fall 1994), The Perils of Provocative Statistics (The Public Interest, Winter 1991), The Misinterpretation of Health Inequalities in the United Kingdom (British Society for Population Studies, 2006).

The most notable of the ways standard measures of differences between outcome rates are affected by the overall prevalence of an outcome is the pattern whereby the rarer an outcome the greater tends to be the relative difference in experiencing it and the smaller tends to be the relative difference in avoiding it.  Thus, among many comparable examples:

·       When test scores are lowered (or test performance improves), relative differences in failure rates tend to increase while relative differences in pass rates tend to decrease.  

·       When poverty declines relative differences in poverty rates tend to increase while relative differences in rates of avoiding poverty tend to decrease.  

·       When mortality declines, relative differences in mortality rates tend to increase while relative differences in survival rates tend to decrease.

·       When overall rates of receiving beneficial health procedures or care (e.g., mammography, immunization, prenatal care, adequate hemodialysis, etc.) increase, relative differences in rates of receiving such procedures or care tend to decrease while relative rates in failing to receive them tend to increase.

·       Banks with relatively liberal lending policies tend to show large relative differences in mortgage rejection rates but small relative differences in mortgage acceptance rates.

·       Relative differences in adverse outcome rates tend to be large among comparatively advantaged subpopulations (e.g., the college-educated, British civil servants), where such outcomes tend to be rare, while relative differences in the opposite outcomes tend to be small among those subpopulation.

Absolute differences between rates and differences measured by odds ratios tend also to be affected by the overall prevalence of an outcome, though in a more complicated way, as described most precisely in the introduction to SR.  Roughly, as uncommon outcomes (those with rates of less than 50% for both groups) become more common, absolute differences between rates tend to increase; as common outcomes (those with rates of more than 50% for both groups) become even more common absolute differences tend to decrease.  Differences measured by odds ratios tend to change in the opposite direction of absolute differences.[i]   Other common measures that are functions of dichotomies, and hence in some manner affected by overall prevalence,  are discussed in various places – e.g., longevity (BSPS 2006), the Gini coefficient (Comment on Boström), the concentration index (Concentration Index sub-page of MHD), the phi coefficient (Section A.13 of SR), Cohen’s Kappa Coefficient (Section A.13a of SR).[ii]  

One point of clarification is in order. When a study finds, for example, that a factor increases some outcome rate from 1% to 3%, whether one states that the factor increased the outcome by 200% or by 2 percentage points or states that the opposite outcome decreased by 2% (i.e., 99% reduced to 97%) or 2 percentage points, all such characterizations would be correct, and none would implicate the issues described in the prior paragraphs.[iii]  But if one were to attempt to compare the size of the effect in the circumstance just described with one where, say, a factor increases a rate of 2% to 5% – or if one were more abstractly to attempt to characterize the difference between 1% and 3% as a large one or a small one – the referenced issues are implicated. And almost none of the institutions referenced in the first paragraph recognize these issues much less know how to address them.

Most of the material made available on this site involve the analysis of health and healthcare disparities, particularly with regard to whether race/ethnic or socioeconomic disparities are increasing or decreasing or otherwise are larger in one setting than another. But as suggested by the examples set out above, the same issues are involved in any interpretation of the size of differences between outcome rates. 

As reflected in the discussion of the works of Carr-Hill and Chalmers-Dixon, Houweling et al., Eikemo et al., and Day et al. in Section E.7 of MHD, more thought has been given to these issues in Europe than in the United States. But with respect to the extent to which the overwhelming majority of work implicating these issues is fundamentally flawed, the situation in Europe is indistinguishable from that in the United States and elsewhere around the world.

From time to time, I have contacted various researchers or institutions about these issues, in recent years usually by email, suggesting that they reevaluate the ways they or those in some manner affiliated with them (as in the case of editors of scientific journal and the authors who publish in those journals) appraise differences between outcome rates.  But even when the emails have been read carefully enough for that the recipient to recognize that there may be a serious problem with current methods, such communications have had limited effect. With the hope that formal letters will have greater effect, I have decided to send such letters to the some of the more influential institutions involved in activities implicating the issues described above. When sending hard copy letters, it is my practice to include links to referenced materials and to post electronic versions of the letters on this site in order to facilitate the recipients’ review of referenced materials. Thus, as letters are sent, links to the letters will be made available below. 

Some of the eventual recipients of the letters are already discussed in various pages on this site, including the National Center for Health Statistics and the Agency for Healthcare Research and Quality (including among many other places Section E.4 of MHD, Section A.6 of the SR, and the 2007 APHA presentation).  Others may be mentioned only in passing, as in the case of Health Policy Group of Harvard Medical School (see Pay for Performance sub-page of MHD), though the work of such entities may be frequently addressed in the comments collected under Section D of MHD. Those comments, by their critique of so much research in major medical and health policy journals published in the United States and Europe, also inferentially implicate the editorial practices of those journals. The Mortality and Survival page does that more directly with regard to journals that publish articles on disparities in cancer mortality and survival, typically without recognizing, for example, that increasing mortality disparities tend to be associated with decreasing survival disparities.  

Whether a particular institution receives attention on this site or receives one of the letters to be listed below typically will have little to do with the institution’s level of understanding of these issues.  For similar misunderstandings exist at essentially all institutions.  Institutions (and researchers) that indicate a recognition that determinations as to the size of a disparity between outcome rates may turn on the measure chosen may seem to reflect a greater understanding of the matter.  But unless such entities also recognize the way each measure is systematically affected by the overall prevalence of an outcome or the need to find a measure that is not so affected, their recognition that different measures may yield different results is of limited value.

Letters to institutions are listed below with link provided to the letter.

National Quality Forum (Oct. 22, 2009)

Robert Wood Johnson Foundation (Apr. 8, 2009)

The Commonwealth Fund (June 1, 2010)

Institute of Medicine (June 1, 2010)