Statistical Reasoning
(July 20, 2008; rev. Apr. 15, 2012)
Prefatory note added October 11, 2018. This page has not been updated in quite a few years. Subsequent, often extended, treatments of the issues addressed in Section A of the page may be found in my Comments for Commission on Evidence-Based Policymaking (Nov. 14, 2016), “The Mismeasure of Health Disparities,” Journal of Public Health Management and Practice (July/Aug. 2016), “Race and Mortality Revisited,” Society (July/Aug. 2014), The Perverse Enforcement of Fair Lending Laws (Mortgage Banking, May 2014), “Measuring Health and Healthcare Disparities,” Proceedings of the Federal Committee on Statistical Methodology 2013 Research Conference (Mar. 2014), and “The Mismeasure of Discrimination,” Faculty Workshop, University of Kansas School of Law (Sept. 20, 2013), and methods workshops given at the following educational institutions: University of Massachusetts Medical School (2015), University of California, Irvine (2015), George Mason University (2014), University of Maryland (2014), University of Minnesota (2014), , Harvard University (2012), American University (2012). Recent treatments of the issues with a focus on the mistaken understanding of the effects of reducing adverse criminal justice, school discipline, and loan outcomes – i.e., the belief that doing so will tend to reduce, rather than increase, relative racial differences in rates of experiencing the outcomes – may be found in “Discipline disparities in Md. Schools,” Daily Record (June 21, 2018), “What the government gets wrong about fair lending,” American Banker (Apr. 9, 2018), “The misunderstood effects of the Baltimore police consent decree,” The Daily Record (Feb. 15, 2018). “United States Exports Its Most Profound Ignorance About Racial Disparities to the United Kingdom,” Federalist Society Blog (Nov. 2, 2017). A recent, fairly comprehensive of the misunderstanding of the effects of relaxing discipline standards on measures of racial disparity may be found in my Letter to Maryland State Department of Education (June 26, 2018) and its attachments, the first three of which are my December 8, 2017 testimony to the U.S. Commission on Civil Rights, my July 17, 2017 letter to the U.S. Departments of Education, Health and Human Services, and Justice, and the handout I used to explain the issue to U.S. Department of Education staff at a March 22, 2018 meeting.
Recent treatments of the issues addressed in Section B, with a focus on problems in analyses of claims of discrimination in pay and loan terms received, may be found in “EEOC, OMB, and the Collection of Data That Can’t Be Analyzed,” Federalist Society Blog, “Partial Picture Issue Undermines Chadbourne Pay Equity Case,” Law360 (Jan. 25, 2017).
***
This page contains links to three groups of items on statistical issues, all of which are also discussed under other headings. Section A (“Correlations Between the Size of Differences Between Rates and the Overall Prevalence of an Outcome”) contains links to articles discussing the patterns whereby measures of differences between rates of experiencing an outcome tend to be correlated with the overall prevalence of an outcome. Without understanding these tendencies it is not possible to determine whether the size of one difference between outcome rates is larger than another in any meaningful sense – including, for example, whether a three-to-one disparity is larger than a two-to-one disparity. As a result of the near universal failure to understand these tendencies, however, very little research or commentary comparing the size of differences between rates of experiencing an outcome, whether in the law of the social and medical sciences, has been of value. These articles are also discussed on the Measuring Health Disparities page (MHD) and its sub-pages. That page also contains links to 22 Conference Presentations conference presentation and over 120 on-line letters to medical or health policy journals further explaining the tendencies generally or with regard to particular sets of data, as well as a number of sub-pages addressing particular issues. A number of the articles on MHD, however, pertain to the role of the referenced tendency with regard to issues other than health disparities. The Scanlan’s Rule page and its various sub-pages, as well as the Mortality and Survival, Measures of Association, Educational Disparities, Lending Disparities, and Discipline Disparities page, also discuss the pattern with regard to an assortment of issues involving matters other than health disparities.
Section B (“Job Segregation”) lists articles explaining the flaws in efforts to prove “job segregation” or “initial assignment discrimination” based solely on the way employees in different groups are distributed within a company and without regard to the composition of applicants seeking or willing to accept various positions. These articles are also found under Section A of the “Employment Discrimination” page.
Section C (“Other Statistical Issues”) contains links to items relating to other statistical issues. It lists only one published article (item C.1). Note 23 of that article briefly discusses the relevance of a number of the articles under Section A. I note, however, that all the reasoning in C.1 regarding the size of disparities should be evaluated from the perspective provided in Section E.6 of the Measuring Health Disparities page. The other links in Section C.1 are to on-line letters that are also collected on the Health Disparities Measurement page. They are collected there in part because they relate to the main issue on that page, which is also the subject of Section A of this page. But they are listed here because they involve other statistical issues as well. A brief parenthetical note describes the issue.
Section D (Miscellaneous) contains a link to an historical document. The document was filed in court as part of the government plaintiffs’ final report at the conclusion of the AT&T consent decree, which from 1973 to 1979 had imposed certain affirmative action obligations on AT&T and the 23 Bell System Operating Companies. The document, which explains various flaws in the goal-setting mechanism that went undiscovered during the term of the decree, illustrates a number of statistical issues.
The articles in Sections B and C are listed in a different style from those in Section A simply because the articles in Section A, as listed in the Measuring Health Disparities page, use the medical journal format. The articles are numbered chronologically but listed from most to least recent (for reason explained in the Measuring Health Disparities page).
Subsequent to the initial creation of this page, a Vignettes page was added. It contains a number of sub-pages that are generally related to statistical issues. The Times Higher Issues sub-page addresses the widespread custom of describing, for example, 3 as being three times greater than 1, rather than three times as great as 1. It provides tables showing the overwhelming predominance of the misusage even in scientific journals, with the notable exception of the New England Journal of Medicine. The sub-page also discusses several related points including the fact that most dictionary definitions of the word “multiplication” are incorrect. The Gender Differences in DADT Terminations page discusses certain misperceptions in reportage that women are disproportionately discharged for violation of the military don’t-ask-don’t-tell policy, including the bases for the perception that women and disproportionately discharged and the bases for comparisons among the military branches (which involves the issue discussed on the Representational Disparities sub-page of the Scanlan’s Rule page). The Adjustment Issues sub-page addresses several points concerning approaches to adjustment of group differences in outcome rates for group differences in particular outcome-related characteristics, including the confusion between the standard adjustment for different prevalences of a characteristic within different groups and determining what the differences between rates would be if the characteristic did not exist. This is also the subject of items C.2, C.4., and C.5 below. The Percentage Points sub-page addresses the way researchers refer to percentage point differences as if they were percent differences. The Odds Ratios and Statistical Significance Vig sub-pages are as yet only sketches and do not warrant description here (save to note that item C.6 below relates to the latter).
A. Correlations Between the Size of Differences Between Rates and the Overall Prevalence of an Outcome
13. The lending industry’s conundrum. National Law Journal Apr. 2, 2012: http://www.law.com/jsp/nlj/PubArticleNLJ.jsp?id=1202547386988&The_lending_industrys_conundrum&slreturn=1
12. Can we actually measure health disparities? Chance 2006:19(2):47-51:
http://www.jpscanlan.com/images/Can_We_Actually_Measure_Health_Disparities.pdf
11a. “Statistical Proof of Discrimination,” in Affirmative Action, An Encyclopedia (James A. Beckman ed.) Greenwood Press, 2004, 838-40:
http://jpscanlan.com/images/Statistical_Proof_of_Discrimination.pdf
11. Understanding racial differences in infant morality. PrenatalEd Update October 2000:
http://www.jpscanlan.com/images/Understanding_Racial_Differences_in_Infant_Mortality.pdf
10. Race and mortality. Society 2000;37(2):19-35 (reprinted in Current 2000 (Feb)): http://www.jpscanlan.com/images/Race_and_Mortality.pdf
9. Both sides misuse data in credit discrimination debate. American Banker July 22, 1998.
8. Mired in numbers. Legal Times Oct. 12, 1996: http://jpscanlan.com/images/Mired_in_Numbers.pdf
(racial impact of mandatory life sentences)
7. When statistics lie. Legal Times Jan. 1, 1996: http://jpscanlan.com/images/When_Statistics_Lie.pdf
(racial disparities in mortgage rejection rates)
6. Getting it straight when statistics can lie. Legal Times Jun 28, 1993:
http://jpscanlan.com/images/Getting_it_Straight_When_Statistics_Can_Lie.pdf
(age disparities in terminations for failure to meet a performance standard)
5. Divining difference. Chance 1994;7(4):38-9,48: http://jpscanlan.com/images/Divining_Difference.pdf
4. Comment on “McLanahan, Sorensen, and Watson's 'Sex Differences in Poverty, 1950‑1980.’'" Signs 1991;16(2):409-13: http://www.jpscanlan.com/images/Signs_Comment.pdf
3. The perils of provocative statistics. The Public Interest 1991;102:3 14:
http://jpscanlan.com/images/The_Perils_of_Provocative_Stat.pdf
2. An issue of numbers. The National Law Journal Mar. 5, 1990:
http://www.jpscanlan.com/images/An_Issue_of_Numbers.pdf
(racial impact of academic eligibility requires, employment discrimination issues)
1. The “feminization of poverty” is misunderstood. The Plain Dealer Nov 11, 1987 (reprinted in Current 1988;302(May):16-18 and Annual Editions: Social Problems 1988/89. Dushkin1988:
http://www.jpscanlan.com/images/Poverty_and_Women.pdf
(feminization of poverty, racial differences in infant mortality rates)
B. Job Segregation
6. "Women Employees' Case against Publix, Built on Wrong Data, Doesn't Compute," Miami Daily Business Review (Aug. 2, 1996)
5. "Multimillion-Dollar Settlements May Cause Employers to Avoid Hiring Women and Minorities for Less Desirable Jobs to Improve the Statistical Picture," The National Law Journal (Mar. 27, 1995)
http://jpscanlan.com/images/multimillion_cor_42405.pdf
4. "Unlucky Stores: Are They All Guilty of Discrimination?” San Francisco Daily Journal (Jan. 29, 1993)
http://jpscanlan.com/images/Unlucky_Stores.pdf
3. "Indiscriminate Reading of Statistics Can 'Prove' Bias Where None Exists," Manhattan Lawyer (Apr. 24, 1989)
2. "Are Bias Statistics Nonsense?" Legal Times (Apr. 17, 1989)
Are Bias Statistics Nonsense? (PDF Format)
1. "Illusions of Job Segregation," The Public Interest (Fall, 1988)
http://www.jpscanlan.com/images/Illusions_cor_42404.pdf
C. Other Statistical Works
6. Study raises issues concerning nonsignificant findings and implications of large differences in health services utilization between the healthy and the unhealthy. BMJ
(responding to Saxena S, Eliahoo , Majeed A. Socioeconomic and ethnic differences in self reported health status and use of health services by children and young people in England: cross sectional study. BMJ 2002;325:520-523): http://www.bmj.com/content/325/7363/520.1/reply#bmj_el_244510
5. Effects of standard adjustment approaches on relative and absolute inequalities. J Epidemiol and Community Health Nov. 2, 2009 (responding to Lynch J, Davey Smith G, Harper S, Bainbridge K. Explaining the social gradient in coronary heart disease: comparing relative and absolute risk approaches. J Epidemiol Community Health 2006:60:436-441): http://jech.bmj.com/cgi/eletters/60/5/436
4. Study shows different adjustment approaches rather than different relative and absolute perspectives. Journal Review May 1, 2008 (responding to Khang YH, Lynch JW, Jung-Choi K, Cho HJ. Explaining age-specific inequalities in mortality from all causes, cardiovascular disease and ischaemic heart disease among South Korean public servants: relative and absolute perspectives. Heart 2008;94:75-82):: http://journalreview.org/v2/articles/view/17591645.html
3. A study with a variety of problems. Journal Review June 2, 2007 (responding to Schulman KA, Berlin JA, Harless, et al. The effect of race and sex on physicians’ recommendations for cardiac catheterization. N Engl J Med 1999;340:618-26): http://www.journalreview.org/view_pubmed_article.php?pmid=10029647&specialty_id=
2. Understanding social gradients in adverse health outcomes within high and low risk populations. J Epidemiol Community Health May 18, 2006, responding to Lynch J, Davey Smith G, Harper S, Bainbridge K. Explaining the social gradient in coronary heart disease: comparing relative and absolute risk approaches. J Epidemiol Community Health 2006:60:436-441: http://jech.bmjjournals.com/cgi/eletters/60/5/436
1. "Measuring Hiring Discrimination," The Labor Law Journal (July, 1993)
http://jpscanlan.com/images/Measuring%20Hiring%20Discrimination.pdf
D. Miscellaneous
1. Addendum to AT&T Final Report
http://www.jpscanlan.com/images/Addendum_to_Final_Report.pdf