James P. Scanlan, Attorney at Law

Educational Disparities/Achievement Disparities

(Apr., 2011, rev. Apr. 17, 2012)

The Measuring Health Disparities, Scanlan’s Rule, Mortality and Survival, Measures of Association, Lending Disparities, and Discipline Disparities pages of this site (and their subpages) address the failure in the law and the social and medical sciences to recognize the way that, for reasons related to the shapes of normal distributions of factors associate with experiencing an outcome, standard measures of differences between outcome rates tend to be affected by the overall prevalence of an outcome.  Most notably, as an outcome increases in overall prevalence, relative differences in experiencing it tend to decrease while relative differences in failing to experience it tend to increase.  Thus, for example, if a test cutoff is lowered (or overall test performance improves) relative differences in pass rates tend to decrease while relative differences in failure rates tend to increase. Absolute differences and odds ratios tend also to be affected by the overall prevalence of an outcome, though in a more complicated way.  Roughly, as uncommon outcomes (less than 50% for both groups) become more common, absolute differences tend to increase; as common outcomes (more than 50% for both groups) become even more common, absolute differences tend to decrease.  The matter is a bit more complicated where the rate is above 50% for one group and below 50% for the other, as explained in the introductory section to the Scanlan’s Rule page.  Differences measured by odds ratios tend to change in the opposite direction of absolute differences.

As suggested by the reference to test outcomes these patterns are also present in data on demographic differences in education achievement.  Indeed, the force of the patterns may be the strongest in the educational setting.  But, as in other contexts, the role of the patterns is almost universally ignored in analyses of educational disparities, even when the disparities involve such things as math proficiency or statistical literacy.  This is a draft of a page addressing some of this misunderstanding.

Some points have been touched upon in prior articles or presentations.  Race and Mortality (Society, Jan/Feb 2000) briefly discussed how supporters of affirmative action in college admissions have pointed to the small relative differences between minority and white graduations rates at elite universities, while opponents have pointed to large relative differences in rates of failing to graduate at those universities.  Neither the supporters nor the opponents recognized that the patterns they cited were to be expected simply because elite universities have high graduation rates.  Slide 13 of  the presentation Measurement Problems in the National Healthcare Disparities  (American Public Health Association 2007) used data from a chart styled “Remarkable Results” in a November 4, 2007 Washington Post article[i], which discussed perceived substantial decreases in achievement gaps at a school in Maryland, to illustrate that the National Center for Health Statistics, relying on relative differences in adverse outcomes, would find the gaps to have increased. 

I present below data from that and another Washington Post article to illustrate the issues, along with information on changes in relative differences, absolute differences, and the only actually useful measure of difference between outcome rates – that described on the Solutions sub-page of the Measuring Health Disparities page (and which I usually term “EES” and which, as noted there, is also known as the probit).  The Solutions/Probit method derives from a pair of rates the difference, in terms of the percentage of a standard deviation, between hypothesized underlying distributions.  It should be recognized, however, that in the educational context, the actual underlying data generally is available to enable one to directly determine whether differences between means have changed.

I illustrate these patterns with data from the Remarkable Results table and data from an April 5, 2011 Washington Post article[ii] that, in finding larger improvements in rates of scoring at advanced levels in math among Asians than whites, relied on percentage point changes as an indicator of the size of increases rates of scoring at those levels. 

There two ways one can examine group changes time.  One can compare the sizes of the disparity at each point in time (which is how the matter is characterized two paragraphs above).  Or one can compare the size of the changes experienced by each group.  Because the Remarkable Results Table presented data for a number of groups, the latter approach is less complicated and is used here with regard to those data. 

A useful feature of the Solutions/Probit approach is that difference between the disparity at two points in time will be the same as the difference between the changes experienced by the groups being compared between the two points in time (as explained in the Subgroups Effects sub-page of the Scanlan’s Rule page).[iii]  To illustrate that point parts of the data from the articles will be analyzed two ways.

Table 1 presents data from the Remarkable Results Table in the 2007 Post article, showing (a) the percentage increase in proficiency rates, (b) the percentage decrease in non-proficiency rates; (3) the percentage point changes in proficiency rates; (d) the estimated effect size derived from the earlier rate and later rate.[iv] 

Table 2 limits the showing to the African American and white changes in math.  One observes a larger percentage increase in proficiency rates for African Americans, a larger percentage decrease in non-proficiency for whites, and a larger percentage point increase for African Americans.  Each change is in the direction that would be expected when the only forces operating are the above-described distributional patterns – i.e., when there occurs no meaningful change.  The EES figure, however, reveals that the change was greater for whites than African Americans (1.0 standard deviations for white compared with .8 standard deviations for African Americans).  Thus the disparity increased. 

Creation of the Discipline Disparities page in April 2012, which page makes reference to the figures in Table 2 caused me to recognize a need also to present the information in Table 2 also in terms of the differences between African Americans and whites at two points in time (something I had previously done only for the  Asian/White data in the April 5, 2011 Washington Post article, see Tables 3 and 4 below).  Thus, Table 2a shows the differences between African Americans and whites at two points in time.  From this perspective, too, each change in the direction that would be expected when the only forces operating are the above-described distributional patterns, though, properly measured, the disparity increased.

(A shortcoming of Tables 2a as an example is that the reader may be inclined to think there is something significant in the fact that, while the pairs of EES figures and the pairs of absolute difference figures mean different things in the two tables, those pairs are identical in the two tables.  But that those pairs of figures are identical in the two tables is simply a coincidence.[v]  As noted elsewhere on this page, that the arithmetic difference between the figures in each pair of figures is the same – i.e., 12 percentage points in the case of the pairs of absolute differences  and .20 standard deviations in the case of the pairs of EESs – is not a coincidence.

Table 3 is based on data presented in the April 5, 2011 Washington Post article. Like Tables 1 and 2 is shows ) the percentage increase in scoring at the advanced level, (b) the percentage decrease in failure to score at the advanced level; and (3) the percentage point changes in scoring at the advanced level; (d) the estimated effect size.

In Maryland the Asian improvement was larger according to measures (a), (b), and (c).  When that occurs, the group with the larger increase by those measures will invariably have the larger increase according to (d).  In Virginia, however, whites had a larger percentage increase in scoring at the advanced level.  Thus, one can determine which group had the larger change only by reference to the EES.  While Asians experienced a larger change in both states (according to EES), the difference was larger in Maryland than Virginia (.13 standard deviations compared with .9 standard deviations), which is consistent with the fact that in Maryland the actual difference in changes was large enough to cause all the Asian advantage to appear in all measures. 

Table 4 analyzes the data in terms of the disparities at two points it time. Thus, it presents (a) the ratio of the Asian to white favorable outcome rate; (b) the ratio of the white to Asian adverse outcome rate; (c) the absolute differences between rates; (d) the disparity as measured by the EES. 

 As discussed above, the difference between the EES scores reflecting the group changes over time will equal the difference between the EES scores reflecting the disparity at each point in time.  It may be noted that in Virginia the Asians change reflected by the EES was .09 larger than that for whites, the difference in disparity changed by only .08.  The difference between the .08 and .09 figures is simply a result of rounding/inexactness of the procedure as implemented by the Solutions database.