James P. Scanlan, Attorney at Law

New York Proficiency Disparities

(Aug. 19. 2013) 

Prefatory note added May 30, 2016:  Subsequent to the last material updating of this page, the issues it addresses have been treated at length in my “Race and Mortality Revisited,” Society (July/Aug. 2014) and my letter to the American Statistical Association (Oct. 8, 2015).

***

This subpage is one of several subpages to the Educational Disparities page of jpscanlan.com.  Like the other educational disparities pages or subpage (and many pages of this site), this page addresses the problematic nature of standard measures of differences between outcome rates for appraising the differences in the circumstances of two groups reflected by those rates.  The most comprehensive treatments of this issue are treated on the Scanlan’s Rule page and its subpage and the October 9, 2012 Harvard University Measurement Letter.

On August 7, 2013 the organization NYCAN issued a report analyzing results of the 2013 New York State Assessment tests for English Language Arts (ELS) and Math in grades 3 through 8.  The report found substantial reductions in proficiency rates.  Measuring demographic differences in terms of absolute (percentage point) differences between proficiency rates (an increasingly common approach, as reflected on the main Educational Disparities page and the Disparities by Subject and Education Trust GC Study subpages), subpage), the report found all disparities to have decreased, though only slightly in some cases.

As discussed on many places on this site, for reasons related to the shapes of the underlying distributions, when an outcome generally decreases, relative differences in experiencing the outcome will tend to increase while relative differences in failing to experiencing the outcome are tend to decrease.  As best explained in the Introduction to the Scanlan’s Rule page and on pages 21-24 of the Harvard University Measurement letter, absolute differences tend also to change when an there occur overall changes in the prevalence of an outcome, though in a more complicated way than the two relative differences.  As pertinent to the situation of declining rates of proficiency, where both rates start below 50% general declines will tend to reduce absolute differences between rates.  When both rates start above 50%, general declines will tend to increase absolute differences (at least until the point where one group’s rate drops below 50%.  But the distributionally-driven pattern of changes in absolute differences between rates is difficult to predict when one group’s rates starts above 50% and the other group’s rate starts below 50% and when either rate crosses the 50% line. 

Table 1 below consistently shows the typical pattern of relative differences whereby with the general declines in proficiency rates relative differences in proficiency rates increased while relative differences in failing to achieve proficiency increased.[i]  Thus, one relying on relative differences in proficiency rates to measure disparities, as was done in the Harvard Civil Rights Project’s No Child Left Behind Study (see the Harvard CRP NCLB Study subpage) would have found increasing disparities. 

The table also shows the pattern of changes in absolute differences between rates relied on in the study, as well as the estimated effect size (EES) of the differences (a method explained, among other places, on the Educational Disparities page and which involves deriving from a pair of rate the difference between the means of the hypothesized underlying, normal distributions of factors associated with experiencing the outcome).  These patterns were usually consistent with the pattern reflected by absolute differences (though tending to show somewhat smaller decreases in disparities).  But there is one exception.  While the percentage point difference in math proficiency between whites and Hispanics decreased slightly, the EES increased from .56 standard deviations to .60 standard deviations.[ii]

Table 1.  Proficiency Rates for Advantaged and Disadvantaged Demographic Groups (AG & DG) in Grades 3 to 8 for ELS and Math in New York in 2012 and 2013, with Ratio of Rates Proficiency and Non-Proficiency and Absolute Differences between Rates, with Estimated Effect Size. 

Although the absolute difference changed in the same direction as the EES in all but one case, such fact should not be read as suggesting that the absolute difference is a useful measure of educational disparities.  As discussed in several places in the Harvard letter, reliance on absolute difference is causing a host of problems in various contexts.  A particular problem with regard to achievement disparities as reflected by the proficiency rates of advantaged and disadvantaged groups is that, as proficiency generally increases (a more common case than what was observed in New York between 2012 and 2013), absolute differences between rates will tend to increase in harder subjects (particularly when rates remain below 50% for all groups) but decrease in easier subjects (particularly when rates start above 50% for all groups).  But neither change will necessarily reflect that the disparity in the circumstances of the two groups has changed at all or even reflect that the disparity has not changed in the opposite of the change in the absolute difference.  See the Disparities by Subject subpage.