James P. Scanlan, Attorney at Law

Annie E. Casey Foundation 2014 Proficiency Disparities Study

(May 12, 2014)

This page is closely related to the Education Trust GC Study subpage of the  Educational Disparities page of jpscanlan.com, which also involves a study that attempted to appraise changes in demographic differences in achieving/failing to achieve certain levels of reading skills without recognizing the way the measure employed tends to be affected by the prevalence of an outcome.  To understand this page, the reader should have a general understanding of the patterns by which measures tend to changes as the prevalence of an outcome changes, as explained on many pages of this site.  Specifically, and as pertinent here, when a favorable outcome increases in overall prevalence the group with the lower baseline rate (i.e., the disadvantaged group) will tend to experience a larger proportionate increase in its rate, while the advantaged group will tend to experience a larger proportionate decrease in the adverse outcome.  Correspondingly, as favorable outcome rates increase relative differences in favorable outcomes will tend to decrease while relative differences in adverse outcomes will tend to increase.  Absolute (percentage point) differences will also change as the prevalence of an outcome changes.  Roughly, as uncommon outcomes (less than 50 percent for both groups being compared) become more common, absolute differences between rates tend to increase; as common outcomes (greater than 50 percent for both groups being compared) become even more common, absolute differences tend to decrease.  Where the rate of either outcome is less than 50 percent for one group and more than 50 percent for the other group – or crosses the 50 percent point for either group – the prevalence-related pattern is difficult to predict.  See the introduction to the Scanlan’s Rule page of jpscanlan.com.  Figure 4 of the 2012 Harvard Applied Statistics Workshop provide some insight to the role of the 50 percent mark (since as one moves toward the point the proportion of the population associated with a given movement across the X-axis increases until the 50 percent point is reached and thereafter decreases). 

 

In 2014, the Annie E. Casey Foundation issued a report titled Early Reading Proficiency in the United States.  The study examined differences by income in achieving proficiency among fourth graders, which it generally discussed in terms of rates of failing to achieve proficiency, in 2003 and 2013.  While the study found that proficiency generally improved for all groups, it found that racial differences in non-proficiency rates increased.  But it is not clear how it measured disparities. 

The report described its findings in these cryptic terms:

Scores for higher- and lower-income groups improved in the past decade, but

proficiency levels increased significantly more for higher-income students (17 percent

improvement) than for their lower-income peers (6 percent improvement). As a result,

the gap in proficiency rates between low-income and higher-income children

widened by nearly 20 percent over the past decade and got worse in nearly every

state.

The phrasing sounds as if the study report was examining proportionate changes in proficiency rates, which typically would be greater for the disadvantaged group than the advantaged group.  But the study’s Table 1, while providing information only on overall changes by states, shows proportionate changes in rates of failing to achieve proficiency.  These typically are larger for the advantaged group than the disadvantaged group.

On the other hand, Figure 2 is a color coded map of states according to ranges of percentage point (absolute) differences between high and low proficiency rates.  The Figure also refers to percentage point differences when describing the largest proficiency gaps by states.

While remaining uncertain as to the meaning of the quoted statement,[i] and the nature of the 20 percent increase it mentions, I believe the study provides a useful illustration of the problems with reliance on absolute differences as a measure of proficiency disparities especially when contrasted with the Education Trust Glass Ceiling study addressed in the Education Trust GC Study subpage. 

For illustrative purposes, I set out in Table 1 the patterns which the two relative differences and the absolute difference changes as the prevalence of an outcome change as reflected by altering the cutoffs on a test where the means scores of the advantaged and disadvantaged groups differ.  The table is a variation on Table 1 of the  British Society for Population Studies 2006 Conference Paper, which underlies many graphical and tabular illustrations of the ways measures tend to change as the prevalence of an outcome changes.  See the Conference Presentations subpage of the Measuring Health Disparities.  But rather the employing the .50 standard deviation between means underlying the BSPS table, Table 1 is based on a difference between underlying means of .87, which is based on the 2013 rates of failure to achieve proficiency for low and high low income fourth graders shown in the study’s Table 1  (80 percent for low income and 49 percent for high income). 

The table shows the ratios of the advantaged group’s rate of falling above each point to the disadvantaged group’s rate of falling above the point and the ratio of the disadvantaged group’s rate of falling below each point to the advantaged group’s rate of falling below each point.  These shows that, regardless of the rates ranges at issue, general increases in favorable outcome rates tend to result in decreased relative differences in favorable outcome rates and increased relative differences in adverse outcome rates, and I will not discuss them further.  But, by means of color-coding the table illustrates certain points about absolute differences.

The Education Trust Glass Ceiling study examined changes in racial/ethnic and income differences among fourth graders in falling below the basic level and in achieving the advanced level during a period when, as in the Casey Foundation study, achievement levels were improving.  The Education Trust study measured disparities in terms of percentage point differences.  The situation for falling below basic is roughly reflected in the blue-colored rows (K through O).  General decreases in rates of falling below basic accord with movements down the table and decreases in absolute differences between rates tend to decrease.  On the other hand, as roughly reflected in the red-colored row (A through D),  rates of reaching the advanced level are in ranges where improvements (reflecting by moving down the table) tend to increase absolute differences between rates.

As shown in Table 2 of the Casey study, proficiency/non-proficiency are in ranges reflected by the green highlighted rows (G through I) where general improvements will tend usually to increase absolute difference.  That may change, however, with continued overall improvements in some states.

Table 1:  Rates at which advantaged group (AG) and disadvantaged group (DG) fall above and below various levels where underlying means differ by .87 standard deviations, with measure of differences (ref b5312b1)

 

Cut Point	AGAbove	DGAbove	AGBelow	DGBelow	AG/DG Ratio Ab	DG/AG Below Ratio	Abs Df
A	1.00%	0.07%	99.00%	99.93%	14.29	1.01	0.01
B	3.00%	0.30%	97.00%	99.70%	10.07	1.03	0.03
C	5.00%	0.60%	95.00%	99.40%	8.28	1.05	0.04
D	10.00%	1.58%	90.00%	98.42%	6.34	1.09	0.08
E	20.00%	4.36%	80.00%	95.64%	4.58	1.20	0.16
F	30.00%	8.15%	70.00%	91.85%	3.68	1.31	0.22
G	40.00%	13.14%	60.00%	86.86%	3.05	1.45	0.27
H	50.00%	19.22%	50.00%	80.79%	2.60	1.62	0.31
I	60.00%	26.76%	40.00%	73.24%	2.24	1.83	0.33
J	70.00%	36.50%	30.00%	63.50%	1.92	2.12	0.33
K	80.00%	49.20%	20.00%	50.80%	1.63	2.54	0.31
L	90.00%	65.91%	10.00%	34.09%	1.37	3.41	0.24
M	95.00%	77.94%	5.00%	22.07%	1.22	4.41	0.17
N	97.00%	84.38%	3.00%	15.63%	1.15	5.21	0.13
O	99.00%	92.72%	1.00%	7.28%	1.07	7.28	0.06

 

 

Like the Education Trust study (and the other studies addressed on the Educational Disparities page and its subpages), the Case study failed to recognize that measures change as the prevalence of an outcome changes.  I note, however, that in 2014 the Annie E. Casey Foundation issued a study titled “Race for Results:  Building a Path of Opportunity for all Children,” which employed a methodology that is in the direction of the approach that I recommend for measuring differences in the circumstances of advantaged and disadvantaged groups reflected by their differing outcome and that appears to be aimed at addressing the problem arising from the fact that any value for standard measure reflects different strengths of association at different levels of overall prevalence.