James P. Scanlan, Attorney at Law

Education Trust High Achiever Study

(April 23, 2014; rev. April 30, 2014)

 

This page is related to the Education Trust GC Study subpage of Page of the Educational Disparities page of jpscanlan.com.  Both subpages involves studies in which the Education trust attempted to analyze racial/ethnic or income differences in outcome rates without recognizing the extent to which observed patterns are functions of normal distribution and thus are not necessarily reflective of anything to do with the underlying processes that are of concern to the study authors.  As discussed on the other subpage, the criticism of the Education Trust’s analyses on this subpage could as well be said of the health and healthcare disparities research of the Harvard School of Public Health, the Harvard Medical School, the Centers for Disease Control and Prevention , the Agency for Healthcare Research and Quality, the National Center for Health Statistics, and the Institute of Medicine, as well as the virtually every other research institution in the world (as recently addressed at pages 24 to 32 of the Federal Committee on Statistical Methodology 2013 Research Conference paper “Measuring Health and Healthcare Disparities.”)

 

This subpage is discussed in an April 30, 2014 letter to the Education Trust that generally discusses problems with the use of standard measures of differences between outcome rates to appraise demographic differences without consideration of the way such measures tend to be affected by the prevalence of an outcome.    

 

***

 

On April 2, 2014, the Education Trust released a report titled “Falling Out of the Lead: Following High Achievers Through High School and Beyond.”  The report was the second of a series, the first of which was a the May 14, 2013 report titled “Breaking the Glass Ceiling of Achievement for Low-Income Students and Students of Color” (which is addressed on the Education Trust GC Study subpage.  As discussed on page 1 of the earlier document, the series is intended to shed light on disparities at the high end of achievement.  As discussed below, it is not possible to shed light on such issues without an understanding of the extent to which observed patterns are functions of normal distributions. 

 

The report focused on the lower rates of subsequent academic achievement of black and Hispanic students compared with white students, and students in the lowest income quartile compared with those in the highest income quartile, among students in the top 25 percent of performers on nationally-administered math and reading tests.  The theme of the report was that high-achieving minority and lower socioeconomic students were not performing as well as would be expected compared with high-achieving white and higher socioeconomic students and that schools may be responsible for this.  An EdSource Today article titled “Achievement gap persists, even among high-performing students, report says,” described the report as follows:  "African-American, Latino and low-income students who start high school performing near the top of their class fall behind other high-achieving peers by the time they graduate, according to a report released Wednesday that offers a new window on entrenched achievement gaps in high school."

But the perceptions in the report and the account of it reflect a misunderstanding of normal distributions.  Persons with an understanding of such distributions would recognize that the results were entirely predictable and not particular suggestive of anything about the way schools advance or retard the achievement of high-performing students from disadvantaged backgrounds.  

Groups that are less likely than other groups to be in high-achievers will be disproportionately represented in the lower segments of the high-achieving population and hence can be expected to have lower achievement rates on most indicators within that population.  Further, the differences in rates at which advantaged and disadvantaged groups within the high-achieving population reach certain levels of academic success can be predicted fairly accurately on the basis of the various groups’ rates of being in the high-achieving population.    

The points can be illustrated with data from Figure 2a (at page 5) indicating that 6 percent of blacks and 35 percent of white were deemed high-achieving on the basis of test scores.  That would suggest that mean scores of blacks and whites differ by about 1.16 standard deviations. 

The 1.16 standard deviation would suggest that 29.5 percent of the high-achieving black students would be above the point reached by the top half of the high-achieving white students and about 10 percent of the high-achieving black students would be above the point reached by the top 25 percent of the high-achieving whites.[1]  Thus, racial differences in achievement among the high-achieving students are entirely to be expected.  Indeed, it would be remarkable to fail to find them.  The same holds for ethnic and socioeconomic differences among the high-achieving students.

One might seek to determine whether the racial or socioeconomic differences among the high-achieving students are larger or smaller than one would expect based on the racial/ethnic and socioeconomic differences in rates of being in the top quarter of all students.  But that can only be done when one has an understanding of what to expect. 

The information in the report or Data Appendix regarding the proportion of higher-achieving black and white students achieving certain measures of success enables us to divine the extent to which the patterns reflected there are consistent with the 1.16 standard deviation difference between means of blacks and whites in overall student population mentioned above. 

The rate at which high-achieving blacks and whites take calculus shown in Table 1 are consistent with a difference between overall means of .98 standard deviations.  That is a rather smaller difference than would be expect based on difference two groups rates of being among the top 25 percent.  The fact that the difference is smaller than would be expected, however, may be somewhat a function of the fact that 94 percent of blacks compared with 87 percent of whites were in schools offering calculus.[2] 

Other indicators of achievement that allow focusing on groupings that can approximate the highest or increasingly higher levels of achievement (and where blacks have lower achievement rates than whites) show results that are fairly close to what one would expect based on the differing black and white rates of being within the top quarter and the associated estimate of 1.16 standard deviations between the means scores.

The black and white rates for taking the SAT or ACT shown in Table 7 are consistent with an difference between overall means of 1.18; the black and white rates for securing a GPA of (a) B or above or (b) A are consistent with overall means of, respectively, (a) 1.17 and (b) 1.22 standard deviations; the black and white rates for (a) attending an inclusive or unclassified 4-year institution or above, (b) attending a moderately selective institution or above, or (c) attending a highly selective institution are consistent with differences between overall means of, respectively, (1) 1.09, (b) 1.12, and (c) 1.09 standard deviations.  All these figures are pretty close to what one would expect based simply on the fact 35 percent of white students compared with 6 percent of black students are in the top quarter of all students. 

This information is set out in Table 1, along with similar information for comparisons of Hispanics with whites and lower socioeconomic groups with higher socioeconomic groups.  The AGHAR and DGHAR columns reflect the proportions of high achieving students from the advantaged and disadvantaged groups in each comparison achieving the identified outcome, while the AGTPA and DGTPR reflect the proportions of the total populations from the advantaged and disadvantaged groups in each comparison achieving the identified outcome.  The EES (for estimated effect size) is based on the latter pair of proportions.   Any conclusions about the extent to which differences in performance of advantaged and disadvantaged groups within the top 25 percent of total students is more than or less than what would be expected given the differences in falling within the top 25 percent should be based on a comparison of the EES figures for particular indicators with the highlighted EES figure based on rates of being among the top 25 percent.

Table 1. Proportions of high-achieving members of advantaged and disadvantaged groups, and proportions of total members of such groups, experiencing various academic outcomes, with estimate of differences between the overall means of each group based on the latter proportions.  [ref n2\b5222b2]

 

Comparison	Indicator	Source	AGHAR	DGHAR	AGTPR	DGTPR	EES
Bl and Wh	High Achievement	F2	100.00%	100.00%	35.00%	6.00%	1.16
Bl and Wh	Highest math calculus	T1	33.00%	24.00%	11.55%	1.44%	0.98
Bl and Wh	Took SAT/ACT	T7	88.00%	76.00%	30.80%	4.56%	1.18
Bl and Wh	GPA = B or better	T8	87.00%	77.00%	30.45%	4.62%	1.17
Bl and Wh	GPA = A	T8	42.00%	19.00%	14.70%	1.14%	1.22
Bl and Wh	Attended four year college	T9	74.00%	68.00%	25.90%	4.08%	1.09
Bl and Wh	Attended moderately or highly selective college	T9	67.00%	54.00%	23.45%	3.24%	1.12
Bl and Wh	Attended highly selective college	T9	34.00%	19.00%	11.90%	1.14%	1.09
Hisp and Wh	High Achievement	F2	100.00%	100.00%	35.00%	11.00%	0.84
Hisp and Wh	Highest math calculus	T1	33.00%	27.00%	11.55%	2.97%	0.68
Hisp and Wh	Took SAT/ACT	T7	88.00%	71.00%	30.80%	7.81%	0.91
Hisp and Wh	GPA = B or better	T8	87.00%	75.00%	30.45%	8.25%	0.87
Hisp and Wh	GPA = A	T8	42.00%	29.00%	14.70%	3.19%	0.80
Hisp and Wh	Attended four year college	T9	74.00%	52.00%	25.90%	5.72%	0.92
Hisp and Wh	Attended moderately of highly selective college	T9	67.00%	41.00%	23.45%	4.51%	0.97
Hisp and Wh	Attended highly selective college	T9	34.00%	34.00%	11.90%	3.74%	0.60
Low and High SES	High Achievement	F2	100.00%	100.00%	48.00%	10.00%	1.23
Low and High SES	Highest math calculus	T1	42.00%	29.00%	20.16%	2.90%	1.05
Low and High SES	Took SAT/ACT	T7	89.00%	77.00%	42.72%	7.70%	1.25
Low and High SES	GPA = B or better	T8	89.00%	83.00%	42.72%	8.30%	1.20
Low and High SES	GPA = A	T8	46.00%	29.00%	22.08%	2.90%	1.12
Low and High SES	Attended four year college	T9	85.00%	54.00%	40.80%	5.40%	1.36
Low and High SES	Attended moderately of highly selective college	T9	78.00%	44.00%	37.44%	4.40%	1.38
Low and High SES	Attended highly selective college	T9	46.00%	16.00%	22.08%	1.60%	1.37

 

The report states (at 2):  “[O]ur analysis focuses on comparing the average performance, which we expect should be similar across racial and socioeconomic groups.”  A footnote attached to that sentence states:  “There are small differences between groups even within the highest achievement quartile. These are described in footnotes in the following section, ‘At the Start.’”  That section then contains two footnotes (notes 9 and 10 at page 4 that suggests differences in number of questions answered by different racial and socioeconomic groups were small.  Not enough data is provided to allow one to know precisely what such figures mean.  But, as indicated above, the difference in rates of being in the top quarter are sufficient to give one a strong basis to believe that one will expect racial differences among higher-achieving students that are very like those observed.  There is no basis to be in any way surprised by these data.  Nor is there a basis for offering hypotheses about the reasons for the differences without recognizing the statistical reasons to expect the differences.