Employment Tests
(Aug. 27, 2008)
In challenging the perception that large relative difference in adverse outcome rates reflect large disparities I have often noted that the lowering of test cutoffs is universally regarded as reducing the discriminatory impact of employment tests, because doing so reduces relative differences in pass rates, even though it increases the relative difference in failure rates.
Few, however, actually recognize that the lowering of a cutoff tends to increase relative differences in adverse outcome rates. And at least one teacher competency test was challenged based on the large relative difference in disqualification rates. See A3of MHD. A court of appeals once upheld a claim that a performance standard was too high on the basis of a large relative difference between the rates of failing to meet the requirement. See A6 of MHD (“Getting is Straight When Statistics Can Lie,” Legal Times (June 28, 1993). Improvements in training that would allow all employees to succeed on the job have been suggested as a way to reduce a large racial relative disparity in rates of termination for poor performance. Where refusal to hire applicants with arrest records has been deemed to disproportionately disqualify minority job applicants, limiting the disqualifying criterion to conviction records has been suggested as a less discriminatory alternative. In each case, it has gone unrecognized that measures that would allow more persons to succeed would tend to increase the disparity in adverse outcome rates that caused the concern in the first place.
Nevertheless, at least in employment and at least with regard to something that is obviously an employment test, it remains an article of faith that the lower the cutoff, the less discriminatory the test. And intuitively that seems to make a great deal of sense.
But I have repeatedly illustrated that when a cutoff is lowered two measures of difference increase and two measures of difference decrease, and, by definition, the difference between means derived from the pass or failure rates at each cut point remains the same. Does that mean that lowering a cutoff does not affect the discriminatory impact of a test?
In circumstances where everyone who passes the test experiences the favorable outcome and everyone who fails the test experiences the adverse outcome, lowering the cutoff would seem not to reduce the discriminatory impact of a test in a meaningful sense. But in the standard employment setting, it is assumed that among persons who pass the test, persons will be selected for hire or promotion in accordance with criteria that are not closely related to test performance. The implications of such selection can be illustrated by treating the persons selected from among persons in the two groups who passed the test as if they have equal chances of selection.
Table 1 (Emp Test Table) illustrates the implications of random selection among test passers from groups whose average on the test differs by one half a standard deviation. And we observe that the lower the cutoff, the lower will be the derived difference between means. The extent to which such difference, at each cutoff, is lower than half a standard deviation will be greatest where the overall selection rate among test passers is lowest. Such fact, of course, should be evident once one recognizes that the closer the selection rate approaches 100 percent, the closer the situation approaches one where all test passers experience the favorable outcome and all test failers experience the adverse outcome. In any case, there is a sound statistical basis for the perception that lowering cutoffs reduces the discriminatory impact of an employment test where selection from among persons who pass the test is not strongly correlated with test performance.
Interestingly, an important 1982 Supreme Court decision held that the discriminatory impact of a test would be evaluated at the point where the test limited the pool of persons eligible for further consideration. Thus, under the scheme for the appraisal of the impact of a test envisioned by that decision, lowering cutoffs would not reduce the discriminatory impact of a test.
(I have written about that decision, offering a different reason from the Court for reaching the same result. See J Law Reform 1985. The point in the preceding paragraph would apply to my reasoning as well as to the Court’s reasoning.)
There are, however, circumstances involving tests or criteria that operate like tests where all persons who pass the test (meet the criterion) experience the favorable outcome and all who fail the test experience the adverse outcome. These include licensing exams, teacher competency testing where test passage is a requirement to keeping one’s position, on the job performance standards, and mortgage eligibility requirements. In such contexts (and leaving aside issues of whether the ostensibly successful outcome is always in one’s interest), I continue to intuitively regard the lower cutoffs (with corresponding smaller relative differences in success rates and larger relative differences in failure rates) to be less discriminatory. But I cannot find a statistical basis for such view. Probably, it is based on an erroneous equation of the absolute well-being of the disadvantaged group with a smaller discriminatory effect.
|