James P. Scanlan, Attorney at Law

Case Study II

(June 10, 2013; rev. Jun 7, 2014)

Prefatory note: This subpage is related to Issues 1 and 2 of the Case Study subpage of the Scanlan’s Rule page of jpscanlan.com, which (as clarified by the Case Study Answers  subpage) addresses the problematic nature of standard measures of differences between outcome rates for appraising the likelihood that an employer engaged in discrimination.  Somewhat more elaborate treatments of the point, which also explain a sound method for generally appraising the strength of the forces causing a pair of rates to differ, may be found in Section D of the  Harvard University Measurement Letter (Oct. 9, 2012); Appendix B of the Federal Reserve Board Measurement Letter (Mar. 4, 2013and Section B of the Department of Justice Measurement Letter (Apr. 23, 2012); Section B of the University of Kansas School of Law Faculty Workshop paper “The Mismeasure of Discrimination” (Sept. 20, 2013); and  Section B of Federal Committee on Statistical Methodology 2013 Research Conference paper  “Measuring Health and Healthcare Disparities”(Apr. 2014).  The purpose of the instant subpage is to illustrate implications, with respect to the quantifying of the forces causing a pair of rates to differ, of the fact that sometimes certain parts of the pool of persons potentially subject to a process receive no attention at all from that process (a matter addressed at pages 21-22 of the Kansas School of Law paper.  The issue is somewhat related to the subject of my "Measuring Hiring Discrimination" (Labor Law Journal, July 1993), which involved the interpretation of data in a study of the way “tester” job applicants were treated.

***

Table A below is a slight variation on Table 1 of the Case Study subpage, which presented hypothetical hire rates of an advantaged group (AG) and a disadvantaged group (DG) along with various measures of differences between those rates or between the opposite outcome (rejection) rates.  But, whereas Table 1 of the Case Study subpage presented relative differences between hire rates and relative differences between rejection rates, Table A presents rate ratios for hire and rejection.  This has lately been my preferred method of presenting such information and it was used in the illustrative tables of the letters to Harvard, the Federal Reserve Board, and the Department of Justice.  See Harvard Letter a 6 n6.  The difference between rate ratios and relative differences is only a formal one and neither it, nor the choice of which rate to use in the numerator of a rate ratio, has any bearing on the issues addressed here. 

Table A.  Hypothetical Patterns of Hire Rates of Advantaged and Disadvantage Groups at Four Employers and Measure of Differences between Rates of Hire or Rejection (Based on .5 Standard Deviation Difference Between Means of Underlying Normal Distribution)

As explained in various places (perhaps best in the Harvard Letter), while each of the four standard measures of differences between outcome rates would rank these employer differently with respect to the strength of the forces causing AG’s and DG’s rates to differ, so far as can be divined from the information in the table, there is no difference among the employers.  Each reflects a situation where the difference between the means of the underlying hypothesized distributions of factors related to selection or rejection is .5 standard deviations.

But that reasoning is based on an assumption that all applicants are considered by the employer.  There may be many situations where all applicants are not considered.[i]  For example, an employer may receive so many more applications than there are positions to be filled that it will examine only a proportion of those applications.  Similarly, when filling particular vacancies, an employer may examine only applications submitted within a certain period prior to the date of the vacancy. 

Table B is an effort to model such situations based on the assumptions that (a) each employer’s applicants are equally divided between AG and DG; (b) each employer examines only 2.5 times as many applications as there are position to fill; and (c) the process of determining which applications will be considered is neutral with respect to whether an application is submitted by a member of AG or DG.  Thus, the figures in the AG and DG hire rate columns reflect the proportion of each group’s applicants whose applications were examined who were hired – i.e., fractions with the same numerators as in Table 1 but with denominators reduced as a result of the employer’s limiting its review to 2.5 times as many applications as there were positions to fill.   As with Table A, the other figures are calculated on the basis of the data in the hire rate fields. 

The rankings have been eliminated because it has already been demonstrated that they are essentially meaningless.  But the EES (for “estimated effect size”) reflecting the difference between the underlying means estimated on the basis of each pair of selection rates have been added as the final column.

Table B.  Hypothetical Patterns of Hire Rates of Advantaged and Disadvantage Groups at Four Employers and Measure of Differences between Rates of Hire or Rejection, with Estimated Effect Size, Based on Applications Actually Examined by the Employer.

Because employers C and D did not have more applications available than the number they would typically consider (given that the overall hire rates were above 50% at both employers and hence that the employers did not have even twice as many applications as they had positions to fill), those employers examined all applications.  Thus, all values for those employer remain as they appeared in Table A. 

In the case of employers A and B, however, the only figure that remains the same is the ratio of AG’s selection rate to DG’s selection rate.  That figure is unchanged because the two groups experience equal proportionate reductions in the denominators of the fractions that constitute their selection rates and thus experience equal proportionate increases in their selection rates.  Ratios of two rates remain constant when the rates change equal proportionate amounts. 

But the rejection rate ratio changes because applications that were not considered at all, and which previously were regarded as rejected applications, are subtracted from the numerator for the rejection rate.  That subtraction does not constitute a proportionate reduction of the numerator but constitutes a larger proportionate reduction in the group with the smaller number of rejections.  I will not belabor the reasons that the absolute differences and odds ratios also change.

It warrants note, however, that because the selection rate ratio does not change, observers who rely on that measure are inclined regard it as unnecessary to consider whether all applications are considered. 

The key figure, however, is the EES.  These figures for employers A and B are higher than before.  And it is these higher figures that better reflect the strength of the forces causing the AG and DG selection rates to differ at these employers.  So under this scenario, the strength of those forces would appear to be greater for employer A than employer B and for both employers A and B than employers C and D. 

Thus, proper analyses of employment discrimination claims must consider only those applications that actually were examined by the employer. 

I am at this point uncertain as to the range of contexts other than analyses of employment discrimination claims where similar issues are implicated.   Such issues would not seem to apply to lending claims since all loan applications are considered by in the decision-making process being examined.

The issue may be implicated in the subject of Table 2 of the Harvard Letter (at 22), which addresses changes in black and white coronary artery bypass graft (CABG) rates during a period of general increases in CABG rates.  I have previously noted that EES may not be a suitable measure of differences in such circumstance because the risk distribution of myocardial infarction (MI) patients may be truncated portions of larger distributions.  See Second Comment on Werner Circulation 2005.  An additional problem would exist if, say, only 10% of acute myocardial infarction patients were suitable candidates for CABG.  In that situation, rather than a reduction of the EES from .58 to .48, as shown in the table, one would observe an increase in the EES from .98 to 1.37.[ii]