Statistical Significance SR
(May 15, 2010)
The Statistical Significance sub-page of the Vignettes page of this site will eventually treat a variety of issues concerning statistical significance, mainly relating to the way the concept is misunderstood. Such treatment will include criticism of the treatment of the level of statistical significance as a reflecting the strength of an association, given that a z-score is a function of the size of the sample as well as the strength of association.
Notwithstanding such criticism, however, there is reason to consider whether in circumstances where the sizes of the samples do not change, the z-score might provide a measure of association unaffected by the overall prevalence of an outcome. The fact that the same features of normal distributions that underlie the z-score underlie the approach described on the Solutions subpage of the Scanlan’s Rule page suggests reason to think that z-score might provide such a measure even if its practical utility might be limited.
Table A below explores that possibility using data from BSPS 2006 Table 1. BSPS Table 1 sets out the success and failure rates for the advantaged and disadvantaged groups (at various cutoffs defined by the advantaged group failure rates) where the difference in means scores differ by .5 standard deviations. Table A below determines the Z-score based on the hypergeometric method assuming that each group is comprised of 100 persons. In order for the z-score to provide a measure of association unaffected by the overall prevalence of an outcome, it must remain unchanged as the cutoff is raised or lowered. (It of course will vary depending on the sample sizes.) As shown in Table 1, however, the z-score changes as the cutoff is raised or lowered and hence the z-score does not provide a measure of difference between outcome rates that is unaffected by the overall prevalence of an outcome.
|
Table A Illustration of z values for bsps T 1 (pools of 100)
|
|
CutPoint
|
AGPas
|
DGPass
|
Z
|
|
A 99
|
1.00%
|
0.24%
|
0.68
|
|
B 97
|
3.00%
|
0.87%
|
1.10
|
|
C 95
|
5.00%
|
1.62%
|
1.34
|
|
D 90
|
10.00%
|
3.75%
|
1.75
|
|
E 80
|
20.00%
|
9.01%
|
2.21
|
|
F 70
|
30.00%
|
15.39%
|
2.47
|
|
G 60
|
40.00%
|
22.66%
|
2.64
|
|
H 50
|
50.00%
|
30.85%
|
2.76
|
|
I 40
|
60.00%
|
40.52%
|
2.76
|
|
J 30
|
70.00%
|
50.80%
|
2.78
|
|
K 20
|
80.00%
|
63.31%
|
2.62
|
|
L 10
|
90.00%
|
78.23%
|
2.28
|
|
M 5
|
95.00%
|
87.29%
|
1.92
|
|
N 3
|
97.00%
|
91.62%
|
1.64
|
|
O 1
|
99.00%
|
96.56%
|
1.17
|
The fact that the values change notwithstanding that the strengths of association and sample sizes do not change seem also to raise an issue about the validity of the measure. And it does show that the likelihood of finding a statistical significance is greater where outcome values are in middle ranges. Possibly these issue have been frequently addressed by statisticians.
|