Based on Krus, D.J., & Helmstadter, G.C. (1993) The problem of negative reliabilities. Educational and Psychological Measurement, 53, 643-650.

THE PROBLEM OF NEGATIVE RELIABILITIES

David J. Krus and Gerald C. Helmstadter
Arizona State University

 

Negative coefficients of reliability, sometimes returned by the standard formulae for estimation of the internal-consistency reliability, are neither theoretically nor numerically correct. Alternate strategies for test development in this special case are suggested.

 

The occurrence of negative coefficients of the internal consistency reliability is puzzling to both test users and test developers. It presents a perennial problem for writers of textbooks on measure­ment theory who typically marshal a variety of reasons to explain this phenomenon. For example, Magnusson (1966, p.67) maintains that

 

“when the computations are based on a small sample of individuals, and the reliability coefficient is zero for the population, one may obtain negative coefficients of reliability” 

 

Discussing the properties of the coefficient of reliability, Wiersma and Jurs (1990, p.166) assert that

 

“it has been noted that, theoretically, reliability coefficients can take on values from 0 to +1, inclusive. In practice, however, negative reliability coefficients may appear. In this case, the reliability coefficient is reported to be zero”

 

The purpose of the present discussion is to address this and some other issues surrounding the appearance of the negative coefficients of internal consistency reliability.

Background

 In the context of the Spearman-Yule theory of true and error scores, the internal consistency reliability is defined as

 

(1)

 

The Spearman (1910) and Brown (1910) computational formula for estimation of reliability is based on splitting the test into two halves, a and b, and substituting into their formula to obtain

 

(2)

 

the split-half coefficient of reliability. The constant 2 in the numerator of the formula is associated with the fact that the original test was halved. As the result was an estimate of the reliability of a test twice as long as each half, the formula became known as a prophecy formula rather than as a coefficient of internal-consistency reliability based on the coherence of the two halves of the test.

 

The Spearman-Brown conceptualization of test reliability seems to have led to a particular simplification of a formula which expresses the more general notion that if items would correlate with each other as closely as with themselves the test would be perfectly internally consistent (Cronbach, 1951). Consider a test composed of k items and its corresponding matrix of inter-item correlations R, which for r = 2 becomes

 

(3)

 

The coefficient of reliability  can be written as a proportionally adjusted ratio of inter-item correlations to all elements of matrix R which for k = 2 can be conceptualized as

 

(4)

 

Simplification of Eq. 4 for k = 2 as

 

(5)

 

leads to the Spearman-Brown formula (Eq. 2), associating the coefficient of reliability with the length of the test. This traditional simplification of Eq. 4 for k = 2 likely led psychometricians to ignore another possible simplification of Equation 4, namely

 

(6)

 

expressing reliability as a ratio of the correlation between the two halves of the test to its first principal component and providing a clue to the factorial composition of the test or a test battery analyzed.

The Second principal component of the coefficient of Reliability

Consider again the correlation matrix R in Equation 3. This matrix contains two eigenvalues  and , corresponding to the determinant of the  term of the characteristic equation

(7)

 

and equal to

 

(8)

 

and

 

(9)

 

Dividing by the trace of the matrix R, the variance contributions  and  of the first (Eq. 8) and second (Eq. 9) principal components of R (Eq. 3) are defined as

 

(10)

 

and

 

(11)

 

In turn, the coefficient of correlation can be defined in terms of the variance contributions of its principal components as

 

(12)

 

The Equation 10, the denominator of the Equation 6 indicates that the Spearman-Brown formula is interpretable as a ratio of the intercorrelations between the split halves of a test to the variance contribution of the first principal component of its corresponding data (Krus and Helmstadter, 1987).

For the positive values of the  coefficient, Equation 6 correctly estimates the reliability, as defined by the Equation 1. For the negative values of the  coefficient, Equation 6 does not correctly estimate the coefficient of reliability, unless one allows the variance of the true scores to assume negative values. That would involve imaginary numbers, which is not plausible.

 

Let us consider the coefficient of correlation as defined by the Equation 12 as the difference of the variance contributions of its principal components. Note that one can rewrite Equation 6 by substituting the right-hand side expression in the Equation 12 for the  coefficient in the Equation 6 as

 

(13)

 

If the  coefficient of correlation is negative, the variance contribution of the second principal component becomes the best estimate of the obtained variance and the variance contribution of the first principal component becomes the best estimate of the error variance in the Spearman-Yule theoretical model of reliability. The correct estimate of the internal consistency reliability is in this case the expression

 

(14)

 

Substituting from (10) and (11), the Equation 14 can be written as

 

(15)

 

Consider the properties of the Spearman-Brown coefficient of reliability as defined by Equation 1 and the coefficient of reliability as defined by Equation 15 for the negative values of the  coefficient of correlation. For the positive values of this coefficient, the S-B formula correctly estimates the split-half coefficient of reliability. For the for the negative values of the  coefficient of correlation, the reliabilities estimated by the S-B formula are not only theoretically wrong, taking on the negative values, but also of the wrong magnitude.

Discussion

We should stress the theoretical significance of the  coefficient of reliability, demonstrating that the negative coefficients of reliability returned by the extant formulae are neither conceptually, nor numerically correct. In those, admittedly rare instances, the researcher should consider the possibility that the test consists of more than one dimension and that the dimensions are either (approximately) orthogonal or negatively related. To verify this possibility, supporting factor analytic studies of the test battery or its subscales is recommended. If the test or a test battery is indeed bi-factorial or multi-factorial, separate coefficients of reliability should be calculated for each dimension.

References

 Brown, W. (1910). Some experimental results in the correlation of mental abilities. British Journal of Psychology, 3, 296-322.

Cronbach, L (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297 37

Krus, D. J. and Helmstadter, G. C. (1987). The relationship between correlations and internal consistency notions of test reliability. Educational and Psychological Measurement, 47, 911-915.

Magnusson, D. (1966). Test theory. Rending, MA: Addison-Wesley.

Spearman, C. (1910). Correlation calculated from faulty data. British Journal of Psychology, 3,271-295.

Wiersma , W. and Jurs, S. 0. (1990). Educational Measurement and Testing. (2nd Ed.) Boston: Allyn and Bacon.