Cruise Scientific                 Visual Statistics Studio                 Measurement and Scaling

Elaboration of Krus, D.J., & Fuller, E.A. (1982) Illustration of theory of true and error scores on idealized data sets.  Educational and Psychological Measurement, 42, 837-841.

 

Conceptual model of the theory of true and error scores
using idealized data sets.

David J. Krus
Arizona State University

 

Summary. - Theory of true and error scores is illustrated on idealized data set demonstrating that the internal consistency reliability, defined as the true scores / obtained scores ratio equals the coefficient of determination, computed as the square of coefficient of correlation between true and obtained scores.

 

The formulation of the Spearman-Brown coefficient of reliability was the culmination of a decade-long effort (Spearman, 1904a, 1904b, 1907, 1910, 1913; Brown, 1910) to develop the body of theory behind the concept of reliability in the sense of internal consistency. The resulting theory of true and error scores embodied principles which later led to the formulation of factor analytic theory and modern test theory.

 

Frequently reduced to a single formula, the theory of true and error scores seems to provide rationale only for adjustment of reliability coefficients with respect to the length of a test. Long forgotten is the initial tour de force implied in the formulation of the theory of true and error scores: the assumption of existence of true test scores known only to an omniscient being, and the subsequent demonstration of the power of human reason to measure the extent of their presence. Also, the link connecting the theory of true and error scores with subsequent developments within the area of measurement theory is frequently only suggested, or is missing.

 

To capture the spirit of discovery inherent in the initial stages of its formulation, to facilitate understanding of its historical relevance, and to promote acquisition of its theorems and assumptions, we constructed an idealized data sets The purpose of this paper is to describe its essential features. 

 

Construction of the Idealized Data Set

Obtained test scores Xo (top line) are assumed to consist of true X_t and error X_e components. The error components are randomly distributed around the true scores (line two). To obtain the true scores, divide the obtained scores into two halves, A and B (line three and four)

    

 

 

 

 

 

and arrange the scores and their associated variance-covariance (var-cov) and correlation (cor) matrices as

 

                         

 

 

Relationships among the components of the model

 

 

 

Notes

A] Means of the split  halves of a test should be equal.

B] Variances of the split  halves should be equal.

C] The assumptions of equality of means and variances was removed by the Kuder-Richardson formula 20 for binary [0, 1] items where p equals the number of 1’s divided by n and q the number of 0’s divided by n.

D] In the Cronbach’s alpha, the item variance terms of binary items in the K-R-20 were replaced by the item variance terms of continuous items.

E] Theoretical means of the obtained and of the true scores are equal. Thoretical mean of the error scores is zero.

F] Theoretical variances of the true and error scores are additive, since the r_te as well as cov_te are equal to zero.

G] Theoretical conceptualization of the internal consistency reliability (r_xx) is as the proportion of true variance in the obtained scores.

H] During the real-life data analysis, only the variables A and B are known, as the X_t and X_e are theoretical concepts. To solve this problem, let us consider the principal components of the correlation matrix R for the A and B variables

 

This matrix contains two eigenvalues,  and , which can be obtained from the determinant of the  term of the characteristic equation

 

 

For the case of two variables, the first eigenvalues equals

 

To get the variance of the first principal component of the coefficient of correlation r_ab, we have to multiply the above eigenvalue by the reciprocal of the trace of the correlation matrix R (in this case equal to 1/2). This allows us to define the internal consistency reliability as a ratio of the correlation r_ab and the variance of its first principal component, i.e.,

 

 

I] Following complex algebraic considerations, Spearman and Brown expressed the above formula as

 

 

 

As the above formula happens to have 2 in the numerator, it is frequently interpreted not as the theoretical precursor of the internal consistency reliability, but as a mere correction for the reliability of a test shortened by the split-half or by the odd-even procedures for obtaining the halves of the test by the factor of two. Subsequently, the S-B “prophecy” formula was modified as (cf., Gulliksen, 1950, p. 78)

 

 

Since

 

 

Gulliksen (ibid., p.81) concludes that “as k approaches infinity, r_xx approaches unity,” leaving aside the question what should be the reliability of a test consisting of an infinite series of random numbers.

Discussion

 

Formulation of the theory of true and error scores is one of the profound intellectual achievements and its reconstruction by using the idealized data sets provides numerous insights into formal aspects of theory building. When asked how he arrived at the multitude of algebraic relations he described during his extraordinarily productive professional life, Karl Friedrich Gauss replied: “durch planmassiges Tattonieren “ - through systematic, concrete experimentation [with numbers]. Frequently, this part of the creative process of building formal structural relationships is obscured by their description in the deductive form, often formulated after the groundwork of the systematic empirical trying was laid out. The method of building a theory by using idealized data sets simulates the heuristic process of discovery of numerical relationships. It can perhaps also convey its accompanying sense of adventure and discovery.

 

References

Brown, W. (1910) Some experimental results in the correlation of mental abilities. British Journal of Psychology, 3, 296-322.

Gulliksen, H. (1950) Theory of mental tests. New York: Wiley.

Spearman, C. (1904) The proof and measurement of association between two things. American Journal of Psychology, 15, 72-101.

Spearman, C. (1904) General intelligence objectively determined and measured. American Journal of Psychology, 15, 201-292.

Spearman, C.  (1907) Demonstration of formulae for true measurement of correlation. American Journal of Psychology, 18, 161-169.

Spearman, C. (1910) Correlation calculated from faulty data. British Journal of Psychology, 3, 271-295.

Spearman, C.  (1913) Correlations of sums and differences. British Journal of Psychology, 5, 417-426.