Cruise Scientific   ¨   Visual Statistics Studio   ¨   Measurement and Scaling

Abbreviation of Krus, D.J. & Blackman, H. S. (1988) Test reliability and homogeneity from the perspective of the ordinal test theory. Applied Measurement in Education, 1, 79-88.

Test homogeneity from the perspective of the ordinal test theory
David J. Krus
Arizona Stare University
Harold S. Blackman
Idaho State University

 

Abstract.- Algebraic formulation of the coefficient of homogeneity, complemented by visualization of relationships underlying this coefficient by ordered graphs adjacent to matrices of the expanded components of variance is discussed within the framework of theoretical considerations pertaining to properties of hierarchical structures of data matrices.

 

Etymology of the term, scale can be traced to the Latin word scala meaning “a ladder.” Homogenous data matrices should form ladder-like structures and an index of homogeneity should reflect the degree data structures approximate this ideal lattice form. Loevinger’s (1948, p. 519) formulation of this coefficient involved complicated algebraic considerations, however her underlying rationale was straightforward; to develop a coefficient reflecting the ratio of the observed variance to the maximum possible variance that could have been obtained from the same data, provided the total variance remains constant and the item difficulties stay invariant.

 

The Imaginary Jar

 

To form a mental picture of this idea, imagine a jar filled with the mixture of oil and water where oil bubbles are raising toward the surface along the vertical paths. Shaking the jar decreases the variability of the mixture while the gradual separation of the oil and water increases the variance between the oil and water molecules. The minimal algebraic model of this conceptual model is shown in Figure 1

 

 

Calculation of column means (item difficulties), column variances, and the total variance (irrespective of the row-column structure of the data) for both data matrices, as shown in Figure 2

 

 

 

demonstrates that these indices remain constant, as the reordering of data within the columns of data matrices changes only the variance along the vertical (row) dimension of our imaginary jar.

 

 

Cepalic stratification of variables

 

For reasons related to construction of dendrograms reflecting the hierarchical structure of the marginal referents of data matrices, we have to introduce the notion of the expanded (exp) variance components. As traditionally conceptualized, a variable has three cepalic (from L. caepa, onion) components: a core, enveloped by two layers.

 

                                              

 

The outermost layer is characterized by the arithmetic mean, associated with the obtained scores. The mean can be removed from the obtained scores by subtraction. By removing the outermost layer, we obtain the deviation scores (from the arithmetic mean). Since the arithmetic mean was peeled off from the obtained data, the mean of the deviation scores is always zero. The statistics most closely associated with the second layer is variance. It can be removed by dividing deviation scores by its square root, thus removing the second layer. What remains is the core; the standard scores with mean equal to zero and standard variance equal to one, i.e.,

 

                                                           

 

where  are the obtained scores,  are the deviation scores (  ), and  are the standard scores . We can also add layers to the obtained scores. This was done by Ronald Fisher who coined his analysis of variance in terms of extended (ext) scores with variance components he called the sum-of-squares. Here we add another layer, called the expanded (exp) scores, as

 

                                       

 

where  are the extended scores and  are the expanded scores. In the case of a single variable, the variance of the extended scores equals  and the variance of the expanded scores equals .

 

Expanded Components of Variance

 

Within the above context, the total expanded variance of a data matrix X is defined (cf., Krus and Wilkinson, 1986) as

 

(1)                                                      

 

where  symbolizes elements of the decomposed matrix X. This total variance can be partitioned into its row component

 

(2)                                                 

 

and its column component

 

(3)                                                 

 

For the discussed instance of the data matrix X, the variance of the elements of the decomposed matrix X equals 6.234. Using Eq. (1), its corresponding expanded variance can be computed as  which equals 399. The  expression on the right-hand side of Eq. (2) equals

 

(4)                                                    

 

The expanded variance component for the rows of this instance of the data matrix equals . The expanded variance component for the columns of this instance of the data matrix X is 81. The  expression for the ordered data equals

 

(5)                                                    

 

The expanded variance component for the rows of this ordered data matrix equals 307. The expanded variance component for the columns of this ordered data matrix remains equal to 81.

 

Hierarchical Structures

 

At this point, our conceptual model can be visualized as shown in Fig. 2 and Fig. 3, depicting dendrograms associated with the skew-positive matrices in (4) and (5)

 

 

 

Information content of THE DATA matrix

 

The relationship between the expanded variance of the data matrix X,

 

                                                             

 

the column-additive function

 

                                                         

 

binarized as shown in Fig. 4.

 

 

and the information conceptualized in terms of (1,0) bits of the information theory is defined by identities in Eq. 6

 

(6)                                         

and in Eq. 7

(7)                                                   

 

For the discussed instance of data matrix X, the identities defined by Eq. 6 are

 

                             

 

 

 

                               

These identities demonstrate that, invisible, reflected by magnitudes of the extracted variance components as defined by Equation. 7, the binary digits (bits) of the information theory determine the information value of empirical data.

 

Information content and properties of the ordered DATA matrix

 

The identities, defined by Eq. 6 and Eq. 7 hold also for the ordered data matrix X*,

                                                            

and its associated column-additive function  

 

                                                         

binarized as shown in Fig. 5

 

 

 

The binary matrices B and B* contain information about the degree scales s and s* approximate the theoretical implicational scale S*. Let us consider a logical function

 

(8)                                              

 

where the right arrow symbolizes the implication function of the propositional calculus with its associated truth table

 

                                                            

and the ampersand signifies the conjunctive function with its associated truth table

 

                                                             

 

For an instance of three attributes, this theoretical structure can be demonstrated as shown in Fig. 6.

 

 

 

The characteristic triangular patterns of zeroes and ones in the supra- and infra-diagonal regions of data ordered in the ascending direction is accentuated in (Fig. 7)

 

 

These theoretical considerations suggest that the coefficient of homogeneity reflects the degree to which the data are congruent with the principles of sound logical reasoning.

 

 Coefficient of Homogeneity

 

The expanded variance components, computed by Equations 1, 2, and 3, were arranged in Table 1 and adjusted by their respective degrees of freedom.

 

Table 1. Expanded variance components for the obtained and ordered data.

 

 

According to postulates of the theory of the true and error scores (Gulliksen, 1950), the obtained scores consists of the true and error components and the true variance can be obtained by subtracting the error variance from the obtained test scores. Thus, forming a ratio of the (adjusted) variance components of the obtained and ordered data, the coefficient of homogeneity  can be defined as

 

(9)                                                

 

For the discussed instance of the data matrix, the coefficient of homogeneity can be computed by using Eq .(9) as (94.333 - 11.667)/(102.333 – 3.666) which equals .838. Since the expanded coefficients of variance are proportional to the sum of squares of the analysis of variance (cf., Hoyt, 1941), the coefficient of homogeneity can be also defined as

 

(10)                                                  

 

using the sums-of-squares and the mean-squares (MS) variance components. In terms of the standard components of variance, shown in Table 2,

 

Table 2. Standard variance components for the obtained and ordered data.

 

the coefficient of homogeneity can be defined as

 

(11)                                                 

 

For the discussed instance of the data matrix, the coefficient of homogeneity can be computed by using Eq .(11) as (.236 - .029) / (.256 – .009) which equals .838. Visualizing the extracted components as shown in Figure 8,

 

Fig. 8. Proportions of variance accounted for
by the extracted variance components

 

we may conclude that about 6% of the total variance is “illogical”, i.e., is deficient with respect to ordinality of subjects’ responses to the questions (stimuli), defining the column marginal referents of your data.

 

DISCUSSION

 

Among the advantages of rendering test homogeneity using concepts and computational procedures developed within the framework of the ordinal test theory is the option to visualize the relationships between the row (or column) marginal referents of the data matrices by ordered graphs. Since the branches of dendrograms adjacent to skew-positive dominance matrices indicate the presence or absence of logical contradictions in the analyzed data, the internal consistency of data matrices can be explicated in terms of the degree they are compliant with the principles of logical reasoning.

 

REFERENCES

 

Gulliksen, H. (1950). Theory of mental tests. New York: Wiley.

Hoyt, C. (1941). Test reliability estimated by analysis of variance, Psychometrika, 6, 153-160.

Krus D. J. (1977), Order analysis: An inferential model of dimensional analysis and scaling. Educational and Psychological Measurement, 37, 587-601.

Krus, D. J. & Ceurvorst, R, W. (1979) Dominance, information, and hierarchical scaling of variance space. Applied Psychological Measurement, 3, 515-527.

Krus, D. J., & Kennedy, P. H. (1980) Dimensionality of hierarchical and proximal data structures. Applied Psychological Measurement, 4, 313-321.

Krus, D, J., & Wilkinson. S. M. (1986). Matrix differencing as a concise expression of test variance. Educational and Psychological Measurement, 46, 179-183.

Loevinger, J. (1947). A systematic approach to the construction and evaluation of tests of ability. Psychological Monographs, 61(Whole No. 4).

Loevinger, J. (1948). The technic of homogeneous tests compared with some aspects of scale analysis and factor analysis. Psychological Bulletin, 45, 507-529.