Factor Analysis

Let us consider an extended definition of the specification equation of the variance components of the total variance. So far we have discussed only the specification equation for the multiple regression, partitioning the total variance in the criterion variable into the information and error components. Sometimes it is feasible to conceptualize the specification equation as that the total variance consists of three components: common variance, variance specific to a variable and error variance, i.e.,

 In the above equation the common variance was designated as , the specific variance as and the error variance as .

Total Variance

Before we can estimate the common variance components, we have to solve for eigenvalues indexing the total variance of the data set. As an example, consider a matrix of inter-correlations between two variables and , presented in the following table.

Solving for eigenvalues as in the course of the principal components analysis we find the first eigenvalue,  equal to 1.60 and the second eigenvalue,  equal to .40. At this point we have to determine the number of factors to be extracted.

Number of Factors to Extract

Humphrey-Ilgen Parallel

Several criteria were proposed in this respect. Perhaps the best is the Humphrey-Ilgen Parallel Analysis. To use this procedure, two data matrices are factor analyzed simultaneously and their eigenvalues are plotted. One is the analysis of the 'real' data. The second is the analysis of random numbers. The matrix of random numbers should have the same size as the matrix of "real" data. The intersection point of eigenvalues from both solutions determines the number of factors to be extracted.

A graphical rendering of this procedure, based on hypothetical data is presented below, indicating that three “nonrandom” factors should be extracted.

 

Cattel's Scree Test

Another criterion is Cattell's Scree Test. Eigenvalues, this time obtained from a single analysis, are plotted and an inflection point of the resulting curve (scree) is determined by visual inspection. The location of the inflection points indicates the number of factors to be extracted.

 

 

Kaiser's Criterion

The third procedure often used in this context is Kaiser's Criterion, stating that, as many factors should be extracted as variables with eigenvalues greater than or equal to one. The rationale behind this criterion is that interpretation of proportions of variance, smaller than the variance contribution of a single variable, are of dubious value. Kaiser's criterion is the one most frequently used since it does not require visual inspection of eigenvalue plots and is easily computerized. For our example, there was one eigenvalue greater than one (), thus one factor should be extracted according to this criterion.

Communality

Factor analysis is a statistical technique used to identify a smaller number of underlying dimensions, or factors, that can be used to represent relationships among interrelated variables. The amount of the common factorial variance is initially unknown and has to be estimated. The most often used method for obtaining the communality estimate is to find the squared multiple correlations of each variable with all other variables. This could be a formidable task if larger data sets were analyzed. However, squared multiple correlations can be obtained directly from the diagonal of the inverted correlation matrix. Consider a correlation matrix

The inversion of the above matrix is

The squared multiple correlation, considering the variable X₁ as a criterion and the remaining variables (for the example X₂) as predictors, can be computed as

For our example of two variables, the squared multiple correlation is .36, equal, indeed, to the squared original coefficient of correlation.

Iterations in Factor Analysis

The equation to be solved to determine eigenvalues of the principal factor solution proper is

where  is a matrix of the reduced rank, i.e., a correlation matrix with communality estimates replacing the unit values in the main diagonal. For the example

Computing the determinant

results in a quadratic equation

Solving by quadratic formula, for the first root only (only one factor has to be extracted),

so

and

Division of the eigenvalue by the trace of the original correlation matrix R (for our example the trace equals 2) gives the proportion of the variance extracted as .96/2 = .48. This value is used as a new communality estimate and the process is repeated. For two variables, the iteration process can also be written in an abstract form as

Solving the determinant

and expanding the binomial

gives the first root of the quadratic equation as equal to

so

The variance contribution is computed as

and for iteration

Thus, for the example, the initial estimate of communality .  Iterating,

Since r is a constant

and, for the case of two variables, the estimated communality must thus be equal to the correlation of the variable with other variables; for our example equal to .60. This makes sense, since in the case of two variables, the coefficient of correlation indexes the amount of shared variance between them.

Factor Analysis vs. Principal Components Analysis

The factor analysis is obtained by finding the characteristic roots (eigenvalues) and vectors (v) of the correlation matrix with the squared multiple correlations each variable with other variables in the main diagonal. On the other hand, the principal components analysis is obtained by finding the characteristic roots (eigenvalues) and vectors (v) of the correlation matrix with 1`s in the main diagonal. Thus the principal components analysis analyzes the total variance of variables included in the analysis and factor analysis analyzes the variance common to variables analyzed.

Eigenvectors

The computation of eigenvectors follows the same procedure as described earlier for the principal components solution.

Structural Solution

The characteristic equation

 

 

is solved, leading to the structural, indeterminate solution

 

 

The structural solution thus can be conceptualized as

 

 

Restriction

Imposing the restriction that the column sum of the squared factor loadings has to equal the first eigenvalue ()

 

 

Matrix of Factor Loadings

By taking square roots of the above eigenvector results in a matrix (in this case a vector) of factor loadings, presented in the following table.

 

 

As pointed out earlier, a factor loading is a product-moment correlation of a variable with a factor. The matrix of factor loadings thus permits interpretation of a factor in terms of common properties of variables.