Variance Components of Multiple Regression

One of the basic properties of the multiple regression model is that, in the standard form, the coefficient of multiple determination equals the standardized variance of the predicted scores, i.e., that

 

                                                                   

 

Let us use this basic relationship to describe the variance components of regression weights.

Variance Components of Regression Weights

The variance of the predicted scores can be also written as

 

                                                                  

 

For two predictor variables

 

                                                             

 

and the coefficient of multiple determination can be written as

 

                                                         

 

Expanding the above expression results in

 

                                            

 

Terms with the summation signs in the above expression signify the variance of the first predictor variable, coefficient of correlation, and variance of the second predictor variable. Recall that variance of the standard scores equals one. Thus, the above equation can be simplified as

 

                                                       

 

The above equation shows that the variance components of the coefficient of multiple determination are amounts of variances indexed by the beta weights and their covariates. The presence of the covariance term makes the interpretation of beta weights difficult. However, if the correlation between predictor variables equals zero, the covariance term disappears and the above equation can be written as

 

 

 

At this point, consider the matrix algebra equation

 

                                                                 

 

which can be expanded, for two predictors, as

 

                                                            

and

                                                            

 

Since, if the predictor variables are not correlated

 

 

and

 

Thus

 

 

The above equation can be satisfied if

 

and

 

 

Inspacting the above equations, we may notice a close affinity between rehression and multiple regression in the special case the predictor variables are not correlated. In the regression analysis

                                                                  

 

                                                                    

 

                                                                  

and

                                                                   

 

In the special case when the predictor variables are not correlated, in the multiple regression analysis

                                                                   

and

                                                        

 

Multiple Regression Analysis with Orthogonal Predictor Variables     

Let us build a multiple regression model with two predictors that are not correlated. The data matrix for this example is

 

                                                            

 

In standard scores, the regression model for the above data is

                           

 

Unlike within the regression models for the correlated predictor variables, the variance contributions of the predictor variables sum to the variance of the predicted scores.

Orthogonal Correlation Structure of Multiple Regression Analysis

By correlating variables in the above table, one can obtain the orthogonal correlation structure of the multiple regression analysis, in terms of the coefficients of determination, as

                                       

 

The identity submatrices indicate which components are orthogonal. The variance contributions of each predictor variable sum to the variance of the predicted scores and the variances of the predicted and error components sum to one, the variance of the criterion variable. For the example, .55 + .03 equals .58, and .58 + .42 equals 1.00. The error component is not correlated with any other component of the model, but the criterion variable, where it indexes the amount of the variance that was not predicted. The multiple regression analysis when the predictor variables are not correlated is simple, elegant, and interpretable.

There are three main ways to construct regression models with orthogonal predictor variables. The method of successive partialling and the method of principal components analysis make correlations between predictor variables equal to zero. The Helmert's method of orthogonal contrasts is an algorithm for generating variables that are not correlated. We will discuss each one of these methods in turn.