Variance Components of Multiple Regression

One of the basic properties of the multiple regression model is that, in the standard form, the coefficient of multiple determination equals the standard variance of the predicted scores.

In this chapter, let us use this basic relationship to describe the variance components of the coefficient of multiple determination.

Decompose the Coefficient of Multiple Determination

The standard variance of the predicted scores can be also written as

and

Two Correlated Predictors

Since, for two predictor variables

 

 

the coefficient of multiple determination can be also written as

 

 

Expanding the above expression results in

 

 

Terms with the summation signs in the above expression signify the variance of the first predictor variable

 

 

coefficient of correlation

 

 

and variance of the second predictor variable 

 

 

Thus, the above equation can be simplified as

 

 

The above equation shows that the coefficient of multiple determination can be indexed by the squared beta weights and their covariance term.

Two Orthogonal Predictors

The presence of the covariance term makes the interpretation of beta weights difficult. However, if the correlation between predictor variables equals zero (r12 = 0), the covariance term disappears and the above equation can be written as

 

 

Beta Weights and Correlations

At this point, consider the matrix algebra equation

 

 

which can be expanded, for two predictors, as

 

 

and

 

 

Now, set the above equation as equal to equation derived in the previous paragraph following an assumption that the predictor variables are not correlated

 

 

as

 

 

The above equation can be satisfied if

 

 

and

 

 

That is, the first beta weight is equal to the correlation between the first predictor and the criterion variable and the second beta weight is equal to the correlation between the second predictor and the criterion variable.

Bivariate Regression Analysis

We can conclude from the above equations that the properties of the bivariate regression where

 

 

and, since

 

 

then

 

 

and

 

 

can be restored, if the predictor variables are not correlated.

Multiple Regression Analysis

For this special case, the coefficient of multiple determination can be computed as the sum of the squared correlations between the predictors and the criterion variable.

 

 

where k signifies the number of predictor variables with their inter-correlations all equal to zero.

Recall that the coefficient of multiple determination equals the standard variance of the predicted scores.

 

 

Thus 

 

 

The variance contributions of the predictor variables sum to the variance of the predicted scores as shown below.

 

 

Multiple Regression Analysis with Orthogonal Predictor Variables

Let us build a multiple regression model with two predictors that are not correlated. The data matrix for this example is

In standard scores, the regression model for the above data is

Unlike within the regression models for the correlated predictor variables, the variance contributions of the predictor variables sum to the variance of the predicted scores (.55 + .03=.58).

Correlational Structure of Multiple Regression Analysis

By correlating variables in the above table, one can obtain the correlational structure of the regression analysis as

Note that the coefficients of determination are computed in the above table.

The variance contribution of each predictor variable fully explain the amount of the predicted variance, .55 + .03 equals .58. The multiple regression when the predictor variables are not correlated is simple, elegant, and interpretable.

Construct Orthogonal Predictor Variables

There are three main ways to construct regression models with orthogonal predictor variables. The method of successive partialing and the method of principal components analysis make correlations between predictor variables equal to zero. The Helmert's method of orthogonal contrasts is an algorithm for generating variables that are not correlated. We will discuss each one of these methods in turn.