Multiple Regression Analysis

Multiple regression analysis is a method for explanation of phenomena and prediction of future events. A coefficient of correlation between variables X and Y is a quantitative index of association between these two variables. In its squared form, as a coefficient of determination, indicates the amount of variance (information) in the criterion variable Y that is accounted for by the variation in the predictor variable X. A multivariate counterpart of the coefficient of determination is the coefficient of multiple determination,. In multiple regression analysis, the set of predictor variables  is used to explain variability of the criterion variable .

Matrix of Correlations R

Initially, a matrix of correlations R is computed for all variables involved in the analysis. This matrix can be conceptualized as a supermatrix, containing submatrices , and a scalar number 1

annotated as

An intuitive approach to the multiple regression analysis is to sum the squared correlations between the predictor variables and the criterion variable to obtain an index of the over-all relationship between the predictor variables and the criterion variable. However, such a sum is often greater than one, suggesting that simple summation of the squared coefficients of correlations is not a correct procedure to employ.

In fact, a simple summation of squared coefficients of correlations between the predictor variables and the criterion variable is the correct procedure, but only in the special case when the predictor variables are not correlated. If the predictors are related, their inter-correlations must be removed that only the unique contributions of each predictor toward explanation of the criterion are included.

Fundamental Equation of Multiple Regression Analysis

The fundamental equation of the multiple regression analysis is

The expression on the left side signifies the coefficient of multiple determination (or squared multiple correlation coefficient). The expressions on the right side are the transposed matrix of cross-correlations (Rxy'=Ryx), the matrix of inter-correlations to be inverted (Rxx^-1), and the matrix of cross-correlations (Rxy).

The premultiplication of the matrix of cross-correlations by its transpose changes the coefficients of correlation into coefficients of determination. The function of the inverted matrix of the inter-correlations is to remove the redundant variance from the matrix of inter-correlations of the predictor set of variables.

Operations

The fundamental equation of regression analysis contains two distinct operations. The first operation is the postmultiplication of the transpose of cross-correlations by the inverse of inter-correlations, resulting in the matrix of beta weights B

The second operation is the premultiplication of the cross-correlations, by the beta weights, resulting in the coefficient of multiple determination

Notice that in the case of the multiple regression, submatrices of cross-correlations and beta weights are in reality vectors. As you will realize later, the convention of signifying both the matrices and the vectors by capital letters facilitates the discussion of canonical analysis of which the multiple regression is a special case.

An Example

At this point, let us consider a hypothetical example of the multiple regression analysis, consisting of two predictor variables X and a criterion variable Y. As an example you may consider scores on an aptitude test and scores on test of motivation as predictors of academic performance.

Correlations

Supermatrix R

Use a computer program to compute the correlations for all variables (X1, X2, and Y) involved in the analysis. The resulting correlation matrix will look like this.

 

Partition the Supermatrix

The matrix contains submatrices , , , and a scalar number 1.00. 

1. Correlation Matrix Rxx

First, examine the inter-correlation between the two predictors. 

Notice that a variable correlates perfectly with itself. 

R_x1x1 = 1.00 and  R_x2x2 = 1.00

The correlation between the two predictor variables is .30.

R_x1x2 = .30  and  R_x2x1 = .30

We may label the above correlation matrix as Rxx.

2. Correlation Column Vector Rxy

Next, examine the cross-correlation between each predictor variable and the criterion variable.

We may label the above correlation vector as Rxy.

3. Correlation Row Vector Ryx

Third, examine the cross-correlation between the criterion variable and each predictor variable.

We may label the above correlation vector as Ryx.

4. Correlation Scalar Ryy

The criterion variable correlates perfectly with itself. Ryy = 1

Compute Beta Weights and Coefficient of Multiple Determination

Inverse of Inter-Correlations

The first question is whether the matrix of inter-correlations is invertible. Its determinant, computed as (1)(1)-(.30)(.30), equals .91; the matrix is not singular and can be inverted by changing signs of the off-diagonal elements and dividing by the determinant

 

 

as

 

Beta Weights

The standard regression coefficients, beta weights, are computed as

 

 

with resulting vector of beta weights

 

Coefficient of Multiple Determination

The coefficient of the multiple determination is computed as

 

 

This operation weights the cross-correlations by the beta weights

 

 

The coefficient of multiple determination equals .16 + .42 which equals .58. The coefficient of multiple correlation R is obtained by taking a square root of the coefficient of multiple determination and equals .76.

For our hypothetical experiment we can conclude that if the reliability of our measurements would be perfect, 58 percent of variability in the academic achievement could be explained by the students' scores on the test of aptitude and the test of motivation.

Computer Outputs

Similar results can be obtained from a computer program for multiple regression with output summarized as 

Multiple Regression Model

Standard Scores

The multiple regression model, expressed in standard scores, pertains to partitioning of variance of the criterion variable into its predictable and error components

 

 

where the scores of the predictable component equal

 

 

and the scores of the residual component equal

 

 

The multivariate regression model is an extension of the bivariate regression model where

 

 

and

 

 

Deviation Scores

The bivariate regression model can be expressed in deviation scores by substituting right hand sides of equations for translation of standard scores into deviation scores,

 

 

and

 

 

 into the equation which defines the predictable component

 

 

and, solving for the predicted scores

 

 

The geometric equation of a line, expressed in deviation scores is

 

 

Substituting the equation translating beta weights into b weights into the last but one equation

 

 

results in the statistical equation expressing the bivariate regression model in the deviation scores

 

 

The multivariate regression model, expressed in deviation scores is

 

 

where the regression weights equal

 

Obtained Scores

To express the bivariate regression model in the obtained scores, substitute right hand sides of the equations

 

 

and

 

 

into the equation expressing the bivariate regression model in deviation scores

 

 

and solve for the predicted scores in the obtained scores form

 

 

The geometric equation of a line, expressed in obtained scores is

 

 

Since the regression weights if the models expressed in the obtained or in the deviation scores are the same, i.e.,

the intercept, A, equals

 

 

The predictable component of the multivariate regression model expressed in the obtained scores is

 

 

where the intercept equals

 

 

The residual component of the regression model expressed in the obtained scores equals

 

 

For the example, the first regression coefficient equals .32 and the second regression coefficient equals .60. In this particular example, the beta and the b weights are the same, since the predictor variables and the criterion variable have the same variances. The intercept can be computed as A = 3.00 - .32(3.00) - .60(3.00) which equals .23.

 

An Example

Regression analysis can be expressed in three basic modes, using standard, deviation, and obtained scores. The most concise mode is that using standard scores. 

Regression Equation in Standard Scores

 

Using beta weights from the above table, regression equations for the predicted and error scores can be written, using standard scores, as

 

 

and the equation expressing the residual component as

 

 

Using the above equations, the multiple regression can be built as

 

 

Means

The means of the predicted and error scores sum to the mean of the criterion variable

 

 

 

Since the variables are in the standard form, all the means are zero.

Variances

The variances of the predicted and the error scores sum to the variance of the criterion variable.

 

 

Since the criterion variable is in the standard form

 

 

the variance of the predicted scores equals the coefficient of multiple determination

 

 

and the variance of the error scores equals the coefficient of multiple alienation

 

 

The variances components contributed by the predictor variables (.10 + .36 = .46) do not sum to the variance of the predicted scores (.58), since the predictor variables are correlated.

 

Regression Equation in Deviation Scores  

Using b weights from the above table,  regression equations in deviation scores for the predicted and residual components of the multiple regression model can be written as

 

and

 

Using the above equations, the multiple regression can be built as

 

 

Means

The means of the predicted and error scores sum to the mean of the criterion variable

 

  Since the variables are in the form of deviation scores, all the means are zero.

Variances

The specification equation for partitioning of variance by the multiple regression analysis states that the variances of the predicted and the error scores sum to the variance of the criterion variable

 

 

Dividing its both sides by the variance of the criterion variable can standardize the above equation as

 

 

This equation can be also expressed as

 

 

The coefficient of multiple determination equals

 

 

and the coefficient of multiple alienation equals

 

 

The variances contributed by the predictors (.20+.73 = .93) do not sum to the variance of the predicted scores (1.16), since the predictor variables are correlated.

Regression Equation in Obtained Scores

Using the b weights and intercept from the above table, the regression equation in the obtained scores can be written as

 

 

The equation, expressing the residual component in the obtained scores, is

 

 

Using the above equations, the multiple regression can be built as

 

 

Means

The means of the predicted and error scores sum to the mean of the criterion variable

 

 

Since the mean of the error component must be zero, the mean of the predicted scores must equal the mean of the criterion variable

 

 

Variances

The variances of the predicted and the error scores sum to the variance of the criterion variable, thus defining the specification equation for partitioning of variance by the multiple regression analysis as

 

 

Dividing both sides of the above equation by the variance of the criterion variable can standardize the above equation as

 

 

The above equation may be also written as

 

   

The coefficient of multiple determination equals

 

 

and the coefficient of multiple alienation equals

 

 

Note that the variances of the components of the predictor scores (.20 + .73 = .93) do not sum to the variance of the predicted variable (1.16), as the predictor variables are correlated and their weighted composites contain the covariance terms.

 

Multiple Regression Analysis of Incarceration Rates

The number of people a society puts into prisons varies greatly and provides an insight into a character of a society. Incarceration rates for the early 1990s for five societies are shown below

The incarceration rates are shown per 100,000 population; the countries were selected from a larger study. The United States, incarcerating more people than any other nation, contrasts with Japan, which is among countries where the crime rates are small. The incarceration rates of the United States and these of Japan differ by a factor of 10.

The major predictors of the crime rates are the variables capturing the numbers of broken families and the degree to which assets of a society are distributed unequally. In this particular study, the first predictor variable is the number of divorces per 10,000 population. The second predictor variable is the ratio, contrasting the percentage of the GNP going to the richest and poorest segments of the population. The criterion variable is the incarceration rate per 100,000 population.

The data matrix for this example is

The correlation supermatrix is

Next, predict Y from X1 and X2. Results of a multiple regression analysis are shown below

The multiple regression equation in obtained scores is

The predicted and error variables can then be computed and summarized as shown below. 

Examine the standard variance components. About 88 percent of variance in international comparisons of incarceration rates can be accounted for by the disintegration of families and by the extremely unequal distribution of wealth. Note that the original study from which a small sample of countries was selected for instructional purposes provides a more realistic estimate of the variance accounted for. A conservative estimate is that more than 50 percent of variance in international comparisons of incarceration rates can be accounted for by the disintegration of families and by the extremely unequal distribution of wealth. 

Take the square root of .88. The sample multiple R is equal to .938. Is the sample multiple R significantly different from zero at the .05 level? The F test can answer the above question.

Test for Multiple R 

Is the multiple R different from zero? (Is the linear relationship between a set of predictors and the criterion variable significant?)

F Ratio

Express the F ratio in terms of the proportions of variance accounted for and not accounted for.

 

 

where k is the number of predictor variables and N is the sample size.

Compute the F ratio

First, compute the F ratio.

 

Probability

Next, locate the position of the obtained F value in the F distribution with 2 degree of freedom associated with the numerator and 2 degrees of freedom associated with the denominator. The probability associated with a F-ratio of 7.33 or larger is .12 as shown below.

Compare the obtained probability (.12) to the chosen significance level (.05). The observed probability is larger than .05. The finding is not statistically significant.

Report the Results

A multiple regression analysis was conducted to evaluate how well the family disintegration and the unequal distribution of wealth predicted the incarceration rate. The two predictors were the number of divorces per 10,000 population and the ratio, contrasting the percentage of the GNP going to the richest and poorest segments of the population. The criterion variable was the incarceration rate per 100,000 population. The sample multiple correlation coefficient was .938. The linear relationship between the set of predictors and the criterion variable was not significant, F(2,2) = 7.33, p = .12.  Approximately 88 percent of variance in international comparisons of incarceration rates can be accounted for by the disintegration of families and by the extremely unequal distribution of wealth.