Stepwise Multiple Regression

In multiple regression, we are interested in predicting a criterion variable from a set of predictors. The REGRESSION procedure provides five methods to select predictor variables. They are forward selection, backward elimination, stepwise selection, forced entry, and forced removal. To learn more choose Help / Topics from the SPSS main menus. Click the Index tag and type stepwise selection in the textbox. The index `stepwise selection in Linear Regression` will be highlighted. Next, click the Display button.

• Forward selection begins with no predictors in the regression equation. The predictor variable that has the highest correlation with the criterion variable is entered into the equation first. The rest variables are entered into the equation depending on the contribution of each predictor.
  
   
• Backward elimination begins with all predictor variables in the regression equation and sequentially removes them. Two removal criteria are available.
  
   
• Stepwise selection is a combination of forward and backward procedures.
  
  
  Step 1
  
  The first predictor variable is selected in the same way as in forward selection. If the probability associated with the test of significance is less than or equal to the default .05, the predictor variable with the largest correlation with the criterion variable enters the equation first.
  
  
  Step 2
  
  The second variable is selected based on the highest partial correlation. If it can pass the entry requirement (PIN=.05), it also enters the equation. 
  
  Step 3
  
  From this point, stepwise selection differs from forward selection: the variables already in the equation are examined for removal according to the removal criterion (POUT=.10) as in backward elimination.
  
  Step 4
  
  Variables not in the equation are examined for entry. Variable selection ends when no more variables meet entry and removal criteria.
  
   
• Multiple Regression, Hierarchical Multiple Regression, and Stepwise Multiple Regression
  
  In a (standard) multiple regression analysis, the researcher decides how many predictors to enter and all the predictors enter the regression model simultaneously.

In a hierarchical multiple regression, the researcher decides not only how many predictors to enter but also the order in which they enter. Usually, the order of entry is based on logical or theoretical considerations.

In a stepwise multiple regression analysis, the number of predictors to be selected and the order of entry are both decided by statistical criteria (e.g., entry or removal criterion).  
 

A corporation is interested in predicting job satisfaction among its employees. Ten employees were randomly chosen to fill out the information on the criterion variable (job satisfaction) and the predictor variables such as salary, job security, ratings of work environment, years of service, etc. Apply the stepwise method to select the best set of predictor variables into the regression equation.


SPSS for Windows

A. Enter data: One criterion and three predictor variables.

B. From the menus choose: Analyze \ Regression \ Linear.

First, select Y as the Dependent variable. Second, move the three predictor variables to the Independent(s) list. Third, click on the down arrow and select the Stepwise Method.

Last, click on Statistics. Choose R squared change, Descriptives, Part and partial correlations. Click Continue. Click OK.
 

SPSS Printout

A. Examine the correlation matrix.

Ideally, if each of the predictors is significantly correlated with the criterion variable and the predictors are not related to each other, a high multiple R will be obtained.

• Examine the cross correlations.
  
  Which predictor variable has the largest correlation with the criterion variable Y?
  
  Answer:
  
  X1 has the largest correlation with the criterion variable Y, r = .858, p = .001.
  
   
• Examine the inter-correlations.
  
  Large correlations between the predictor variables can substantially affect the results of multiple regression analysis. Note that the correlation between X2 and X3 equals .804.
   

B. Stepwise Selection Method

Model 1: Regression Equation with One Predictor Variable

X1 has the largest correlation with the criterion variable, p < .05. The predictor variable X1 is the first predictor to be entered into the regression equation.

• Model Summary
  
  Examine the R Square.

What is the proportion of the variation in the criterion variable Y explained by the regression model with one predictor X1? 

Answer:

About 74 % of the variation in the criterion variable Y can be explained by the regression model with one predictor X1. 

• Coefficients.
  
  Examine the B and Beta Weights.

Develop a regression equation which contains the predictor X1.

1. Regression equation in obtained scores: ________________________
 

Answer:

Regression equation in obtained scores: Y' = .92X1 + .975
 

2. Regression equation in standard scores: ________________________
 

Answer:

Regression equation in standard scores: Zy' = .858ZX1
 

• Test the regression coefficient of the entered variable (X1).
  
  Is the regression coefficient associated with X1 significantly different from zero? What is the t ratio and the observed significance level? Is it significant at the .05 level?
  
  Answer:
  
  The t value is 4.729. The observed significance level associated with X1 is .001. The regression coefficient associated with X1 is significantly different from zero.
   

Examine the Excluded Variables

Examine the absolute values of the partial correlations for variables not in the equation.

 

Model 1:  This is the first step of the stepwise regression in which only one predictor, X1, is used to predict Y. Recall that X1 has the largest correlation with the criterion variable.

The two  predictor variables,  X2 and X3, are excluded from model 1.

Beta In: These are the standardized regression coefficients for each predictor, should they be added to the regression equation. The beta value associated with X2 is larger (.535). It indicates the predictor X2 would make the greater contribution of the two excluded predictors.

Partial Correlation:

The partial correlation between X2 and Y is .849 after the effect of X1 was removed from both X2 and Y. The observed significance level associated with X2 is .004, which passes the entry requirement (p < .05). 

The partial correlation between X3 and Y is .452 after the effect of X1 was removed from both X3 and Y. The observed significance level associated with X3 is .222, which does not pass the entry requirement (p > .05).

Decision

The predictor variable X2 has the largest partial correlation. The observed significance level associated with X2 is .004, which passes the entry requirement (p < .05). The second predictor variable to be entered into the equation will be X2.

 

Model 2: Regression Equation with Two Predictor Variables

• Examine the R square.
  
  
  What is the proportion of the variation in the criterion variable Y explained by the regression model with two predictors X1 and X2? 

About 93% of the variation in the criterion variable Y can be explained by the regression model with two predictors,  X1 and X2.

The adjusted corrected for the number of predictors equals .905. The difference between the obtained and adjusted R square is small in our case.

R square may be overestimated when the data sets have few cases (n) relative to number of predictors (k).  Adjusted R square can be computed as

n = sample size and k = number of predictors

Data sets with few cases relative to number of predictors will have a greater difference between the obtained and adjusted R square.

 

• Test of Significance
  
  Test of  
   

Is the regression model with two predictors (X1 and X2)  significantly related to the criterion variable Y? 

 

The regression model with two predictors (X1 and X2) is significantly related to the criterion variable Y, F(2,7) = 44,073, p < .01.

X1 and X2 account for about 93% of the variance in the criterion variable Y and that this finding is statistically significant. 
 

• Examine the R Square Change.

(1) About 74 % of the variation in the criterion variable Y can be explained by the regression model with one predictor X1.

(2) About 93% of the variation in the criterion variable Y can be explained by the regression model with two predictors,  X1 and X2.

(3) An additional 19% of the variance in the criterion variable Y is contributed by X2. 

 

• Coefficients

Examine the B and Beta Weights.

Develop a regression equation that contains the above two predictors.

1. Regression equation in obtained scores: ________________________
 

Regression equation in obtained scores: Y' = .587X1 + .506X2 + .112

Note that the value of B weight associated with each predictor is influenced by all other predictors in the regression equation.   
 

2. Regression equation in standard scores: ________________________
 

Regression equation in standard scores: Zy' = .548ZX1 + .535ZX2 

 

3. Part Correlation

The predictor X1 entered the regression equation first and the predictor X2 entered the regression equation next. Recall that an additional 19% of the variance in the criterion variable Y is contributed by X2. The signed square root of the R square change is called the semi-partial correlation or the part correlation. The semi-partial or part correlation between X2 and Y after removing the effect of X1 from X2 is .436.   
 

• Test the regression coefficients of the entered variables. Both are significantly different from zero.
  
   

Examine the Variables Already in the Equation for Removal

No variables meet the removal criterion.

Examine the Statistics for the Variable not in the Equation for Entry

This is the second step of the stepwise regression in which two predictors, X1 and X2, are used to predict Y and the predictor variable,  X3, is excluded from model 2. Note that the observed significance level associated with X3 is .158, which is too large for entry (p > .05).

Decision: X3 will not be included. The best regression equation will be the equation that contains two predictor variables, X1 and X2.  

Reason: Since predictor variables X2 and X3 are highly correlated (r = .804), X3 adds relatively little in prediction when X2 is in the regression equation.

 Reading

• Multiple Regression by Palgrave Maxmillan

Discussion

• What are some of the problems with stepwise regression?