Estimation of Statistical Significance

Euler, Gosset, and Fisher

Leonhard Euler wrote over 800 books and articles on mathematics. Among many other accomplishments, he laid the foundation for the theory of higher transcendental functions by introducing the beta and gamma transcendental functions. Student of Karl Pearson, William S. Gosset proposed the use of one of Euler's higher transcendental functions, t-distributions, in lieu of the normal distribution in 1908. Karl Pearson was less than enthusiastic about this proposal.

The t-test gradually replaced the z-test during the years following the 1925 publication of Fisher's paper Application of Student's Distribution. More than increasing precision of the probability estimates for small sample sizes, the t-test provided for eventual acceptance of Fisher's analysis of variance methods. At the same time, the use of the true variance was replaced by the use of the unbiased variance, the ns were be replaced by the degrees of freedom, and the normal distribution by an infinite number of t-distributions. This, in turn, precluded the use of a single table, associating values of the t-ratio with their corresponding probability values. Theoretically, infinite numbers of t-tables were needed.

Yes-No Dichotomies vs. Continuous Probability Continua

This problem was circumvented by introduction of the null hypothesis, and by selecting few probabilities (typically p < .05, p < .01) deemed necessary for its rejection. Even though the t-test is undoubtedly more general than the z-test, the implications of the changes within the theory of statistics, concomitant to its introduction, contributed to some intractable problems currently haunting the discipline.

The Normal and t-Distributions

Compare the normal distribution

  and a typical t-distribution.

Theoretically, the normal distribution and the t-distribution are identical only for the infinite number of the degrees of freedom. Practically, you may see for yourself that for most but minute ns, the differences between the normal and t-distributions are not substantial.

The z and t-Tests of Statistical Significance

The t-test is an analogue of the z-test where the degrees of freedom replace the n and the t-distribution replaces the normal distribution. Thus the z-test of statistical significance 

 changes into the t-test of statistical significance 

by replacing the n with the degrees of freedom equal to n - 2. n = total sample size (e.g., n = n0 + n1). 

The formula for the t-test using group means and variances is

It provides a link between the correlational methods and statistical methods estimating the probability that differences between two means are large enough to be statistically significant.

The Independent Measures Design

As an example of an independent-samples t test, the effect of caffeine on keeping students awake was evaluated. Randomly select five students who are preparing for the final examination. At midnight,  randomly give two of them a cup of decaffeinated coffee (X₀), and three of them a cup of a regular coffee (X₁). 

Independent Measures Design

Independent Variable

Dependent Variable

Data Set

The parent vector X identifies students with respect who drank which type of coffee. The dependent variable Y is number of hours each student continued to study past midnight.

Regression Analysis

The primary function of the general linear model is to predict outcomes if variables related to these outcomes are known and can be quantified. For the example, our task is to predict number of hours students continued to study past midnight (Y) from the type of coffee students drank (X). The results of a regression analysis are presented below.

 

First, combine the data values from two groups into one total group.

Without group membership information, the best prediction of the outcome is the overall mean. The overall mean of the total group is 3 and the total variance is 2 (the mean of squared deviation scores) as shown below.

The distances between the values of the dependent variable Y and the overall mean (MY) are graphed below.

 

 

For the group coded as 0, its group mean can be computed as (1 + 2) / 2 = 1.5. For the group coded as 1, its group mean can be computed as (3 + 4 + 5) / 3 = 4. The predicted values are displayed below.

The variance due to different group means (experimental treatments) is equal to 1.50. 

The distances between the predicted values (group means) and the overall mean are graphed below. 

 

The variance which can not be explained by the predictor variable is called error variance and the error variance is equal to .50.

The distances between the values of the Y variable and their respective group mean are graphed below.

 

The variance in the the dependent variable Y (2.00) can be partitioned into the predictable (1.5)  and error (.50) components as shown below.

 

Divide the variance components by the total variance of 2.00.

The standard variance of the dependent variable Y (2/2=1) is partitioned into the predictable (1.5/2=.75)  and error (.5/2=.25) components. Approximately 75% of the variance in study time was accounted for by whether a student was assigned to drink decaffeinated coffee or regular coffee. About 25 percent of the variance was due to unidentifiable factors. 

The lowercase sigma has two forms, and . Note that (also written as ) signifies standard variance.

Research Question

We have learned that the mean study time was 1.5 hours for students who drank decaffeinated coffee and 4 hours for students who drank regular coffee. Do students who drink regular coffee continue to study longer than those who drink decaffeinated coffee? 

The Independent -Samples t Test

1. The t Square Ratio

The coefficient of determination for variables X and Y is .75, the coefficient of alienation is .25, and the degrees of freedom are 5 - 2. The t-square 

 

is computed as ( .75 / .25) 3 which equals 9.

2. The t Ratio

Take the square root of the t square ratio. The t value equals 3. 

3. Probability

Locate the position of the obtained t value in the t distribution with three degrees of freedom. We find that the probability associated with a t-ratio of 3.00 or more is .0288. 

Approximately 3 percent of the time you would get a t ratio of 3.00 or more by chance. Since the probability associated with the t-ratio is less than .05, the researcher would declare the result to be statistically significant.

4. Report the Results

An independent-samples t test was conducted to evaluate the effect of caffeine on keeping students studying past midnight. Students who drank regular coffee (M = 4.00, SD = 1.00) continued to study longer than those who drank decaffeinated coffee (M = 1.50, SD = .707). The t test was significant, t(3) = 3, p < .05. About 75 percent of the variance in study time was accounted for by whether a student drank regular coffee or decaffeinated coffee.

The Repeated Measures Design

An experiment using the same subjects for all conditions of the experiment is called the repeated measures design. Using the repeated-measures design, the researcher reduces the experimental error by subtracting the variability due to subjects.

As an example of a related-samples t-test, therapy outcomes of five clients suffering from arachnophobia were evaluated by measuring their physiological reactions of fear to a picture of a spider before (Y₀) and after (Y₁) therapy.

Conduct a related-samples t test to access the effectiveness of a therapy for clients suffering from arachnophobia.

Research Question

We have learned that the mean on the pretest was 14.8 and the mean on the post test was 11.2. Does the therapy help the clients reduce their physiological reactions to a picture of a spider?

The Related-Samples t-Test (The Paired-Samples t Test)

The framework of the correlated-samples t-test, for the arachnophobia example, is

The relevant formula is

where

k = number of experimental conditions. For our example, k = 2. The degrees of freedom are n - 1. n = number of pairs of scores. 

Note that the researcher reduces the experimental error by subtracting the variability due to subjects (a).  

Notice that the predicted variable consists of group means. The variance due to different group means is equal to 3.24. Divide the predictable variance component by the total variance of 12.8. About 25 percent (3.24/12.8 = .25) of the variance in Y is accounted for by the intervention.

Sum the two scores and compute the mean for each subject. The average performance of each subject over two experimental conditions, for the example, is

 

 

The variance due to row means (individual differences) is equal to 8. Divide the variance by the total variance of the dependent variable Y. Approximately 62.5 percent (8 / 12.8 = .625) of the variance in Y is accounted for by individual differences.

Formula

Alternatively, you can compute the variance due to subjects directly by using the following formula.   

The numerator is the variance of the row sums (Y0 + Y1), which equals 32. Convert it to the variance of the row means (sum/2) by dividing 32 by (2*2). Furthermore, divide the variance of the row means by the total variance of the variable Y: (32)/(4*12.8) = .625. Approximately 63 percent of the variance in Y can be explained by individual differences. 

Note that the variability due to subjects is subtracted from the error term.

1. The t Square Ratio

The numerator of the t-square formula (the information term) is .253, which is the percentage of variance accounted for by the intervention. The denominator of the t-square formula (the error term) is 1.00 - .253 - .625, which equals .122. The degrees of freedom are n - 1. n = number of pairs of scores. Thus, the t-square ratio equals ( .253 / .122 ) (5-1) which is 8.30.

2. The t Ratio

Take the square root of the t square ratio. The t value equals 2.88. 

3. Probability

Locate the position of the calculated t value in the t distribution with 4 degrees of freedom. We find that the probability associated with a t-ratio of 2.88 or more is .0225. Only two percent of the time you would get a t ratio of 2.88 or more by chance. Since the probability associated with the t-ratio is less than .05, the researcher would declare the result to be statistically significant.

4. Report the results

A related-samples t test was conducted to evaluate the effectiveness of a therapy for clients suffering from arachnophobia. The results indicated that the posttest scores (M = 11.2, SD = 4.21 ) were significantly lower than the pretest scores (M = 14.8, SD = 2.49), t(4) = 2.88, p = .0225. The therapy helps the clients reduce their physiological reactions to a picture of a spider. About 25% of the variance is accounted for by the intervention.
     

Summary

The key formulae of the t-test are summarized in the table below. For the independent measures design, the t-square ratio equals

where the degrees of freedom are equal to (n -2). n = total sample size.

For the repeated measures design, the t-square ratio equals

where

The degrees of freedom are n - 1. n is the number of pairs of scores. 

Advantages

Most textbooks use the following formula to compute a t ratio for two independent samples 

and use the following formula to compute a t ratio for two related samples. 

   

However, the t square ratio is more parsimonious than the above traditional formulae. The t square ratio directly provides the information about the effect size (percentage of variance accounted for). Also, the knowledge of the t square ratio is readily transferred to learning the properties of the F test. 

Probability Associated with the t Ratio

The t ratio is used to test the difference between two sample means. First, compute the t ratio. Next, locate the position of the t value in the t distribution with a given degrees of freedom. The probability associated with the t value can be obtained from any statistical program. Last, compare the observed probability to the chosen significance level (e.g., .05). When the observed probability is equal to or less than the chosen significance level, we declare the result to be statistically significant.