One-way ANOVA: Post Hoc Tests

Chapter 10: Analysis of Variance: One-way Analysis of Variance

One-way ANOVA: Post Hoc Tests

Recall that when we use an ANOVA to explore the differences between three or more population means we are testing the following hypotheses:
\[\begin{array}{rcl}
H_0&:& \text{All population means are equal.}\\\\
H_a&:& \text{Not all population means are equal.}
\end{array}\]

As you can see, the alternative hypothesis only states that a difference between population means exists, but does not specify which means are supposed to be different. Consequently, if the null hypothesis of equal means is rejected, additional testing is required to determine which pairs of means are different.

If we want to investigate the differences between each pair of means separately, we unavoidably have to deal with the multiple testing problem. To prevent this problem from occurring, a specially-designed post hoc test can be run.

Post Hoc Analysis
A post hoc test is a supplementary test that is done after the null hypothesis of an ANOVA is rejected, in order to determine which population mean differences are significant and which are not.

Post hoc tests are designed to simultaneously compare all pairs of means, while at the same time controlling the family-wise error rate, i.e., the overall probability of making a Type I error when conducting multiple tests. This is done by correcting the significance level of each individual test, such that the overall Type I error rate across all comparisons remains equal to #\alpha#.

For each pair of populations (#i#, #j#), the following hypotheses are tested:
\[\begin{array}{rcl}
H_0&:& \mu_i = \mu_j \text{ (The population means are equal.)}\\\\
H_a&:&\mu_i \neq \mu_j \text{ (The population means are not equal.)}
\end{array}\]

In practice, the calculations of a post hoc test are best left to statistical software which will generally produce a summary table of the results, including the #p#-value of the tests, as well as confidence intervals for the mean differences.

If the #p#-value is less than or equal to #\alpha#, or the confidence interval of the mean difference does not include #0#, then we may conclude that the population means are different.

One-way ANOVA Post Hoc Tests
For any given type of ANOVA, there are a multitude of post hoc tests available. In the context of a one-way ANOVA, we suggest you choose between the following two alternatives:

If the assumption of equal population variances (homoscedasticity) is satisfied, then Tukey-Kramer method for Honestly Significant Differences (HSD) is recommended.
If the assumption of equal population variances is violated, then the Games-Howell post hoc test is recommended.

Tukey's HSD Test Statistic and Null Distribution
The Tukey-Kramer method for Honestly Significant Differences (HSD) is based on the Studentized range distribution, which has two values for degrees of freedom, #I# and #N-I#.

Suppose #Q# has the Studentized range distribution with degrees of freedom #I# and #N-I#, and let #q_{I,N-I,\alpha}# denote the #1 - \alpha# quantile of the Studentized range distribution, i.e., the number #q_{I,N-I,\alpha}# satisfies \[P(Q \geq q_{I,N-I,\alpha}) = \alpha\]

Then for any #\alpha# in #(0,1)#, the CI for each pairwise difference #\mu_i - \mu_j# among #I# populations at overall confidence level #1 - \alpha# is: \[\bigg(\bar{Y}_i - \bar{Y}_j - q_{I,N-I,\alpha}\sqrt{\cfrac{MS_{error}}{2}\Big(\cfrac{1}{n_i} + \cfrac{1}{n_j}\Big)},\,\,\, \bar{Y}_i - \bar{Y}_j + q_{I,N-I,\alpha}\sqrt{\cfrac{MS_{error}}{2}\Big(\cfrac{1}{n_i} + \cfrac{1}{n_j}\Big)}\bigg)\]

We can test #H_0 : \mu_i - \mu_j = 0# against #H_A : \mu_i - \mu_j \neq 0# simultaneously for each pairwise difference among #I# treatment groups using the test statistic \[Q = \cfrac{\bar{Y}_i - \bar{Y}_j}{\sqrt{\cfrac{MS_{error}}{2}\Big(\cfrac{1}{n_i} + \cfrac{1}{n_j}\Big)}}\] which has the Studentized range distribution with degrees of freedom #I# and #N-I# under #H_0#.

The #p#-value for each test is computed as #P(Q \geq |q|)#, where #q# is the computed value of #Q# for that test. Given some significance level #\alpha#, for each simultaneous hypothesis test we would reject #H_0# and conclude that there is a significant difference between #\mu_i# and #\mu_j# if the P-value is no larger #\alpha#.

When the design is balanced, with #J# observations per sample, we replace #\cfrac{MS_{error}}{2}# with #\cfrac{MS_{error}}{J}# in the above formulas, and call it Tukey’s method instead.

An example of the results of a Games-Howell post hoc test run in #\mathrm{SPSS}# can be seen below.

Based on these results, we conclude that all pairs of cities, with the exception of Paris-Berlin (#p=0.021#) and Berlin-Milan (#p=0.032#), significantly differ from each other at the #0.01# level of significance.

An example of the results of a Tukey's HSD post hoc test run in #\mathrm{R}# can be seen below.

Based on these results, we conclude that only the means of treatments C and B (#p=0.008#) significantly differ from each other at the #0.01# level of significance.