Chapter 10: Analysis of Variance: One-way Analysis of Variance
One-way ANOVA: Post Hoc Tests
Recall that when we use an ANOVA to explore the differences between three or more population means we are testing the following hypotheses:
\[\begin{array}{rcl}
H_0&:& \text{All population means are equal.}\\\\
H_a&:& \text{Not all population means are equal.}
\end{array}\]
As you can see, the alternative hypothesis only states that a difference between population means exists, but does not specify which means are supposed to be different. Consequently, if the null hypothesis of equal means is rejected, additional testing is required to determine which pairs of means are different.
If we want to investigate the differences between each pair of means separately, we unavoidably have to deal with the multiple testing problem. To prevent this problem from occurring, a specially-designed post hoc test can be run.
Post Hoc Analysis
A post hoc test is a supplementary test that is done after the null hypothesis of an ANOVA is rejected, in order to determine which population mean differences are significant and which are not.
Post hoc tests are designed to simultaneously compare all pairs of means, while at the same time controlling the family-wise error rate, i.e., the overall probability of making a Type I error when conducting multiple tests. This is done by correcting the significance level of each individual test, such that the overall Type I error rate across all comparisons remains equal to #\alpha#.
For each pair of populations (#i#, #j#), the following hypotheses are tested:
\[\begin{array}{rcl}
H_0&:& \mu_i = \mu_j \text{ (The population means are equal.)}\\\\
H_a&:&\mu_i \neq \mu_j \text{ (The population means are not equal.)}
\end{array}\]
In practice, the calculations of a post hoc test are best left to statistical software which will generally produce a summary table of the results, including the #p#-value of the tests, as well as confidence intervals for the mean differences.
If the #p#-value is less than or equal to #\alpha#, or the confidence interval of the mean difference does not include #0#, then we may conclude that the population means are different.
For any given type of ANOVA, there are a multitude of post hoc tests available. In the context of a one-way ANOVA, we suggest you choose between the following two alternatives:
- If the assumption of equal population variances (homoscedasticity) is satisfied, then Tukey-Kramer method for Honestly Significant Differences (HSD) is recommended.
- If the assumption of equal population variances is violated, then the Games-Howell post hoc test is recommended.

Based on these results, we conclude that all pairs of cities, with the exception of Paris-Berlin (#p=0.021#) and Berlin-Milan (#p=0.032#), significantly differ from each other at the #0.01# level of significance.
An example of the results of a Tukey's HSD post hoc test run in #\mathrm{R}# can be seen below.

Based on these results, we conclude that only the means of treatments C and B (#p=0.008#) significantly differ from each other at the #0.01# level of significance.
Or visit omptest.org if jou are taking an OMPT exam.