Post hoc Comparisons

Usually, after obtaining a statistically significant F test from the ANOVA , one wants to know which means contributed to the effect; that is, which groups are particularly different from each other. One could of course perform a series of simple t-tests to compare all possible pairs of means. However, such a procedure would capitalize on chance. The reported probability levels would actually overestimate the statistical significance of mean differences. For example, suppose you took 20 samples of 10 random numbers each, and computed 20 means. Then, take the group (sample) with the highest mean and compare it with that of the lowest mean. The t-test for independent samples will test whether or not those two means are significantly different from each other, provided that they were the only two samples taken. Post-hoc comparison techniques on the other hand, specifically take into account the fact that more than two samples were taken. They are used as either hypothesis testing or exploratory methods.
For more information, see the ANOVA chapter.