Why can’t we just use multiple one-sample or two-sample t-tests for analyses that involve more than two groups?
The main reason we can’t use multiple two-sample t-tests to evaluate whether the means of more than two samples differ is because we increase our chances of getting false positives (see interleaf 8). When we run one test, the chance of getting a false positive is your alpha level, which is typically 5% (0.05). The chance that you don’t make that error is therefore 95%. Now run two tests. What is the probability that you do not get a false positive? It’s the probability that neither the first nor the second test resulted in a false positive. So that’s 0.95 x 0.95 = 0.9025 (think back to chapter 5 on probabilities). That means your probability of making a type I error increases!
We can control for this by using a Multiple Comparison Procedure like the Tukey-Kramer test.
There's another reason why we can't do one-sample tests specifically. Check out section 12.5. There's a really good example here how indirect comparisons (where we test whether the mean of each group differs from the hypothesized mean separately) can lead to errors with interpreting the results. I hope these two sections will help clarify things!
