Multiple t tests and Type I error

·       As a simple example, you know that there is a 0.50 probability of obtaining “heads” in a coin flip. If I flip the coin four times, what is the probability of obtaining a heads one or more times across all four flips? 

·       For two coin flips, the probability of not obtaining at least one heads (i.e., getting tails both times) is 0.50 × 0.50 = 0.25.

·       The probability of one or more heads in two coin flips is 1 – 0.25 = 0.75. Three-fourths of "two coin flips" will have at least one heads.

·       So, if I flip the coin four times, the probability of one or more heads is 1 – (0.50 × 0.50 × 0.50 × 0.50) = 1 – (0.50)⁴ = 1 – 0.625 = 0.9375; you will get one or more heads in about 94% of sets of "four coin flips".

·       Similarly, for a statistical test (such as a t test) with α= 0.05, if the null hypothesis is true then the probability of not obtaining a significant result is 1 – 0.05 = 0.95.

·       Multiply 0.95 by the number of tests to calculate the probability of not obtaining one or more significant results across all tests. For two tests, the probability of not obtaining one or more significant results is 0.95 × 0.95 = 0.9025.

·       Subtract that result from 1.00 to calculate the probability of making at least one type I error with multiple tests: 1 – 0.9025 = 0.0975.

·       Example (p. 162): You are comparing 4 groups (A, B, C, D). You compare these six pairs (α= 0.05 for each): A vs B, B vs C, C vs D, A vs C, A vs D, and B vs D.

·       Using the convenient formula (see p. 162), the probability of not obtaining a significant result is 1 – (1 – 0.05)⁶ = 0.265, which means your chances of incorrectly rejecting the null hypothesis (a type I error) is about 1 in 4 instead of 1 in 20!!

·       ANOVA compares all means simultaneously and maintains the type I error probability at the designated level.