Part XV — Multiple Testing

What happens to error rates when many hypotheses are tested — the central statistical hazard of strategy research. Corrections (Bonferroni, FDR) and the trading-specific tests (White's Reality Check, Hansen's SPA) that account for a searched universe of strategies.

Topics

Topic	Focus
Multiple Comparisons	Why error rates explode when many tests are run
Bonferroni Correction	Controlling family-wise error by dividing the level
False Discovery Rate	Controlling the expected fraction of false positives
Data Snooping Bias	How repeated strategy testing manufactures false discoveries
White's Reality Check	A bootstrap test for the best strategy in a searched universe
Hansen's SPA Test	A studentized improvement on White's Reality Check