I Trained 2,000 Models on a Coin Flip and the Best One Looked Great
Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. I generated a target that cannot be predicted, gave 60 pure-noise variables to 2,000 models, and kept the best one. It called 55.3% correctly, which taken at face value is significant at p = 0....