ML 35: The Best Cross-Validated Score Is the Search's Best Luck
Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Grid searches of an RBF support vector machine on 200 rows of a simulated task, twenty times per grid size. Untuned, the cross-validated score was 0.013 below the model’s accuracy on 20,000 fresh rows....