ML 19: A Planted Shortcut Wins the Test Set and Loses Everywhere Else
Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A marker pixel, blank or nearly so in every real digit image, is lit on 90% of the 3s, 5s, 8s and 9s in both training and test data. A random forest trained on it scores 0....