ML 14: Boosting Overfits Its Probabilities Long Before Its Accuracy
Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Gradient-boosted trees on the breast-cancer data, ten half splits, 2,000 rounds. With clean labels and a learning rate of 0.1 the test error falls to 0.046 and, from round 30 on, never rises....