Cross-Validation

ML 2: Cross-Validation Estimates the Error of a Model You Did Not Train

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. On 2,000 simulated least-squares datasets of 100 rows, where the exact test error of each fitted model is known, the correlation between the 10-fold CV estimate and that error is -0.01. The CV estimates spread 2....

ML 3: Not Every Leak Leaks, and the Ones That Do Can Be Measured

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Fit a step on all the rows, then cross-validate, and compare with the same step fitted inside the folds. Standardising leaks +0.05 points on the breast cancer data and mean imputation -0....

ML 13: Out-of-Bag Error Is Free, Pessimistic When Small and Blind to Copies

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Out-of-bag error against five-fold cross-validation and a held-out half. With 500 trees it lands within 0.004 of the held-out error on breast cancer and 0.002 on digits, about as close as cross-validation, from a single fit....

ML 35: The Best Cross-Validated Score Is the Search's Best Luck

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Grid searches of an RBF support vector machine on 200 rows of a simulated task, twenty times per grid size. Untuned, the cross-validated score was 0.013 below the model’s accuracy on 20,000 fresh rows....

ML 36: Shuffling a Time Series Is Safe Until the Errors Remember

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Seven simulated time-series settings, twenty series each, cross-validated three ways on 1,000 steps and compared with the error on the next 250. On a stationary autoregression with its lags as features, shuffled five-fold cross-validation was within 3% of the future error for linear regression, nearest neighbours and a forest....

ML 37: Rows From the Same People Are Not New Data

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. 2,000 simulated rows from 2,000, 400, 100, 40 or 20 subjects, each subject with a feature signature and an outcome propensity that do not carry over. With 2,000 subjects, row-level and subject-level cross-validation matched accuracy on new people to within a point....

ML 44: Cross-Validation Folds Are Not Independent Samples, and the t-Test Assumes They Are

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Two algorithms with exactly equal expected accuracy, by symmetry, compared on 400 data sets of 300 rows. The paired t-test on ten cross-validation folds declared a difference at the 5% level in 14....