Cross-Validation on The Standard Errorhttps://jongha-jeon-dev.github.io/standarderror/tags/cross-validation/Recent content in Cross-Validation on The Standard ErrorHugo -- 0.123.7en-gbTue, 13 Oct 2026 00:00:00 +0000ML 2: Cross-Validation Estimates the Error of a Model You Did Not Trainhttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-2-cv-target/Thu, 08 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-2-cv-target/K-fold CV is described as estimating how well your model will do. Where that is known exactly, the estimate is uncorrelated with it. What CV estimates is how models like yours do on average.ML 3: Not Every Leak Leaks, and the Ones That Do Can Be Measuredhttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-3-leakage/Thu, 08 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-3-leakage/'Fit every preprocessing step inside the folds' is right, and taught as if every violation were equally bad. Measured, seven of them range from nothing to a third of the accuracy scale.ML 13: Out-of-Bag Error Is Free, Pessimistic When Small and Blind to Copieshttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-13-oob/Mon, 12 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-13-oob/A random forest estimates its own test error from the rows each tree did not see. With enough trees that is as good as cross-validation, for a sixth of the work. With few trees, or rows that are not independent, it is not.ML 35: The Best Cross-Validated Score Is the Search's Best Luckhttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-35-tuning/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-35-tuning/Cross-validation estimates one model fairly. The best of many cross-validated scores is the maximum of noisy estimates, and it overstates the model it picks.ML 36: Shuffling a Time Series Is Safe Until the Errors Rememberhttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-36-time-series/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-36-time-series/The rule against shuffled folds is right for a reason that is narrower than time: something the model cannot see changing slowly. Forward splitting is not the cure it is sold as.ML 37: Rows From the Same People Are Not New Datahttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-37-subjects/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-37-subjects/When subjects contribute many rows each, row-level cross-validation measures how well the model knows the people it has met, and extra rows from the same people are worth little.ML 44: Cross-Validation Folds Are Not Independent Samples, and the t-Test Assumes They Arehttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-44-model-tests/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-44-model-tests/Two algorithms built to be exactly equal, compared 400 times. The most common test found a difference one time in seven, the most thorough-looking one more often than not, and three others were honest.