Model-Evaluation on The Standard Error

Model-Evaluation on The Standard Errorhttps://jongha-jeon-dev.github.io/standarderror/tags/model-evaluation/Recent content in Model-Evaluation on The Standard ErrorHugo -- 0.123.7en-gbTue, 13 Oct 2026 00:00:00 +0000ML 1: Your Test Score Has an Error Bar, and It Is Wider Than Your Gainhttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-1-error-bar/Thu, 08 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-1-error-bar/A test accuracy is a proportion measured on a finite sample. On 360 test digits its error bar is wider than the gap between the top two of three good classifiers, and repeating the split does not measure what it seems to.ML 2: Cross-Validation Estimates the Error of a Model You Did Not Trainhttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-2-cv-target/Thu, 08 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-2-cv-target/K-fold CV is described as estimating how well your model will do. Where that is known exactly, the estimate is uncorrelated with it. What CV estimates is how models like yours do on average.ML 45: A Test Set Answers Honestly Only the First Timehttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-45-holdout/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-45-holdout/A test set is not spent by training on it. It is spent by asking it questions whose answers change what you do next, and it can be spent without touching a single weight.ML 47: The Model With the Best AUC Was Never the Best When Misses Cost Ten Times Morehttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-47-auc-decisions/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-47-auc-decisions/AUC summarises a ranking over every threshold at once. A decision uses one threshold, and the model that ranks best on average is not always the one that ranks best there.