Evaluation on The Standard Errorhttps://jongha-jeon-dev.github.io/standarderror/tags/evaluation/Recent content in Evaluation on The Standard ErrorHugo -- 0.123.7en-gbWed, 14 Oct 2026 00:00:00 +0000ML 19: A Planted Shortcut Wins the Test Set and Loses Everywhere Elsehttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-19-shortcut/Mon, 12 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-19-shortcut/The model with the better test score is usually taken to have learned the task better. Plant one telltale pixel in the digits and the better score belongs to the model that learned the pixel.ML 51: A Benchmark Is a Sample of Questionshttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-51-benchmark-sample/Wed, 14 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-51-benchmark-sample/A benchmark score's error bar is correct, and a leaderboard of such scores still puts close models in the wrong order. Comparing on the same questions, and counting questions rather than prompts, is most of the repair.ML 53: The Prompt Is Part of the Scorehttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-53-prompt-format/Wed, 14 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-53-prompt-format/The same model on the same problems scored 0.735 or 0.500 depending on what came before them, and the gap between two models shrank from 0.127 to about zero. A score is the model under one way of asking.ML 54: Multiple Choice Grades the Wrong Answers You Offeredhttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-54-multiple-choice/Wed, 14 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-54-multiple-choice/Ranking answer options by likelihood is a different test from asking for the answer. A model that never produced a four-digit sum picked it out of four options 95.5% of the time, or 32%, depending on the options.ML 55: Every Judge Ranked Itself Firsthttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-55-judge/Wed, 14 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-55-judge/Three models graded each other's arithmetic by how right each answer looked to them. All three preferred their own answers on every task, and forty real labels were enough to correct them.