Llm-Evaluation on The Standard Error

Llm-Evaluation on The Standard Errorhttps://jongha-jeon-dev.github.io/standarderror/tags/llm-evaluation/Recent content in Llm-Evaluation on The Standard ErrorHugo -- 0.123.7en-gbWed, 14 Oct 2026 00:00:00 +0000ML 51: A Benchmark Is a Sample of Questionshttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-51-benchmark-sample/Wed, 14 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-51-benchmark-sample/A benchmark score's error bar is correct, and a leaderboard of such scores still puts close models in the wrong order. Comparing on the same questions, and counting questions rather than prompts, is most of the repair.ML 52: A Leaked Benchmark Shows in the Set, Not in the Itemhttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-52-contamination/Wed, 14 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-52-contamination/A model trained on a benchmark's answers scored higher on them and no higher on anything else. Its own likelihoods could not say which problems had leaked; a fresh set from the same source could say that some had.ML 53: The Prompt Is Part of the Scorehttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-53-prompt-format/Wed, 14 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-53-prompt-format/The same model on the same problems scored 0.735 or 0.500 depending on what came before them, and the gap between two models shrank from 0.127 to about zero. A score is the model under one way of asking.ML 54: Multiple Choice Grades the Wrong Answers You Offeredhttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-54-multiple-choice/Wed, 14 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-54-multiple-choice/Ranking answer options by likelihood is a different test from asking for the answer. A model that never produced a four-digit sum picked it out of four options 95.5% of the time, or 32%, depending on the options.ML 55: Every Judge Ranked Itself Firsthttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-55-judge/Wed, 14 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-55-judge/Three models graded each other's arithmetic by how right each answer looked to them. All three preferred their own answers on every task, and forty real labels were enough to correct them.