Uncertainty on The Standard Error

Uncertainty on The Standard Errorhttps://jongha-jeon-dev.github.io/standarderror/tags/uncertainty/Recent content in Uncertainty on The Standard ErrorHugo -- 0.123.7en-gbTue, 13 Oct 2026 00:00:00 +0000Coverage 1: 90% Coverage Is a Promise About Averages, Not About Youhttps://jongha-jeon-dev.github.io/standarderror/lectures/coverage-1-marginal/Fri, 11 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/coverage-1-marginal/Conformal prediction's guarantee is exact, distribution-free and finite-sample, and it is an average over the test distribution. Conditioned on the model's own confidence it runs from 82% to 95% - and then it turns out that the unevenness is the score's fault, not the method's.Coverage 2: Your Calibration Error Is Mostly Your Bin Counthttps://jongha-jeon-dev.github.io/standarderror/lectures/coverage-2-bins/Thu, 01 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/coverage-2-bins/Expected calibration error has a floor that a perfectly calibrated model also hits. On a thousand predictions most of a real model's ECE is that floor - and the fix that removes it makes the number noisier, not truer.Coverage 3: Temperature Scaling Cannot Change What You Predicthttps://jongha-jeon-dev.github.io/standarderror/lectures/coverage-3-temperature/Wed, 07 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/coverage-3-temperature/Dividing every logit by the same number cannot change a single answer. It can change which answers you decline to give - a little if you abstain on max probability, a lot if you abstain on entropy, and not at all on margin.Coverage 4: Overconfidence Arrives Before the Model Stops Improvinghttps://jongha-jeon-dev.github.io/standarderror/lectures/coverage-4-training/Thu, 08 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/coverage-4-training/The syllabus said overconfidence would arrive when validation loss stopped improving. Three training runs with checkpoints say it arrives much earlier, and only on text the model has not seen.Coverage 5: The Split You Chose Is Hiding How Variable Your Coverage Ishttps://jongha-jeon-dev.github.io/standarderror/lectures/coverage-5-split/Thu, 08 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/coverage-5-split/Episode 1 promised an error bar wider than exchangeability allows. Most of that was my formula. What is left appears only when you split the way a deployment does.ML 25: Weight Scaling Keeps Dropout's Answers, Not Its Doubtshttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-25-dropout/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-25-dropout/Switching dropout off and scaling the weights gives almost the same labels as averaging many dropout masks. It does not give the same probabilities.