Calibration on The Standard Errorhttps://jongha-jeon-dev.github.io/standarderror/tags/calibration/Recent content in Calibration on The Standard ErrorHugo -- 0.123.7en-gbTue, 13 Oct 2026 00:00:00 +0000Coverage 2: Your Calibration Error Is Mostly Your Bin Counthttps://jongha-jeon-dev.github.io/standarderror/lectures/coverage-2-bins/Thu, 01 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/coverage-2-bins/Expected calibration error has a floor that a perfectly calibrated model also hits. On a thousand predictions most of a real model's ECE is that floor - and the fix that removes it makes the number noisier, not truer.Coverage 3: Temperature Scaling Cannot Change What You Predicthttps://jongha-jeon-dev.github.io/standarderror/lectures/coverage-3-temperature/Wed, 07 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/coverage-3-temperature/Dividing every logit by the same number cannot change a single answer. It can change which answers you decline to give - a little if you abstain on max probability, a lot if you abstain on entropy, and not at all on margin.Coverage 4: Overconfidence Arrives Before the Model Stops Improvinghttps://jongha-jeon-dev.github.io/standarderror/lectures/coverage-4-training/Thu, 08 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/coverage-4-training/The syllabus said overconfidence would arrive when validation loss stopped improving. Three training runs with checkpoints say it arrives much earlier, and only on text the model has not seen.ML 5: Resampling Moves the Threshold and Breaks a Logistic Model's Probabilitieshttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-5-imbalance/Thu, 08 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-5-imbalance/'Balance the classes before training' is the first fix taught for a rare positive class. Measured, it never improved the ranking, multiplied a logistic model's probabilities tenfold, and did nothing to its decisions that a threshold does not. Boosted trees lost ranking and, with some methods, gained calibration.ML 14: Boosting Overfits Its Probabilities Long Before Its Accuracyhttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-14-boosting/Mon, 12 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-14-boosting/Whether boosting overfits depends on what you measure. Over 2,000 rounds its test error never turns up after the first few dozen; its test log loss turns after a few dozen, and below a learning rate of 0.1 shrinkage only postpones the turn.ML 34: Recalibration Fixes the Average, Given Enough Rowshttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-34-calibration/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-34-calibration/Platt scaling and isotonic regression learn a correction from held-out rows. With few rows they can make a model worse, and with many they fix the average, not each group.ML 43: Naive Bayes Counts Correlated Evidence Twice, and Only Its Probabilities Noticehttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-43-naive-bayes/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-43-naive-bayes/The independence assumption is wrong and naive Bayes still classifies well, as the textbooks say. What the assumption breaks is the one thing the model's name promises: its probabilities.ML 46: The Right Threshold Is a Cost Ratio, If the Probabilities Are Calibratedhttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-46-threshold/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-46-threshold/0.5 is the right threshold only when both mistakes cost the same. The threshold that matches the costs needs calibrated probabilities, and even the perfect model may save little.