https://jongha-jeon-dev.github.io/standarderror/tags/machine-learning/Recent content in Machine-Learning on The Standard ErrorHugo -- 0.123.7en-gbWed, 14 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-1-error-bar/Thu, 08 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-1-error-bar/A test accuracy is a proportion measured on a finite sample. On 360 test digits its error bar is wider than the gap between the top two of three good classifiers, and repeating the split does not measure what it seems to.https://jongha-jeon-dev.github.io/standarderror/lectures/school-maths-1-shortcut/Fri, 18 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/school-maths-1-shortcut/One of them adds. The other reads the answer off the line above and reverses it, including when that line is wrong. Their held-out losses differ by less than a tenth of a nat, and the leaky one's is *lower*.https://jongha-jeon-dev.github.io/standarderror/lectures/coverage-1-marginal/Fri, 11 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/coverage-1-marginal/Conformal prediction's guarantee is exact, distribution-free and finite-sample, and it is an average over the test distribution. Conditioned on the model's own confidence it runs from 82% to 95% - and then it turns out that the unevenness is the score's fault, not the method's.https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-1-kinks/Thu, 10 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-1-kinks/Autodiff returns a float at every point where the derivative does not exist, and the float is a choice. The choices disagree inside one library, which makes differentiation a function of the expression rather than of the function - and then almost none of it turns out to happen in a real transformer.https://jongha-jeon-dev.github.io/standarderror/lectures/topology-1-h0-single-linkage/Wed, 09 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/topology-1-h0-single-linkage/The 0-dimensional persistent homology of a Rips filtration returns the same floats as scipy's single-linkage dendrogram — so the barcode is thirty lines of union-find, and it inherits chaining: three points strung between two clusters take the two-cluster signal from 6.70 to 1.00.https://jongha-jeon-dev.github.io/standarderror/lectures/numerical-analysis-1-step-size/Thu, 03 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/numerical-analysis-1-step-size/Gradient descent is forward Euler on the gradient flow, so the learning rate is a discretisation step with a stability limit — exactly 2/λ_max on a quadratic, and on a network 2.04 times larger than that formula says, because the curvature moves to meet whatever step you chose.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-2-cv-target/Thu, 08 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-2-cv-target/K-fold CV is described as estimating how well your model will do. Where that is known exactly, the estimate is uncorrelated with it. What CV estimates is how models like yours do on average.https://jongha-jeon-dev.github.io/standarderror/lectures/coverage-2-bins/Thu, 01 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/coverage-2-bins/Expected calibration error has a floor that a perfectly calibrated model also hits. On a thousand predictions most of a real model's ECE is that floor - and the fix that removes it makes the number noisier, not truer.https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-2-saturation/Thu, 10 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-2-saturation/The softmax Jacobian's norm is fixed within a factor of two by its largest probability alone, no matter how many positions it has. So one head of this model has committed hard enough that the gradient which could change its mind is gone - and it is the head the model cannot do without.https://jongha-jeon-dev.github.io/standarderror/lectures/topology-2-barcode-numbers/Wed, 09 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/topology-2-barcode-numbers/This episode was written to say that a barcode loses its dynamic range in high dimensions and the display therefore fails. The range does collapse, by a factor of 62 — and the summary becomes more informative, because concentration tightens the null far faster than it shrinks the signal.https://jongha-jeon-dev.github.io/standarderror/lectures/numerical-analysis-2-gradient-check/Thu, 03 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/numerical-analysis-2-gradient-check/The error of a finite difference is a U in the step size, so a smaller step is not a safer one — and the step decides what your gradient check can see, by eight orders of magnitude in float64 and nine more once you change the precision. In bfloat16 a gradient 17% wrong passes at every step size.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-3-leakage/Thu, 08 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-3-leakage/'Fit every preprocessing step inside the folds' is right, and taught as if every violation were equally bad. Measured, seven of them range from nothing to a third of the accuracy scale.https://jongha-jeon-dev.github.io/standarderror/lectures/coverage-3-temperature/Wed, 07 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/coverage-3-temperature/Dividing every logit by the same number cannot change a single answer. It can change which answers you decline to give - a little if you abstain on max probability, a lot if you abstain on entropy, and not at all on margin.https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-3-layernorm/Fri, 11 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-3-layernorm/Its Jacobian has rank exactly d minus 2, and the two missing directions are the invariances the layer was built to have. Then the residual connection restores the rank everywhere except the last norm, where it becomes an exact invariance of the entire network.https://jongha-jeon-dev.github.io/standarderror/lectures/coverage-4-training/Thu, 08 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/coverage-4-training/The syllabus said overconfidence would arrive when validation loss stopped improving. Three training runs with checkpoints say it arrives much earlier, and only on text the model has not seen.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-4-learning-curves/Thu, 08 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-4-learning-curves/Fit a power law to a pilot study's errors and it tells you what more data will buy. Checked against what more data did buy, it was optimistic every time, sometimes past the Bayes error.https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-4-sampling/Fri, 11 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-4-sampling/A sampled token is a step function of its logits, so the derivative is zero almost everywhere and undefined elsewhere. On a decision small enough to enumerate, the three things people return instead can be compared against the gradient that does exist.https://jongha-jeon-dev.github.io/standarderror/lectures/coverage-5-split/Thu, 08 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/coverage-5-split/Episode 1 promised an error bar wider than exchangeability allows. Most of that was my formula. What is left appears only when you split the way a deployment does.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-5-imbalance/Thu, 08 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-5-imbalance/'Balance the classes before training' is the first fix taught for a rare positive class. Measured, it never improved the ranking, multiplied a logistic model's probabilities tenfold, and did nothing to its decisions that a threshold does not. Boosted trees lost ranking and, with some methods, gained calibration.https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-5-embeddings/Fri, 11 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-5-embeddings/The embedding table is 65 near-orthogonal points on a sphere, so there is no local neighbourhood and a descent step leaves you nearest to the token you started from. The same gradient, asked to rank five candidates instead of to point, recovers almost all of the available gain.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-6-separable/Thu, 08 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-6-separable/Logistic regression is taught as finding the maximum-likelihood coefficients. When a hyperplane separates the classes there is no maximum, and what comes back is decided by when the optimiser stopped.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-7-units/Thu, 08 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-7-units/A penalty strength is taught as a property of the model. A lasso penalty is a price per unit of coefficient, so a feature's units decide what it pays - and whether it stays.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-8-selection/Thu, 08 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-8-selection/The lasso is used to find the features that matter. Tuned by cross-validation it keeps pure noise, cannot tell a feature from a correlated stand-in, and on real data keeps features whose sign it cannot decide.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-9-bias-variance/Thu, 08 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-9-bias-variance/Test error is said to be U-shaped in model complexity. Computed exactly, it is for the number of neighbours and is not for the number of trees: one knob trades bias for variance, the other only averages.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-10-double-descent/Thu, 08 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-10-double-descent/Past the point where a model fits its training data exactly, test error falls again. Measured on random features, the fall is real, never reaches the best small model, and disappears as a reason once the fit is penalised.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-11-tree-stability/Mon, 12 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-11-tree-stability/A decision tree is valued because it can be read. Refit on bootstrap resamples of the same data, its accuracy holds still and almost everything a reader would read off it moves.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-12-impurity/Mon, 12 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-12-impurity/A forest's default feature importance is computed on the data it was fitted to. On a design where the answer is known, it never ranks the only useful feature first, and ranks it last in 19 of 20 datasets.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-13-oob/Mon, 12 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-13-oob/A random forest estimates its own test error from the rows each tree did not see. With enough trees that is as good as cross-validation, for a sixth of the work. With few trees, or rows that are not independent, it is not.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-14-boosting/Mon, 12 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-14-boosting/Whether boosting overfits depends on what you measure. Over 2,000 rounds its test error never turns up after the first few dozen; its test log loss turns after a few dozen, and below a learning rate of 0.1 shrinkage only postpones the turn.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-15-extrapolation/Mon, 12 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-15-extrapolation/Tree ensembles are said to approximate any function. Inside the training range they do; outside it their predictions go flat, below the largest label they saw, and the best single model becomes the worst.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-16-permutation/Mon, 12 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-16-permutation/Shuffle a column and see how much worse the model does: it is the importance episode 12 recommended. It answers a narrower question than it seems to, on rows the shuffle invented.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-17-pdp/Mon, 12 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-17-pdp/A partial dependence plot is read as a feature's effect. With correlated inputs it is an average over combinations that never occur, and on a simulation with a known effect a forest's shows about a third of it.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-18-shapley/Mon, 12 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-18-shapley/Shapley values split a prediction among its features fairly and uniquely — relative to a reference that the theory leaves to you. Computed exactly, the choice changes the story for most predictions.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-19-shortcut/Mon, 12 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-19-shortcut/The model with the better test score is usually taken to have learned the task better. Plant one telltale pixel in the digits and the better score belongs to the model that learned the pixel.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-20-rashomon/Mon, 12 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-20-rashomon/Two models with the same test accuracy are usually treated as interchangeable. On a hard task, networks that differ only in their seed give different answers on a fifth of the rows.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-21-init-scale/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-21-init-scale/Equal initial weights never become different, so they have to be random. In a deep network, random at the wrong scale fails just as completely.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-22-lr-batch/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-22-lr-batch/With plain SGD and a fixed number of examples, multiplying the batch size and the learning rate by the same factor gives the same training run, until the learning rate reaches the point where training fails.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-23-adam-sgd/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-23-adam-sgd/Tuned on validation data and run over twenty splits, Adam and SGD with momentum reach the same test accuracy. Adam's speed is real but small, and belongs to its learning rate.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-24-batchnorm/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-24-batchnorm/In training mode, a normalised network's answer for one digit depends on which other digits share its batch. That is fine in large, mixed batches and breaks in small or uniform ones.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-25-dropout/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-25-dropout/Switching dropout off and scaling the weights gives almost the same labels as averaging many dropout masks. It does not give the same probabilities.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-26-random-labels/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-26-random-labels/A network that reaches 97% on test digits also fits random labels perfectly. Capacity cannot explain the first; the order in which it learns does.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-27-grokking/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-27-grokking/A network trained on part of a modular-addition table memorises it at once, scores nothing on the rest, and finds the rule thousands of steps later, while its training accuracy sits at 100% the whole time.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-28-augmentation/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-28-augmentation/Shifted copies of 150 digits made a network worse on the test set and far better on shifted digits. Six hundred more real digits did the reverse.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-29-covariate-shift/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-29-covariate-shift/When only the inputs move, the best possible accuracy need not change. Whether a model's accuracy does depends on how wrong the model was all along.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-30-scaling/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-30-scaling/Eight character transformers, one budget of data and steps. The law fitted to the five smallest predicts the next size almost exactly and the largest too optimistically.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-31-pca/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-31-pca/PCA keeps the directions along which the inputs vary most. On real data that can be a choice of units, and the direction that predicts the target can come fourth.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-32-kmeans/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-32-kmeans/k-means always returns k convex pieces of similar spread, and the silhouette score rates the pieces. On uniform noise it rates them higher than it rates the digits.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-33-dimensions/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-33-dimensions/Distances concentrate in high dimensions, for points that fill them. The digits' 64 pixels behave like 8 uniform dimensions, and nearest neighbours work on them until noise columns are added.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-34-calibration/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-34-calibration/Platt scaling and isotonic regression learn a correction from held-out rows. With few rows they can make a model worse, and with many they fix the average, not each group.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-35-tuning/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-35-tuning/Cross-validation estimates one model fairly. The best of many cross-validated scores is the maximum of noisy estimates, and it overstates the model it picks.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-36-time-series/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-36-time-series/The rule against shuffled folds is right for a reason that is narrower than time: something the model cannot see changing slowly. Forward splitting is not the cure it is sold as.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-37-subjects/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-37-subjects/When subjects contribute many rows each, row-level cross-validation measures how well the model knows the people it has met, and extra rows from the same people are worth little.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-38-missing/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-38-missing/Imputation treats a gap as a lost number. Sometimes the gap is a record of a decision, worth more than the number, and the model that uses it depends on the decision staying the same.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-39-selective-labels/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-39-selective-labels/Outcomes are recorded only for the cases someone approved. Whether a model trained on them can judge the rest depends on what the approver knew that the model does not.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-40-target-encoding/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-40-target-encoding/Fitting an encoding only on training rows protects the test set. It does not protect the training rows from their own labels, and with many rare levels that is enough to wreck the model.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-41-averaging/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-41-averaging/Averaging carries a guarantee, and it is weaker than the one usually quoted. When one member is certain about everything, averaging probabilities hands it every decision.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-42-stacking/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-42-stacking/A meta-model trusts whichever base model looks best on the predictions it is fitted to. On in-sample predictions that is the best memoriser; on honest ones it is usually just the best model.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-43-naive-bayes/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-43-naive-bayes/The independence assumption is wrong and naive Bayes still classifies well, as the textbooks say. What the assumption breaks is the one thing the model's name promises: its probabilities.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-44-model-tests/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-44-model-tests/Two algorithms built to be exactly equal, compared 400 times. The most common test found a difference one time in seven, the most thorough-looking one more often than not, and three others were honest.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-45-holdout/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-45-holdout/A test set is not spent by training on it. It is spent by asking it questions whose answers change what you do next, and it can be spent without touching a single weight.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-46-threshold/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-46-threshold/0.5 is the right threshold only when both mistakes cost the same. The threshold that matches the costs needs calibrated probabilities, and even the perfect model may save little.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-47-auc-decisions/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-47-auc-decisions/AUC summarises a ranking over every threshold at once. A decision uses one threshold, and the model that ranks best on average is not always the one that ranks best there.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-48-intervention/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-48-intervention/Importance and what-if analyses describe the model. In a system with a common cause and a downstream signal, the feature the model relied on most did nothing when changed, and the real lever ranked no higher than a feature with no effect.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-49-uplift/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-49-uplift/A model of who will churn is not a model of whom an offer will keep. Targeting the highest risk did no better than chance; targeting the predicted effect did about twice as well.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-50-feedback/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-50-feedback/When a rule decides where to look and learns only from what it finds there, more data does not correct it. It has to account for its own effort, and keep looking where it expects nothing.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-51-benchmark-sample/Wed, 14 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-51-benchmark-sample/A benchmark score's error bar is correct, and a leaderboard of such scores still puts close models in the wrong order. Comparing on the same questions, and counting questions rather than prompts, is most of the repair.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-52-contamination/Wed, 14 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-52-contamination/A model trained on a benchmark's answers scored higher on them and no higher on anything else. Its own likelihoods could not say which problems had leaked; a fresh set from the same source could say that some had.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-53-prompt-format/Wed, 14 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-53-prompt-format/The same model on the same problems scored 0.735 or 0.500 depending on what came before them, and the gap between two models shrank from 0.127 to about zero. A score is the model under one way of asking.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-54-multiple-choice/Wed, 14 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-54-multiple-choice/Ranking answer options by likelihood is a different test from asking for the answer. A model that never produced a four-digit sum picked it out of four options 95.5% of the time, or 32%, depending on the options.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-55-judge/Wed, 14 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-55-judge/Three models graded each other's arithmetic by how right each answer looked to them. All three preferred their own answers on every task, and forty real labels were enough to correct them.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-56-tsne/Wed, 14 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-56-tsne/A cluster ten times wider than another came out the same size, and a gap ten times wider came out about the same width. What survives the picture is who sits next to whom.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-57-bic/Wed, 14 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-57-bic/Given one cluster shaped like a square, the BIC asked for more Gaussian components the more data it saw, 18 at 20,000 points. It was answering a question about shape and sample size, not about clusters.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-58-dbscan/Wed, 14 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-58-dbscan/DBSCAN found two interlocking half-moons that k-means could not, at the right radius. When clusters differ in density, no radius is right for all of them, and the usual way of choosing it missed even the moons.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-59-anomaly/Wed, 14 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-59-anomaly/An isolation forest ranked planted anomalies almost perfectly and then flagged whatever share it was told to. Grouped anomalies and irrelevant features changed the ranking itself.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-60-stability/Wed, 14 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-60-stability/Resampling found k-means' two-way split of a single Gaussian as reproducible as three real clusters. Stability measures whether the method has one best answer, and cannot say that there are no clusters.https://jongha-jeon-dev.github.io/standarderror/posts/two-thirds-of-the-move-before-the-headline/Wed, 19 Aug 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/posts/two-thirds-of-the-move-before-the-headline/A paper measured 'buy the rumour, sell the news' on 4.57 million articles, and the sayings mostly holdhttps://jongha-jeon-dev.github.io/standarderror/posts/the-chip-cycle-is-forecastable/Tue, 18 Aug 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/posts/the-chip-cycle-is-forecastable/A delayed feedback loop, an echo state network, and a result I had the wrong way roundhttps://jongha-jeon-dev.github.io/standarderror/posts/i-trained-2000-models-on-a-coin-flip/Fri, 07 Aug 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/posts/i-trained-2000-models-on-a-coin-flip/A 55.3% hit rate, a p-value of 0.0008, and not one shred of signal in the datahttps://jongha-jeon-dev.github.io/standarderror/posts/your-90-percent-interval-covers-42-percent/Fri, 07 Aug 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/posts/your-90-percent-interval-covers-42-percent/Conformal prediction's guarantee is real, and it does not say what people repeat about ithttps://jongha-jeon-dev.github.io/standarderror/posts/why-my-integrator-exploded-at-355/Thu, 06 Aug 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/posts/why-my-integrator-exploded-at-355/Two silent numerical bugs, and the kind of test that actually catches themhttps://jongha-jeon-dev.github.io/standarderror/posts/your-forecast-is-probably-just-repeating-yesterday/Thu, 06 Aug 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/posts/your-forecast-is-probably-just-repeating-yesterday/A model scores 99% on a series of coin flips. Here is why, and the one number that would have caught it