Calibration

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A model calibrated by construction - labels drawn from its own probabilities - scores an ECE of 0.085 on 200 predictions, 0.035 on 1,000 and 0.008 on 20,000 at 15 bins, and the fitted slope against bins/n is 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. On 10,240 next-character predictions from the committed model, accuracy is 0.5466 at all seven temperatures from 0.5 to 3, identical to the bit, while ECE runs from 0.026 to 0.375. Between predictions the confidence order moves: at T = 2 Kendall’s tau against T = 1 is 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Three training runs of the same 816,128-parameter model, evaluated at every checkpoint on 10,240 held-out predictions. Every run starts slightly underconfident; overconfidence begins after one to ten passes over the training text and grows as training loss pulls away from validation loss....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. At 2% prevalence, logistic regression trained on an oversampled, SMOTE-augmented or class-weighted copy of the data ranks the test set no better than the plain model - AUC -0.0012 for oversampling, small and consistently negative - while its mean predicted probability goes from 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Gradient-boosted trees on the breast-cancer data, ten half splits, 2,000 rounds. With clean labels and a learning rate of 0.1 the test error falls to 0.046 and, from round 30 on, never rises....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Boosted trees on a simulated task were overconfident (calibration error 0.157, log loss 0.748). Platt scaling on 50 calibration rows cut log loss to 0.612; isotonic regression on 50 rows raised it to 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Gaussian naive Bayes on the breast-cancer data, whose features include radius, perimeter and area of the same nuclei, scored accuracy 0.940 against logistic regression’s 0.981, with log loss 0.611 against 0.073; 67% of its errors came with confidence above 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Episode 4’s simulated task with a miss costing r times a false alarm. At r = 10, flagging at 0.5 cost 1.44 to 2.00 per case, against 0.497 for flagging every case....