Conformal-Prediction

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Split conformal at alpha = 0.1 on 24,576 held-out predictions split in half delivers 0.8961 coverage against a finite-sample guarantee of 0.9001, and over 200 calibration splits a mean of 0.8999. Exactly what was promised....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Over 200 random row splits of 24,576 predictions, realised conformal coverage at alpha = 0.1 has an SD of 0.0040, 1.49 times the Beta spread episode 1 compared it with. That comparison was short a term: the coverage is measured on a finite test half, and with the test half’s binomial variance added the excess is 1....