Statistics

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Two algorithms with exactly equal expected accuracy, by symmetry, compared on 400 data sets of 300 rows. The paired t-test on ten cross-validation folds declared a difference at the 5% level in 14....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Three ways to wear out a holdout without training on it. On a leaderboard of 1,000 coin-flip labels, the majority vote of random guesses that scored above 0.5 reached 0.713 after 1,000 submissions and 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Episode 4’s simulated task with a miss costing r times a false alarm. At r = 10, flagging at 0.5 cost 1.44 to 2.00 per case, against 0.497 for flagging every case....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Five models on a simulated task, ten training sets each. A random forest had the highest AUC every time (0.754) and the lowest cost every time when both errors cost the same....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A simulated system with an unmeasured common cause, a habit it drives, a treatment that raises the outcome’s risk, and a test result produced by the outcome. Boosted trees put 0.45 of permutation importance on the test result, and raising it by one standard deviation in the model’s input raised the predicted risk by 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A simulated retention offer, a randomised trial behind it and a budget for 20% of customers. Offering it to the highest predicted churners prevented 20.1 to 22.0 churns per 1,000 customers, about what random targeting prevented (21....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Two simulated districts, ten inspections a day for a year, violations found only where inspectors go; district B’s true rate 20% above A’s. Sending inspectors where more violations had been found put every visit in one district by the year’s end in all 500 runs, the lower-rate district in 46%....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Ten small arithmetic transformers, graded on a pool of 2,500 problems and ranked on benchmarks drawn from it. The models’ 95% intervals covered their pool scores 94% to 95% of the time on average, but on 100 questions the best model ranked first in 50% of benchmarks, and seven close models came out in their true order in 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. The ten largest S&P 500 companies are around 41% of the index against roughly 19% in 1990, and the usual conclusion is that index investors are no longer diversified. There is a formula for that....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. On 31 July 2026 the KOSPI rose 17.91%, the largest one-day gain in its history, and Korean individuals net sold a record 8.2543 trillion won into it. So I priced the rule: selling every 15% drawdown cost 2....