Statistics

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Out-of-bag error against five-fold cross-validation and a held-out half. With 500 trees it lands within 0.004 of the held-out error on breast cancer and 0.002 on digits, about as close as cross-validation, from a single fit....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Gradient-boosted trees on the breast-cancer data, ten half splits, 2,000 rounds. With clean labels and a learning rate of 0.1 the test error falls to 0.046 and, from round 30 on, never rises....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Trained on x between 0 and 1 with y rising as 2x, a random forest has test error 0.12 inside the range, against 0.51 for linear regression, and 2.71 between 1.5 and 2 — 22 times worse — because past x = 1 it never predicts above 2....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A random forest on the diabetes data. Shuffling bmi costs 0.168 of test R²; refitting the forest without bmi costs 0.043, 4 times less. Add an exact copy of bmi and the permutation importances are 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. With y = x1 + 2 x2² and x2 a noisy copy of x1, the true effect of x1 is a straight line of slope 1. With independent inputs, the partial dependence plot of a random forest has slope 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Exact Shapley values, by enumerating all 32 coalitions, for a five-feature gradient-boosted model of the breast-cancer data. Against all training tumours as background, they add up to each prediction minus the mean to within 7e-15....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A marker pixel, blank or nearly so in every real digit image, is lit on 90% of the 3s, 5s, 8s and 9s in both training and test data. A random forest trained on it scores 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. On a simulated task where the best possible rule errs on 27.6% of the 5,000 test rows, five neural networks that differ only in their random seed score between 0.670 and 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. In a simulation where the test inputs move up to two standard deviations while P(y | x) stays fixed, the Bayes accuracy stays between 0.832 and 0.839. A flexible spline logistic model stays within 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. On raw breast-cancer measurements the first principal component holds 98.2% of the variance and is led by worst area, the feature with the largest units; the 95% variance rule keeps 1 component, against 10 on standardised features....