Statistics

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. On a simulated task whose Bayes error is 0.268, with training sets grown to 25,600 rows, the two-parameter power law a n^-b fitted to the smaller sizes predicted too low an error in all 9 fits, and in 4 of them predicted an error below the Bayes error - 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Over 200 random row splits of 24,576 predictions, realised conformal coverage at alpha = 0.1 has an SD of 0.0040, 1.49 times the Beta spread episode 1 compared it with. That comparison was short a term: the coverage is measured on a finite test half, and with the test half’s binomial variance added the excess is 1....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. At 2% prevalence, logistic regression trained on an oversampled, SMOTE-augmented or class-weighted copy of the data ranks the test set no better than the plain model - AUC -0.0012 for oversampling, small and consistently negative - while its mean predicted probability goes from 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Every one of 50 stratified half splits of the Wisconsin breast cancer data is linearly separable on its 30 features, so the unpenalised logistic likelihood has no maximum. scikit-learn returns coefficients of norm 2....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. On the diabetes data, re-expressing one feature at a time in units from a hundredth to a hundred times the standardised ones and refitting cross-validated lasso, 7 of 10 features are kept in some units and dropped in others; only sex, bmi, bp survive every choice....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. With two real effects, one irrelevant feature correlated with a real one and seven of pure noise, cross-validated lasso keeps the real effects (c in at least 100% of datasets, a in at least 78%) and on average 2....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. On a simulated regression where bias, variance and noise can be computed exactly, the three add up to the measured test error to within 0.003. For k-nearest neighbours the variance falls from 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Least squares on p random ReLU features of a 10-dimensional input, 100 training rows, minimum-norm fit. Median test error is 0.271 at p = 20, peaks at 21.6 at p = 100, and falls again to 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Two hundred depth-3 trees fitted to bootstrap resamples of a breast-cancer training set score 0.914 on the same held-out tumours, with a standard deviation of 0.015. Their structure is another matter. The full-data tree’s root feature, worst perimeter, is the root in 50% of resamples; four other features take the rest, and all five have best splits within 5% of each other....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Five features that differ only in how many distinct values they take, and a label that is a coin flip: a random forest’s impurity importance gives the continuous feature 42% of the total and the binary one 5%....