Machine-Learning

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Every one of 50 stratified half splits of the Wisconsin breast cancer data is linearly separable on its 30 features, so the unpenalised logistic likelihood has no maximum. scikit-learn returns coefficients of norm 2....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. On the diabetes data, re-expressing one feature at a time in units from a hundredth to a hundred times the standardised ones and refitting cross-validated lasso, 7 of 10 features are kept in some units and dropped in others; only sex, bmi, bp survive every choice....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. With two real effects, one irrelevant feature correlated with a real one and seven of pure noise, cross-validated lasso keeps the real effects (c in at least 100% of datasets, a in at least 78%) and on average 2....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. On a simulated regression where bias, variance and noise can be computed exactly, the three add up to the measured test error to within 0.003. For k-nearest neighbours the variance falls from 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Least squares on p random ReLU features of a 10-dimensional input, 100 training rows, minimum-norm fit. Median test error is 0.271 at p = 20, peaks at 21.6 at p = 100, and falls again to 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Two hundred depth-3 trees fitted to bootstrap resamples of a breast-cancer training set score 0.914 on the same held-out tumours, with a standard deviation of 0.015. Their structure is another matter. The full-data tree’s root feature, worst perimeter, is the root in 50% of resamples; four other features take the rest, and all five have best splits within 5% of each other....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Five features that differ only in how many distinct values they take, and a label that is a coin flip: a random forest’s impurity importance gives the continuous feature 42% of the total and the binary one 5%....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Out-of-bag error against five-fold cross-validation and a held-out half. With 500 trees it lands within 0.004 of the held-out error on breast cancer and 0.002 on digits, about as close as cross-validation, from a single fit....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Gradient-boosted trees on the breast-cancer data, ten half splits, 2,000 rounds. With clean labels and a learning rate of 0.1 the test error falls to 0.046 and, from round 30 on, never rises....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Trained on x between 0 and 1 with y rising as 2x, a random forest has test error 0.12 inside the range, against 0.51 for linear regression, and 2.71 between 1.5 and 2 — 22 times worse — because past x = 1 it never predicts above 2....