Lectures

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Trained on x between 0 and 1 with y rising as 2x, a random forest has test error 0.12 inside the range, against 0.51 for linear regression, and 2.71 between 1.5 and 2 — 22 times worse — because past x = 1 it never predicts above 2....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A random forest on the diabetes data. Shuffling bmi costs 0.168 of test R²; refitting the forest without bmi costs 0.043, 4 times less. Add an exact copy of bmi and the permutation importances are 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. With y = x1 + 2 x2² and x2 a noisy copy of x1, the true effect of x1 is a straight line of slope 1. With independent inputs, the partial dependence plot of a random forest has slope 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Exact Shapley values, by enumerating all 32 coalitions, for a five-feature gradient-boosted model of the breast-cancer data. Against all training tumours as background, they add up to each prediction minus the mean to within 7e-15....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A marker pixel, blank or nearly so in every real digit image, is lit on 90% of the 3s, 5s, 8s and 9s in both training and test data. A random forest trained on it scores 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. On a simulated task where the best possible rule errs on 27.6% of the 5,000 test rows, five neural networks that differ only in their random seed score between 0.670 and 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A network of 30 hidden layers of 128 units was trained on the digits from weights drawn with standard deviation g / sqrt(fan-in). Constant weights never broke symmetry: after training all first-layer units were identical and accuracy was 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Plain SGD on a two-layer network on the digits, with 20 epochs’ worth of examples at every batch size, swept over learning rates from 0.003 to 6.144. Away from divergence, runs with the same ratio of learning rate to batch size reached the same training loss to within 2....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Adam and SGD with momentum 0.9, each with its learning rate tuned on a validation set, were trained from the same initial weights on 20 splits of the digits and of a simulated task with Bayes error 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A two-layer network with batch normalisation scored 0.990 on 100 test digits in inference mode. In training mode, each digit’s label changed with its random batch companions in 45% of batches of 2, 20% of batches of 4, 3....