Lectures

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Three training runs of the same 816,128-parameter model, evaluated at every checkpoint on 10,240 held-out predictions. Every run starts slightly underconfident; overconfidence begins after one to ten passes over the training text and grows as training loss pulls away from validation loss....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. On a simulated task whose Bayes error is 0.268, with training sets grown to 25,600 rows, the two-parameter power law a n^-b fitted to the smaller sizes predicted too low an error in all 9 fits, and in 4 of them predicted an error below the Bayes error - 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. The gradient of the expectation is p times loss minus its mean, which is episode 2’s softmax Jacobian applied to the losses and sums to exactly zero. Over 65 tokens it is enumerable, so every estimator has an answer to be wrong against....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Every applied account of PCA triages components by the share of variance they carry, and stability does not work that way: an eigenvector is determined by the distance from its eigenvalue to the nearest other one, so the largest component in a decomposition can be the least trustworthy axis in it....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Over 200 random row splits of 24,576 predictions, realised conformal coverage at alpha = 0.1 has an SD of 0.0040, 1.49 times the Beta spread episode 1 compared it with. That comparison was short a term: the coverage is measured on a finite test half, and with the test half’s binomial variance added the excess is 1....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. At 2% prevalence, logistic regression trained on an oversampled, SMOTE-augmented or class-weighted copy of the data ranks the test set no better than the plain model - AUC -0.0012 for oversampling, small and consistently negative - while its mean predicted probability goes from 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. The embedding table’s median pairwise distance is 11.214 and sqrt(2) times its median norm is 11.210 - the distance between orthogonal vectors of that length, matched to four figures, with a median pairwise cosine of +0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Ridge is introduced as a penalty and works as an operation on the spectrum: in the basis where the design is diagonal it multiplies each direction by s²/(s² + α) and does nothing else, which is why it shrinks hardest exactly where the data saw least....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Every one of 50 stratified half splits of the Wisconsin breast cancer data is linearly separable on its 30 features, so the unpenalised logistic likelihood has no maximum. scikit-learn returns coefficients of norm 2....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. The diagonal of the projection matrix always sums to the number of columns, so average leverage is p/n and no data can change it — but nothing stops one row taking a whole unit....