Lectures

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. The softmax Jacobian is diag(p) minus p pᵀ, its quadratic form is exactly a variance under p, and so its spectral norm is trapped between m(1 - m) and 2m(1 - m) where m is the largest probability - checked on 25,443 random distributions with no violations and the upper end attained to nine digits....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Give every worker a universal raise of g and hire entrants below the median. The median slides m/2 ranks, order statistics near the median are spaced 1/(n f(M)) apart, so the median falls once the entry share passes 2 g M f(M) - which for a lognormal is 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Three measurements, and the third one reversed the plan. First: the H0 death spread, as a fraction of its own mean, falls from 4.87 at two dimensions to 0.078 at 768 on pure noise....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A gradient check is a finite difference, and a finite difference has an optimal step size: truncation error falls as the step shrinks and cancellation rises like eps/h, so the total is a U and refining past its bottom makes the answer worse — 9....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Forming X’X squares the condition number, so the formula everybody learns spends twice the digits of the two factorisations nobody is taught. Underneath, least squares is a right angle — the fit is the foot of a perpendicular onto the column space — and reading it that way explains why QR needs no inversion at all, and why the one check that defines a least-squares solution is blind to a normal-equations fit that is wrong by 321 percent....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Fit a step on all the rows, then cross-validate, and compare with the same step fitted inside the folds. Standardising leaks +0.05 points on the breast cancer data and mean imputation -0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. On 10,240 next-character predictions from the committed model, accuracy is 0.5466 at all seven temperatures from 0.5 to 3, identical to the bit, while ECE runs from 0.026 to 0.375. Between predictions the confidence order moves: at T = 2 Kendall’s tau against T = 1 is 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. The LayerNorm Jacobian is the orthogonal projector off the plane spanned by the all-ones vector and the normalised input, divided by sigma - verified against autodiff to 2e-16, rank exactly d minus 2, and with a flat nonzero spectrum because it really is a projector....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Take one lognormal population and distort it twice: the poorest tenth loses 90% of its income, or the richest hundredth has its income multiplied by 2.51. Tuned to the same Gini, both land on 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Positive semi-definiteness is usually presented as a technical condition on a matrix. It is not: w’Sw is the variance of the portfolio w’x, so a negative eigenvalue is a combination of your variables whose variance the matrix reports as negative, and the eigenvector names it....