Mathematics

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Two character transformers, identical in architecture, parameters, seed, steps, schedule and training problems. The one manipulated variable is the order of the lines in the training file. Packed in generation order, every reversed sum lands directly beneath the forward form of the same sum - 5,076 of 5,076, which is all of them - and ‘reverse the line above’ answers the question with no arithmetic....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. relu, abs, clamp, hardtanh and a vector norm all return 0 at their kinks; maximum, minimum and max split the tie and return 0.5; and sqrt(x·x), which is abs(x), returns nan where abs returns 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. The softmax Jacobian is diag(p) minus p pᵀ, its quadratic form is exactly a variance under p, and so its spectral norm is trapped between m(1 - m) and 2m(1 - m) where m is the largest probability - checked on 25,443 random distributions with no violations and the upper end attained to nine digits....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. The LayerNorm Jacobian is the orthogonal projector off the plane spanned by the all-ones vector and the normalised input, divided by sigma - verified against autodiff to 2e-16, rank exactly d minus 2, and with a flat nonzero spectrum because it really is a projector....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. The gradient of the expectation is p times loss minus its mean, which is episode 2’s softmax Jacobian applied to the losses and sums to exactly zero. Over 65 tokens it is enumerable, so every estimator has an answer to be wrong against....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. The embedding table’s median pairwise distance is 11.214 and sqrt(2) times its median norm is 11.210 - the distance between orthogonal vectors of that length, matched to four figures, with a median pairwise cosine of +0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Buses that average a 10-minute gap will, if the gaps are uneven, leave the average passenger sitting in a 20-minute gap and waiting 10 minutes instead of 5. Nobody is misremembering. The same arithmetic makes the average student’s class bigger than the average class, and it has a one-line formula that says exactly how much bigger....