Pytorch on The Standard Errorhttps://jongha-jeon-dev.github.io/standarderror/tags/pytorch/Recent content in Pytorch on The Standard ErrorHugo -- 0.123.7en-gbFri, 11 Sep 2026 00:00:00 +0000Gradients 1: Backprop Is the Chain Rule, and the Chain Rule Has Hypotheseshttps://jongha-jeon-dev.github.io/standarderror/lectures/gradients-1-kinks/Thu, 10 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-1-kinks/Autodiff returns a float at every point where the derivative does not exist, and the float is a choice. The choices disagree inside one library, which makes differentiation a function of the expression rather than of the function - and then almost none of it turns out to happen in a real transformer.Gradients 2: A Confident Attention Head Passes Almost No Gradienthttps://jongha-jeon-dev.github.io/standarderror/lectures/gradients-2-saturation/Thu, 10 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-2-saturation/The softmax Jacobian's norm is fixed within a factor of two by its largest probability alone, no matter how many positions it has. So one head of this model has committed hard enough that the gradient which could change its mind is gone - and it is the head the model cannot do without.Gradients 3: LayerNorm Deletes Exactly Two Directions of Your Gradienthttps://jongha-jeon-dev.github.io/standarderror/lectures/gradients-3-layernorm/Fri, 11 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-3-layernorm/Its Jacobian has rank exactly d minus 2, and the two missing directions are the invariances the layer was built to have. Then the residual connection restores the rank everywhere except the last norm, where it becomes an exact invariance of the entire network.Gradients 4: You Cannot Differentiate Through a Sampled Tokenhttps://jongha-jeon-dev.github.io/standarderror/lectures/gradients-4-sampling/Fri, 11 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-4-sampling/A sampled token is a step function of its logits, so the derivative is zero almost everywhere and undefined elsewhere. On a decision small enough to enumerate, the three things people return instead can be compared against the gradient that does exist.Gradients 5: The Gradient in Embedding Space Does Not Point at a Tokenhttps://jongha-jeon-dev.github.io/standarderror/lectures/gradients-5-embeddings/Fri, 11 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-5-embeddings/The embedding table is 65 near-orthogonal points on a sphere, so there is no local neighbourhood and a descent step leaves you nearest to the token you started from. The same gradient, asked to rank five candidates instead of to point, recovers almost all of the available gain.