https://jongha-jeon-dev.github.io/standarderror/tags/deep-learning/Recent content in Deep-Learning on The Standard ErrorHugo -- 0.123.7en-gbTue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-1-kinks/Thu, 10 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-1-kinks/Autodiff returns a float at every point where the derivative does not exist, and the float is a choice. The choices disagree inside one library, which makes differentiation a function of the expression rather than of the function - and then almost none of it turns out to happen in a real transformer.https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-2-saturation/Thu, 10 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-2-saturation/The softmax Jacobian's norm is fixed within a factor of two by its largest probability alone, no matter how many positions it has. So one head of this model has committed hard enough that the gradient which could change its mind is gone - and it is the head the model cannot do without.https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-3-layernorm/Fri, 11 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-3-layernorm/Its Jacobian has rank exactly d minus 2, and the two missing directions are the invariances the layer was built to have. Then the residual connection restores the rank everywhere except the last norm, where it becomes an exact invariance of the entire network.https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-4-sampling/Fri, 11 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-4-sampling/A sampled token is a step function of its logits, so the derivative is zero almost everywhere and undefined elsewhere. On a decision small enough to enumerate, the three things people return instead can be compared against the gradient that does exist.https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-5-embeddings/Fri, 11 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/gradients-5-embeddings/The embedding table is 65 near-orthogonal points on a sphere, so there is no local neighbourhood and a descent step leaves you nearest to the token you started from. The same gradient, asked to rank five candidates instead of to point, recovers almost all of the available gain.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-21-init-scale/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-21-init-scale/Equal initial weights never become different, so they have to be random. In a deep network, random at the wrong scale fails just as completely.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-22-lr-batch/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-22-lr-batch/With plain SGD and a fixed number of examples, multiplying the batch size and the learning rate by the same factor gives the same training run, until the learning rate reaches the point where training fails.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-23-adam-sgd/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-23-adam-sgd/Tuned on validation data and run over twenty splits, Adam and SGD with momentum reach the same test accuracy. Adam's speed is real but small, and belongs to its learning rate.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-24-batchnorm/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-24-batchnorm/In training mode, a normalised network's answer for one digit depends on which other digits share its batch. That is fine in large, mixed batches and breaks in small or uniform ones.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-25-dropout/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-25-dropout/Switching dropout off and scaling the weights gives almost the same labels as averaging many dropout masks. It does not give the same probabilities.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-26-random-labels/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-26-random-labels/A network that reaches 97% on test digits also fits random labels perfectly. Capacity cannot explain the first; the order in which it learns does.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-27-grokking/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-27-grokking/A network trained on part of a modular-addition table memorises it at once, scores nothing on the rest, and finds the rule thousands of steps later, while its training accuracy sits at 100% the whole time.https://jongha-jeon-dev.github.io/standarderror/lectures/ml-30-scaling/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-30-scaling/Eight character transformers, one budget of data and steps. The law fitted to the five smallest predicts the next size almost exactly and the largest too optimistically.