Neural-Networks on The Standard Errorhttps://jongha-jeon-dev.github.io/standarderror/tags/neural-networks/Recent content in Neural-Networks on The Standard ErrorHugo -- 0.123.7en-gbTue, 13 Oct 2026 00:00:00 +0000ML 21: Initialisation Must Break Symmetry, at the Right Scalehttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-21-init-scale/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-21-init-scale/Equal initial weights never become different, so they have to be random. In a deep network, random at the wrong scale fails just as completely.ML 22: Learning Rate and Batch Size Are One Knob, Until the Rate Hits a Wallhttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-22-lr-batch/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-22-lr-batch/With plain SGD and a fixed number of examples, multiplying the batch size and the learning rate by the same factor gives the same training run, until the learning rate reaches the point where training fails.ML 23: Adam Is a Little Faster and No Betterhttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-23-adam-sgd/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-23-adam-sgd/Tuned on validation data and run over twenty splits, Adam and SGD with momentum reach the same test accuracy. Adam's speed is real but small, and belongs to its learning rate.ML 24: Batch Normalisation Normalises the Batch, Not the Examplehttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-24-batchnorm/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-24-batchnorm/In training mode, a normalised network's answer for one digit depends on which other digits share its batch. That is fine in large, mixed batches and breaks in small or uniform ones.ML 25: Weight Scaling Keeps Dropout's Answers, Not Its Doubtshttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-25-dropout/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-25-dropout/Switching dropout off and scaling the weights gives almost the same labels as averaging many dropout masks. It does not give the same probabilities.ML 26: The Network That Memorises Noise Learns the Signal Firsthttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-26-random-labels/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-26-random-labels/A network that reaches 97% on test digits also fits random labels perfectly. Capacity cannot explain the first; the order in which it learns does.ML 27: Perfect Training Accuracy Was Not the End of Learninghttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-27-grokking/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-27-grokking/A network trained on part of a modular-addition table memorises it at once, scores nothing on the rest, and finds the rule thousands of steps later, while its training accuracy sits at 100% the whole time.ML 28: Augmentation Adds an Assumption, Not Datahttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-28-augmentation/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-28-augmentation/Shifted copies of 150 digits made a network worse on the test set and far better on shifted digits. Six hundred more real digits did the reverse.