Optimization on The Standard Errorhttps://jongha-jeon-dev.github.io/standarderror/tags/optimization/Recent content in Optimization on The Standard ErrorHugo -- 0.123.7en-gbTue, 13 Oct 2026 00:00:00 +0000Numerical Analysis 1: Your Learning Rate Is a Step Sizehttps://jongha-jeon-dev.github.io/standarderror/lectures/numerical-analysis-1-step-size/Thu, 03 Sep 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/numerical-analysis-1-step-size/Gradient descent is forward Euler on the gradient flow, so the learning rate is a discretisation step with a stability limit — exactly 2/λ_max on a quadratic, and on a network 2.04 times larger than that formula says, because the curvature moves to meet whatever step you chose.ML 21: Initialisation Must Break Symmetry, at the Right Scalehttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-21-init-scale/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-21-init-scale/Equal initial weights never become different, so they have to be random. In a deep network, random at the wrong scale fails just as completely.ML 22: Learning Rate and Batch Size Are One Knob, Until the Rate Hits a Wallhttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-22-lr-batch/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-22-lr-batch/With plain SGD and a fixed number of examples, multiplying the batch size and the learning rate by the same factor gives the same training run, until the learning rate reaches the point where training fails.ML 23: Adam Is a Little Faster and No Betterhttps://jongha-jeon-dev.github.io/standarderror/lectures/ml-23-adam-sgd/Tue, 13 Oct 2026 00:00:00 +0000https://jongha-jeon-dev.github.io/standarderror/lectures/ml-23-adam-sgd/Tuned on validation data and run over twenty splits, Adam and SGD with momentum reach the same test accuracy. Adam's speed is real but small, and belongs to its learning rate.