Numerical Analysis 1: Your Learning Rate Is a Step Size
Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. The optimiser everyone uses is an ODE solver, and that is an identity rather than an analogy: gradient descent is forward Euler on the gradient flow. So the learning rate is a step size, and step sizes have stability limits....