ML 23: Adam Is a Little Faster and No Better
Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Adam and SGD with momentum 0.9, each with its learning rate tuned on a validation set, were trained from the same initial weights on 20 splits of the digits and of a simulated task with Bayes error 0....