Deep-Learning

ML 26: The Network That Memorises Noise Learns the Signal First

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A two-layer network with 512 units per layer, trained full-batch on 898 digits, reached 100% training accuracy in 41 steps on the true labels (test accuracy 0.968) and in 73 on entirely random labels (test 0....

ML 27: Perfect Training Accuracy Was Not the End of Learning

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A small network trained full-batch on 40% of the pairs of (a + b) mod 97 reached 100% training accuracy at step 200, when its accuracy on the other pairs was 0....

ML 30: A Power Law Fitted on Small Models Cannot See the Floor

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Eight character-level transformers from 4,288 to 1,789,248 non-embedding parameters were trained for the same 2,000 steps on the same million characters of Shakespeare (4.1 million characters seen). A power law L = 4....