Neural-Networks

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A network of 30 hidden layers of 128 units was trained on the digits from weights drawn with standard deviation g / sqrt(fan-in). Constant weights never broke symmetry: after training all first-layer units were identical and accuracy was 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Plain SGD on a two-layer network on the digits, with 20 epochs’ worth of examples at every batch size, swept over learning rates from 0.003 to 6.144. Away from divergence, runs with the same ratio of learning rate to batch size reached the same training loss to within 2....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Adam and SGD with momentum 0.9, each with its learning rate tuned on a validation set, were trained from the same initial weights on 20 splits of the digits and of a simulated task with Bayes error 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A two-layer network with batch normalisation scored 0.990 on 100 test digits in inference mode. In training mode, each digit’s label changed with its random batch companions in 45% of batches of 2, 20% of batches of 4, 3....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Networks trained with dropout 0.5 were evaluated two ways: with dropout off and the weights scaled, and by averaging up to 300 sampled masks. On the digits the two agreed on 99....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A two-layer network with 512 units per layer, trained full-batch on 898 digits, reached 100% training accuracy in 41 steps on the true labels (test accuracy 0.968) and in 73 on entirely random labels (test 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A small network trained full-batch on 40% of the pairs of (a + b) mod 97 reached 100% training accuracy at step 200, when its accuracy on the other pairs was 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Trained on 150 digits, a network scored 0.917. Four times as many real digits raised that by 0.041; four shifted copies of each image changed it by -0.027, pixel noise by +0....