ML 24: Batch Normalisation Normalises the Batch, Not the Example
Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A two-layer network with batch normalisation scored 0.990 on 100 test digits in inference mode. In training mode, each digit’s label changed with its random batch companions in 45% of batches of 2, 20% of batches of 4, 3....