Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A marker pixel, blank or nearly so in every real digit image, is lit on 90% of the 3s, 5s, 8s and 9s in both training and test data. A random forest trained on it scores 0....
Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Ten small arithmetic transformers, graded on a pool of 2,500 problems and ranked on benchmarks drawn from it. The models’ 95% intervals covered their pool scores 94% to 95% of the time on average, but on 100 questions the best model ranked first in 50% of benchmarks, and seven close models came out in their true order in 0....
Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Two small arithmetic transformers, 200 problems per task, six kinds of context in front of each problem. The main model’s multiplication score was 0.735 after twelve ordinary lines and 0.500 after none....
Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Three small arithmetic transformers, the same problems in the same contexts, two ways of scoring. On four-digit sums none of them generated a single right answer in 200; asked to pick the likeliest of four options, they scored 0....
Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Three small arithmetic transformers answered 300 problems on each of three tasks, and each judged all three models’ answers by its own per-character likelihood, with a threshold set so its overall pass rate was right, up to ties....