Language-Models

ML 30: A Power Law Fitted on Small Models Cannot See the Floor

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Eight character-level transformers from 4,288 to 1,789,248 non-embedding parameters were trained for the same 2,000 steps on the same million characters of Shakespeare (4.1 million characters seen). A power law L = 4....

ML 52: A Leaked Benchmark Shows in the Set, Not in the Item

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. The repository’s main arithmetic model was trained for 600 more steps with 300 benchmark problems inserted into its stream k times, for k from 0 to 32. Five runs with nothing leaked scored 0....

ML 53: The Prompt Is Part of the Score

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Two small arithmetic transformers, 200 problems per task, six kinds of context in front of each problem. The main model’s multiplication score was 0.735 after twelve ordinary lines and 0.500 after none....

ML 54: Multiple Choice Grades the Wrong Answers You Offered

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Three small arithmetic transformers, the same problems in the same contexts, two ways of scoring. On four-digit sums none of them generated a single right answer in 200; asked to pick the likeliest of four options, they scored 0....

ML 55: Every Judge Ranked Itself First

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Three small arithmetic transformers answered 300 problems on each of three tasks, and each judged all three models’ answers by its own per-character likelihood, with a threshold set so its overall pass rate was right, up to ties....