Extrapolation

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Trained on x between 0 and 1 with y rising as 2x, a random forest has test error 0.12 inside the range, against 0.51 for linear regression, and 2.71 between 1.5 and 2 — 22 times worse — because past x = 1 it never predicts above 2....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Eight character-level transformers from 4,288 to 1,789,248 non-embedding parameters were trained for the same 2,000 steps on the same million characters of Shakespeare (4.1 million characters seen). A power law L = 4....