ML 37: Rows From the Same People Are Not New Data
Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. 2,000 simulated rows from 2,000, 400, 100, 40 or 20 subjects, each subject with a feature signature and an outcome propensity that do not carry over. With 2,000 subjects, row-level and subject-level cross-validation matched accuracy on new people to within a point....