Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. On a 20% test split of the UCI digits, 360 images, logistic regression, an RBF SVM and 3-NN score 0.967, 0.986 and 0.986, and one test set’s 95% binomial interval is about +-0....
Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. On 2,000 simulated least-squares datasets of 100 rows, where the exact test error of each fitted model is known, the correlation between the 10-fold CV estimate and that error is -0.01. The CV estimates spread 2....
Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Three ways to wear out a holdout without training on it. On a leaderboard of 1,000 coin-flip labels, the majority vote of random guesses that scored above 0.5 reached 0.713 after 1,000 submissions and 0....
Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Five models on a simulated task, ten training sets each. A random forest had the highest AUC every time (0.754) and the lowest cost every time when both errors cost the same....