Machine-Learning

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Seven simulated time-series settings, twenty series each, cross-validated three ways on 1,000 steps and compared with the error on the next 250. On a stationary autoregression with its lags as features, shuffled five-fold cross-validation was within 3% of the future error for linear regression, nearest neighbours and a forest....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. 2,000 simulated rows from 2,000, 400, 100, 40 or 20 subjects, each subject with a feature signature and an outcome propensity that do not carry over. With 2,000 subjects, row-level and subject-level cross-validation matched accuracy on new people to within a point....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. The breast-cancer data with 30% of cells removed three ways. At random, the imputation that recovered the values best (iterative, error 0.44 sd) gave the best model, log loss 0.091 against 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Simulated applicants, half accepted by an old policy, outcomes observed only for the accepted. When the policy used only the model’s features, a correctly specified model predicted a default rate of 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A simulated task with five numeric features and a category of 10 to 4,000 levels, 4,000 training rows, every encoding fitted on the training rows only. With a category carrying no signal and 1,000 levels, encoding each level by its mean label took boosted trees from test AUC 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Six classifiers on the digits, and the averaged probabilities of all 57 combinations of two or more. 11 combinations beat their best member’s accuracy and 8 fell below their members’ mean. Averaged with a single decision tree, whose probabilities are all 0 or 1, an SVM scoring 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Four base models on a simulated task, 100 to 3,000 training rows, and a logistic meta-model on their probabilities. Fitted on in-sample predictions it gave 1-NN a weight of 13 and reproduced its accuracy exactly (0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Gaussian naive Bayes on the breast-cancer data, whose features include radius, perimeter and area of the same nuclei, scored accuracy 0.940 against logistic regression’s 0.981, with log loss 0.611 against 0.073; 67% of its errors came with confidence above 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Two algorithms with exactly equal expected accuracy, by symmetry, compared on 400 data sets of 300 rows. The paired t-test on ten cross-validation folds declared a difference at the 5% level in 14....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Three ways to wear out a holdout without training on it. On a leaderboard of 1,000 coin-flip labels, the majority vote of random guesses that scored above 0.5 reached 0.713 after 1,000 submissions and 0....