Interpretability

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Two hundred depth-3 trees fitted to bootstrap resamples of a breast-cancer training set score 0.914 on the same held-out tumours, with a standard deviation of 0.015. Their structure is another matter. The full-data tree’s root feature, worst perimeter, is the root in 50% of resamples; four other features take the rest, and all five have best splits within 5% of each other....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A random forest on the diabetes data. Shuffling bmi costs 0.168 of test R²; refitting the forest without bmi costs 0.043, 4 times less. Add an exact copy of bmi and the permutation importances are 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. With y = x1 + 2 x2² and x2 a noisy copy of x1, the true effect of x1 is a straight line of slope 1. With independent inputs, the partial dependence plot of a random forest has slope 0....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Exact Shapley values, by enumerating all 32 coalitions, for a five-feature gradient-boosted model of the breast-cancer data. Against all training tumours as background, they add up to each prediction minus the mean to within 7e-15....

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A simulated system with an unmeasured common cause, a habit it drives, a treatment that raises the outcome’s risk, and a test result produced by the outcome. Boosted trees put 0.45 of permutation importance on the test result, and raising it by one standard deviation in the model’s input raised the predicted risk by 0....