ML 48: What a Model Relies On Is Not What to Change

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A simulated system with an unmeasured common cause, a habit it drives, a treatment that raises the outcome’s risk, and a test result produced by the outcome. Boosted trees put 0.45 of permutation importance on the test result, and raising it by one standard deviation in the model’s input raised the predicted risk by 0....

October 13, 2026 · 8 min · Jongha Jeon

ML 49: Target the Effect, Not the Risk

Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. A simulated retention offer, a randomised trial behind it and a budget for 20% of customers. Offering it to the highest predicted churners prevented 20.1 to 22.0 churns per 1,000 customers, about what random targeting prevented (21....

October 13, 2026 · 9 min · Jongha Jeon