ML 47: The Model With the Best AUC Was Never the Best When Misses Cost Ten Times More
Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Five models on a simulated task, ten training sets each. A random forest had the highest AUC every time (0.754) and the lowest cost every time when both errors cost the same....