ML 46: The Right Threshold Is a Cost Ratio, If the Probabilities Are Calibrated
Disclosure: this post was written with the assistance of an AI system (Claude), which wrote the analysis code, ran the experiments and drafted the text. The topic, the constraints, the data choices and the final review are the author’s. Episode 4’s simulated task with a miss costing r times a false alarm. At r = 10, flagging at 0.5 cost 1.44 to 2.00 per case, against 0.497 for flagging every case....