``Statistical Decision Theory with Counterfactual Loss .''

 

  Abstract

Classical statistical decision theory evaluates treatment choices based solely on observed outcomes. However, by ignoring counterfactual outcomes, it cannot assess the quality of decisions relative to feasible alternatives. For example, the quality of a physician's decision may depend not only on patient survival, but also on whether a less invasive treatment could have produced a similar result. To address this limitation, we extend standard decision theory to incorporate counterfactual losses--criteria that evaluate decisions using all potential outcomes. The central challenge in this generalization is identification: because only one potential outcome is observed for each unit, the associated risk under a counterfactual loss is generally not identifiable. We show that under the assumption of strong ignorability, a counterfactual risk is identifiable if and only if the counterfactual loss function is additive in the potential outcomes. Moreover, we demonstrate that additive counterfactual losses can yield treatment recommendations that differ from those based on standard loss functions, provided that the decision problem involves more than two treatment options.

  Related Papers

Imai, Kosuke, Zhichao Jiang, D. James Greiner, Ryan Halen, and Sooahn Shin. (2023). ``Experimental Evaluation of Algorithm-Assisted Human Decision-Making: Application to Pretrial Public Safety Assessment.'' (with discussion) Journal of the Royal Statistical Society, Series A (Statistics in Society), Vol. 186, No. 2 (April), pp. 167-189. Read before the Royal Statistical Society.
Imai, Kosuke and Zhichao Jiang. (2023). ``Principal Fairness for Human and Algorithmic Decision-Making.'' Statistical Science, Vol. 38, No. 2 (July), pp317-328.
Ben-Michael, Eli, Kosuke Imai, and Zhichao Jiang. (2024). ``Policy Learning with Asymmetric Counterfactual Utilities.'' Journal of the American Statistical Association, Vol. 119, No. 548, pp. 3045-3058.
Ben-Michael, Eli, D. James Greiner, Melody Huang, Kosuke Imai, Zhichao Jiang, Sooahn Shin. ``Does AI help humans make better decisions? A statistical evaluation framework for experimental and observational studies.''

  Software

Shin, Sooahn, Zhichao Jiang, and Kosuke Imai. ``aihuman: Experimental Evaluation of Algorithm-Assisted Human Decision-Making .'' available through The Comprehensive R Archive Network and GitHub.

© Kosuke Imai
 Last modified: Sat May 17 16:32:53 BST 2025