Abstract

Algorithmic and data-driven decisions and recommendations are commonly used in high-stakes decision-making settings such as criminal justice, medicine, and public policy. We investigate whether it would have been possible to improve a security assessment algorithm employed during the Vietnam War, using outcomes measured immediately after its introduction in late 1969. This empirical application raises several methodological challenges that frequently arise in high-stakes algorithmic decision-making. First, before implementing a new algorithm, it is essential to characterize and control the risk of yielding worse outcomes than the existing algorithm. Second, the existing algorithm is deterministic, and learning a new algorithm requires transparent extrapolation. Third, the existing algorithm involves discrete decision tables that are common but difficult to optimize over. To address these challenges, we introduce the Average Conditional Risk (ACRisk), which first quantifies the risk that a new algorithmic policy leads to worse outcomes for subgroups of individual units and then averages this over the distribution of subgroups. We also propose a Bayesian policy learning framework that maximizes the posterior expected value while controlling the posterior expected ACRisk. This framework separates the estimation of heterogeneous treatment effects from policy optimization, enabling flexible estimation of effects and optimization over complex policy classes. We characterize the resulting chance-constrained optimization problem as a constrained linear programming problem. Our analysis shows that compared to the actual algorithm used during the Vietnam War, the learned algorithm assesses most regions as more secure and emphasizes economic and political factors over military factors.

Replication Archive

https://doi.org/10.7910/DVN/SPR616

Related Papers

Zhou, Lingxiao, Kosuke Imai, Jason Lyall, and Georgia Papadogeorgou (2026). ``Dynamic Policy Evaluation and Learning with Spatio-temporal Data.''

Imai, Kosuke, Sooahn Shin, D. James Greiner, and Ryan Halen (2026). ``Triage Score: A Counterfactual Risk Assessment Instrument.''

Koch, Benedikt, Kosuke Imai, and Tomasz Strzalecki (2026). ``An Axiomatic Foundation for Decisions with Counterfactual Utility.''

Koch, Benedikt and Kosuke Imai (2025). ``Statistical Decision Theory with Counterfactual Loss.''

Li, Michael Lingzhi and Kosuke Imai (2025). ``Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning.''

Ben-Michael, Eli, D. James Greiner, Kosuke Imai, and Zhichao Jiang (2025). ``Safe Policy Learning through Extrapolation: Application to Pre-trial Risk Assessment.'' Journal of the American Statistical Association, Vol. 120, No. 551, pp. 1386-1399.

Ben-Michael, Eli, D. James Greiner, Melody Huang, Kosuke Imai, Zhichao Jiang, Sooahn Shin (2025). ``Does AI help humans make better decisions? A statistical evaluation framework for experimental and observational studies.'' Proceedings of the National Academy of Sciences, Vol. 122, No. 38, e2505106122.

Imai, Kosuke and Michael Lingzhi Li (2025). ``Statistical Inference for Heterogeneous Treatment Effects Discovered by Generic Machine Learning in Randomized Experiments.'' Journal of Business & Economic Statistics, Vol. 43, No. 1, pp. 256-268.

Jia, Zeyang, Kosuke Imai, and Michael Lingzhi Li (2025). ``Cramming Contextual Bandits for On-policy Statistical Evaluation.''

Zhang, Yi and Kosuke Imai (2025). ``Individualized Policy Evaluation and Learning under Clustered Network Interference.''

Zhang, Yi, Eli Ben-Michael, and Kosuke Imai (2024). ``Safe Policy Learning under Regression Discontinuity Designs.''

Ben-Michael, Eli, Kosuke Imai, and Zhichao Jiang (2024). ``Policy Learning with Asymmetric Counterfactual Utilities.'' Journal of the American Statistical Association, Vol. 119, No. 548, pp. 3045-3058.

Li, Michael Lingzhi and Kosuke Imai (2024). ``Neyman Meets Causal Machine Learning: Experimental Evaluation of Individualized Treatment Rules.'' Journal of Causal Inference, Vol 12, No. 1, pp. 1-20. Special Issue on Neyman (1923) and its influences on causal inference

Imai, Kosuke and Zhichao Jiang (2023). ``Principal Fairness for Human and Algorithmic Decision-Making.'' Statistical Science, Vol. 38, No. 2 (July), pp317-328.

Imai, Kosuke and Michael Lingzhi Li (2023). ``Experimental Evaluation of Individualized Treatment Rules.'' Journal of the American Statistical Association, Vol. 118, No. 541, pp. 242-256.

Imai, Kosuke, Zhichao Jiang, D. James Greiner, Ryan Halen, and Sooahn Shin (2023). ``Experimental Evaluation of Algorithm-Assisted Human Decision-Making: Application to Pretrial Public Safety Assessment.'' (with discussion) Journal of the Royal Statistical Society, Series A (Statistics in Society), Vol. 186, No. 2 (April), pp. 167-189. Read before the Royal Statistical Society.

Imai, Kosuke, and Aaron Strauss (2011). ``Estimation of Heterogeneous Treatment Effects from Randomized Experiments, with Application to the Optimal Planning of the Get-out-the-vote Campaign.'' Political Analysis, Vol. 19, No. 1 (Winter), pp. 1-19. Winner of Political Analysis Editors' Choice Award.