Abstract

The use of Artificial Intelligence (AI), or more generally data-driven algorithms, has become ubiquitous in today’s society. Yet, in many cases and especially when stakes are high, humans still make final decisions. The critical question, therefore, is whether AI helps humans make better decisions compared to a human-alone or AI-alone system. We introduce a new methodological framework to empirically answer this question with a minimal set of assumptions. We measure a decision maker’s ability to make correct decisions using standard classification metrics based on the baseline potential outcome. We consider a single-blinded and unconfounded treatment assignment, where the provision of AI-generated recommendations is assumed to be randomized across cases with humans making final decisions. Under this study design, we show how to compare the performance of three alternative decision-making systems — human-alone, human-with-AI, and AI-alone. Importantly, the AI-alone system includes any individualized treatment assignment, including those that are not used in the original study. We also show when AI recommendations should be provided to a human-decision maker, and when one should follow such recommendations. We apply the proposed methodology to our own randomized controlled trial evaluating a pretrial risk assessment instrument. We find that the risk assessment recommendations do not improve the classification accuracy of a judge’s decision to impose cash bail. Furthermore, we find that replacing a human judge with algorithms — the risk assessment score and a large language model in particular — leads to a worse classification performance.

Replication Archive

https://doi.org/10.7910/DVN/KMM8WN

Software

aihuman: Experimental Evaluation of Algorithm-Assisted Human Decision-Making — CRAN / GitHub

Related Papers

Zhou, Lingxiao, Kosuke Imai, Jason Lyall, and Georgia Papadogeorgou (2026). ``Dynamic Policy Evaluation and Learning with Spatio-temporal Data.''

Jia, Zeyang, Eli Ben-Michael, and Kosuke Imai (2026). ``Bayesian Safe Policy Learning with Chance Constrained Optimization: Application to Military Security Assessment during the Vietnam War.'' Journal of the Royal Statistical Society, Series A (Statistics in Society), Vol. 189, No. 3, pp. 1448-1472.

Imai, Kosuke, Sooahn Shin, D. James Greiner, and Ryan Halen (2026). ``Triage Score: A Counterfactual Risk Assessment Instrument.''

Koch, Benedikt, Kosuke Imai, and Tomasz Strzalecki (2026). ``An Axiomatic Foundation for Decisions with Counterfactual Utility.''

Koch, Benedikt and Kosuke Imai (2025). ``Statistical Decision Theory with Counterfactual Loss.''

Li, Michael Lingzhi and Kosuke Imai (2025). ``Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning.''

Ben-Michael, Eli, D. James Greiner, Kosuke Imai, and Zhichao Jiang (2025). ``Safe Policy Learning through Extrapolation: Application to Pre-trial Risk Assessment.'' Journal of the American Statistical Association, Vol. 120, No. 551, pp. 1386-1399.

Imai, Kosuke and Michael Lingzhi Li (2025). ``Statistical Inference for Heterogeneous Treatment Effects Discovered by Generic Machine Learning in Randomized Experiments.'' Journal of Business & Economic Statistics, Vol. 43, No. 1, pp. 256-268.

Jia, Zeyang, Kosuke Imai, and Michael Lingzhi Li (2025). ``Cramming Contextual Bandits for On-policy Statistical Evaluation.''

Zhang, Yi and Kosuke Imai (2025). ``Individualized Policy Evaluation and Learning under Clustered Network Interference.''

Zhang, Yi, Eli Ben-Michael, and Kosuke Imai (2024). ``Safe Policy Learning under Regression Discontinuity Designs.''

Ben-Michael, Eli, Kosuke Imai, and Zhichao Jiang (2024). ``Policy Learning with Asymmetric Counterfactual Utilities.'' Journal of the American Statistical Association, Vol. 119, No. 548, pp. 3045-3058.

Li, Michael Lingzhi and Kosuke Imai (2024). ``Neyman Meets Causal Machine Learning: Experimental Evaluation of Individualized Treatment Rules.'' Journal of Causal Inference, Vol 12, No. 1, pp. 1-20. Special Issue on Neyman (1923) and its influences on causal inference

Imai, Kosuke and Zhichao Jiang (2023). ``Principal Fairness for Human and Algorithmic Decision-Making.'' Statistical Science, Vol. 38, No. 2 (July), pp317-328.

Imai, Kosuke and Michael Lingzhi Li (2023). ``Experimental Evaluation of Individualized Treatment Rules.'' Journal of the American Statistical Association, Vol. 118, No. 541, pp. 242-256.

Imai, Kosuke, Zhichao Jiang, D. James Greiner, Ryan Halen, and Sooahn Shin (2023). ``Experimental Evaluation of Algorithm-Assisted Human Decision-Making: Application to Pretrial Public Safety Assessment.'' (with discussion) Journal of the Royal Statistical Society, Series A (Statistics in Society), Vol. 186, No. 2 (April), pp. 167-189. Read before the Royal Statistical Society.

Imai, Kosuke, and Aaron Strauss (2011). ``Estimation of Heterogeneous Treatment Effects from Randomized Experiments, with Application to the Optimal Planning of the Get-out-the-vote Campaign.'' Political Analysis, Vol. 19, No. 1 (Winter), pp. 1-19. Winner of Political Analysis Editors' Choice Award.