``Does AI help humans make better decisions? A statistical evaluation framework for experimental and observational studies.''

The use of Artificial Intelligence (AI), or more generally data-driven algorithms, has become ubiquitous in today's society. Yet, in many cases and especially when stakes are high, humans still make final decisions. The critical question, therefore, is whether AI helps humans make better decisions compared to a human-alone or AI-alone system. We introduce a new methodological framework to empirically answer this question with a minimal set of assumptions. We measure a decision maker's ability to make correct decisions using standard classification metrics based on the baseline potential outcome. We consider a single-blinded and unconfounded treatment assignment, where the provision of AI-generated recommendations is assumed to be randomized across cases with humans making final decisions. Under this study design, we show how to compare the performance of three alternative decision-making systems --- human-alone, human-with-AI, and AI-alone. Importantly, the AI-alone system includes any individualized treatment assignment, including those that are not used in the original study. We also show when AI recommendations should be provided to a human-decision maker, and when one should follow such recommendations. We apply the proposed methodology to our own randomized controlled trial evaluating a pretrial risk assessment instrument. We find that the risk assessment recommendations do not improve the classification accuracy of a judge's decision to impose cash bail. Furthermore, we find that replacing a human judge with algorithms --- the risk assessment score and a large language model in particular --- leads to a worse classification performance.

Imai, Kosuke, Zhichao Jiang, D. James Greiner, Ryan Halen, and Sooahn Shin. (2023). ``Experimental Evaluation of Algorithm-Assisted Human Decision-Making: Application to Pretrial Public Safety Assessment.'' (with discussion) Journal of the Royal Statistical Society, Series A (Statistics in Society), Vol. 186, No. 2 (April), pp. 167-189. Read before the Royal Statistical Society.

Ben-Michael, Eli, D. James Greiner, Kosuke Imai, and Zhichao Jiang. ``Safe Policy Learning through Extrapolation: Application to Pre-trial Risk Assessment.''

Jiang, Zhichao, Eli Ben-Michael, D. James Greiner, Ryan Halen, Kosuke Imai, and Zhichao Jiang. ``Longitudinal Causal Inference with Selective Eligibility''

Imai, Kosuke and Zhichao Jiang. (2023). ``Principal Fairness for Human and Algorithmic Decision-Making.'' Statistical Science, Vol. 38, No. 2 (July), pp317-328.

Koch, Benedikt and Kosuke Imai. ``Statistical Decision Theory with Counterfactual Loss.''

Shin, Sooahn, Zhichao Jiang, and Kosuke Imai. ``aihuman: Experimental Evaluation of Algorithm-Assisted Human Decision-Making .'' available through The Comprehensive R Archive Network and GitHub.