|
|
Despite an increasing reliance on
fully-automated algorithmic decision-making in our day-to-day lives,
human beings still make highly consequential decisions. As
frequently seen in business, healthcare, and public policy,
recommendations produced by algorithms are provided to human
decision-makers to guide their decisions. While there exists a
fast-growing literature evaluating the bias and fairness of such
algorithmic recommendations, an overlooked question is whether they
help humans make better decisions. We develop a general statistical
methodology for experimentally evaluating the causal impacts of
algorithmic recommendations on human decisions. We also show how to
examine whether algorithmic recommendations improve the fairness of
human decisions and derive the optimal decision rules under various
settings. We apply the proposed methodology to preliminary data from
the first-ever randomized controlled trial that evaluates the
pretrial Public Safety Assessment (PSA) in the criminal justice
system. A goal of the PSA is to help judges decide which arrested
individuals should be released. On the basis of the preliminary data
available, we find that providing the PSA to the judge has little
overall impact on the judge's decisions and subsequent arrestee
behavior. Our analysis, however, yields some potentially suggestive
evidence that the PSA may help avoid unnecessarily harsh decisions
for female arrestees regardless of their risk levels while it
encourages the judge to make stricter decisions for male arrestees
who are deemed to be risky. In terms of fairness, the PSA appears to
increase an existing gender difference while having little effect on
any racial differences in judges' decision. Finally, we find that
the PSA's recommendations might be unnecessarily severe unless the
cost of a new crime is sufficiently high. |
Imai, Kosuke, Zhichao Jiang, D. James
Greiner, Ryan Halen, and Sooahn Shin. (2023). ``Authors' Reply to the
Discussion of `Experimental Evaluation of Algorithm-Assisted Human
Decision-Making: Application to Pretrial Public Safety
Assessment.''' Journal of the Royal Statistical
Society, Series A (Statistics in Society), Vol. 186, No. 2
(April), pp. 212–216. |
Jiang, Zhichao, Eli Ben-Michael, D. James
Greiner, Ryan Halen, Kosuke Imai, and Zhichao Jiang. ``Longitudinal Causal Inference with
Selective Eligibility'' |
Ben-Michael, Eli, D. James Greiner, Melody
Huang, Kosuke Imai, Zhichao Jiang, Sooahn Shin. ``Does AI help humans make better decisions? A
statistical evaluation framework for experimental and observational
studies.'' |
Ben-Michael, Eli, D. James Greiner, Kosuke
Imai, and Zhichao Jiang. ``Safe
Policy Learning through Extrapolation: Application to Pre-trial
Risk Assessment.'' |
Imai, Kosuke and Zhichao Jiang. (2023). ``Principal Fairness for Human and
Algorithmic Decision-Making.'' Statistical
Science, Vol. 38, No. 2 (July), pp317-328. |
Koch, Benedikt and Kosuke Imai. ``Statistical Decision Theory with
Counterfactual Loss.'' |
Shin, Sooahn, Zhichao Jiang, and Kosuke
Imai. ``aihuman:
Experimental Evaluation of Algorithm-Assisted Human Decision-Making
.'' available through The Comprehensive R
Archive Network and GitHub.
|