``Experimental Evaluation of Individualized Treatment Rules''

 

  Abstract

In recent years, the increasing availability of individual-level data has led to the active development and application of individualized (or personalized) treatment rules (ITRs) in business, medicine, politics, and other fields. We consider a common setting, in which researchers use the same experimental data to both estimate and evaluate ITRs via cross-validation under a budget constraint. Using the Neyman's repeated sampling framework, we show how to quantify the estimation and validation uncertainties for the experimental evaluation of the ITRs. Specifically, we propose evaluation metrics, develop their unbiased estimators, and derive the exact variances by directly exploiting three randomization procedures of the evaluation design --- the random sampling of units, random assignment of treatment, and random splits of cross-validation. Unlike the existing methods, the proposed methodology does not require modeling assumptions, asymptotic approximation, or resampling method. As a result, it is applicable to any ITR including those based on complex machine learning algorithms. Finally, we propose a new metric, Area under the Prescriptive Effect Curve (AUPEC), to compare the performance of multiple ITRs. Simulation and empirical studies demonstrate the accuracy and wide applicability of our evaluation strategy. The open-source software package is available for implementing the proposed methodology. (Last Revised, June 2020)

  Software

You may be interested in the following software, which implements the proposed method: Li, Michael Lingzhi and Kosuke Imai. ``evalITR: Evaluating Individualized Treatment Rules.'' available through The Comprehensive R Archive Network and GitHub

© Kosuke Imai
 Last modified: Tue Jun 2 16:54:09 EDT 2020