Abstract

Randomized experiments are becoming increasingly common in political science. Despite their well-known advantages over observational studies, randomized experiments are not free from complications. In particular, researchers often cannot force subjects to comply with treatment assignment and to provide the requested information. Furthermore, simple randomization of treatments remains the most commonly used method in the discipline even though more efficient procedures are available. Building on the recent statistical literature, we address these methodological issues by offering general recommendations for designing and analyzing randomized experiements to improve the validity and efficiency of causal inference. We also develop a new statistical methodology to explore causal heterogeneity. The proposed methods are applied to a survey experiment conducted during Japan’s 2004 Upper House election, where randomly selected voters were encouraged to obtain policy information from political parties’ websites. An R package is publicly available for implementing various methods useful for designing and analyzing randomized experiments.

Replication Archive

https://doi.org/10.7910/DVN/B6OJKG

Software

experiment: R Package for Designing and Analyzing Randomized Experiments — CRAN

Related Papers

Blackwell, Matthew, Jacob R. Brown, Sophie Hill, Kosuke Imai, and Teppei Yamamoto (2025). ``Priming bias versus post-treatment bias in experimental designs.'' Political Analysis, Vol. 33, No. 4 (October), pp. 361-377. Winner of Political Analysis Editors' Choice Award.

Goplerud, Max, Kosuke Imai, Nicole E. Pashley (2025). ``Estimating Heterogeneous Causal Effects of High-Dimensional Treatments: Application to Conjoint Analysis.'' Annals of Applied Statistics, Vol. 19, No. 2 (June), pp. 866-888.

Ham, Dae Woong, Kosuke Imai, and Lucas Janson (2024). ``Using Machine Learning to Test Causal Hypotheses in Conjoint Analysis.'' Political Analysis, Vol. 32, No. 3 (July), pp. 329-344.

Johnson, Rebecca A., Tyler Simko, and Kosuke Imai (2024). ``A Summer Bridge Program for First-Generation Low-Income Students Stretches Academic Ambitions with No Adverse Impacts on GPA.'' Proceedings of the National Academy of Sciences, Vol. 121, No. 50, e2404924121.

de la Cuesta, Brandon, Naoki Egami, and Kosuke Imai (2022). ``Improving the External Validity of Conjoint Analysis: The Essential Role of Profile Distribution.'' Political Analysis, Vol. 30, No. 1 (January), pp. 19-45.

Egami, Naoki, and Kosuke Imai (2019). ``Causal Interaction in Factorial Experiments: Application to Conjoint Analysis.'' Journal of the American Statistical Association, Vol. 114, No. 526 (June), pp. 529-540.

Imai, Kosuke, and Zhichao Jiang (2018). ``A Sensitivity Analysis for Missing Outcomes Due to Truncation-by-Death under the Matched-Pairs Design.'' Statistics in Medicine, Vol. 37, No. 20 (September), pp. 2907-2922.

Imai, Kosuke, Dustin Tingley, and Teppei Yamamoto (2013). ``Experimental Designs for Identifying Causal Mechanisms.'' (with discussions) Journal of the Royal Statistical Society, Series A (Statistics in Society), Vol. 176, No. 1 (January), pp. 5-51. Read before the Royal Statistical Society in March, 2012.

Imai, Kosuke, and Teppei Yamamoto (2010). ``Causal Inference with Differential Measurement Error: Nonparametric Identification and Sensitivity Analysis.'' American Journal of Political Science, Vol. 54, No. 2 (April), pp. 543-560.

Imai, Kosuke (2009). ``Statistical Analysis of Randomized Experiments with Nonignorable Missing Binary Outcomes: An Application to a Voting Experiment.'' Journal of the Royal Statistical Society, Series C (Applied Statistics), Vol. 58, No. 1 (February), pp. 83-104.

Ho, Daniel E., and Kosuke Imai (2008). ``Estimating Causal Effects of Ballot Order from a Randomized Natural Experiment: California Alphabet Lottery, 1978-2002.'' Public Opinion Quarterly, Vol. 72, No. 2 (Summer), pp. 216-240.

Imai, Kosuke (2008). ``Sharp Bounds on the Causal Effects in Randomized Experiments with ``Truncation-by-Death.'' Statistics & Probability Letters, Vol. 78, No. 2 (February), pp. 144-149.

Ho, Daniel E., and Kosuke Imai (2006). ``Randomization Inference with Natural Experiments: An Analysis of Ballot Effects in the 2003 California Recall Election.'' Journal of the American Statistical Association, Vol. 101, No. 475 (September), pp. 888-900.