Abstract

Conditioning on variables affected by treatment can induce post-treatment bias when estimating causal effects. Although this suggests that researchers should measure potential moderators before administering the treatment in an experiment, doing so may also bias causal effect estimation if the covariate measurement primes respondents to react differently to the treatment. This paper formally analyzes this trade-off between post-treatment and priming biases in three experimental designs that vary when moderators are measured: pre-treatment, post-treatment, or a randomized choice between the two. We derive nonparametric bounds for interactions between the treatment and the moderator under each design and show how to use substantive assumptions to narrow these bounds. These bounds allow researchers to assess the sensitivity of their empirical findings to priming and post-treatment bias. We then apply the proposed methodology to a survey experiment on electoral messaging.

Replication Archive

https://doi.org/10.7910/DVN/JZ55TF

Related Papers

Goplerud, Max, Kosuke Imai, Nicole E. Pashley (2025). ``Estimating Heterogeneous Causal Effects of High-Dimensional Treatments: Application to Conjoint Analysis.'' Annals of Applied Statistics, Vol. 19, No. 2 (June), pp. 866-888.

Ham, Dae Woong, Kosuke Imai, and Lucas Janson (2024). ``Using Machine Learning to Test Causal Hypotheses in Conjoint Analysis.'' Political Analysis, Vol. 32, No. 3 (July), pp. 329-344.

Johnson, Rebecca A., Tyler Simko, and Kosuke Imai (2024). ``A Summer Bridge Program for First-Generation Low-Income Students Stretches Academic Ambitions with No Adverse Impacts on GPA.'' Proceedings of the National Academy of Sciences, Vol. 121, No. 50, e2404924121.

de la Cuesta, Brandon, Naoki Egami, and Kosuke Imai (2022). ``Improving the External Validity of Conjoint Analysis: The Essential Role of Profile Distribution.'' Political Analysis, Vol. 30, No. 1 (January), pp. 19-45.

Egami, Naoki, and Kosuke Imai (2019). ``Causal Interaction in Factorial Experiments: Application to Conjoint Analysis.'' Journal of the American Statistical Association, Vol. 114, No. 526 (June), pp. 529-540.

Imai, Kosuke, and Zhichao Jiang (2018). ``A Sensitivity Analysis for Missing Outcomes Due to Truncation-by-Death under the Matched-Pairs Design.'' Statistics in Medicine, Vol. 37, No. 20 (September), pp. 2907-2922.

Imai, Kosuke, Dustin Tingley, and Teppei Yamamoto (2013). ``Experimental Designs for Identifying Causal Mechanisms.'' (with discussions) Journal of the Royal Statistical Society, Series A (Statistics in Society), Vol. 176, No. 1 (January), pp. 5-51. Read before the Royal Statistical Society in March, 2012.

Imai, Kosuke, and Teppei Yamamoto (2010). ``Causal Inference with Differential Measurement Error: Nonparametric Identification and Sensitivity Analysis.'' American Journal of Political Science, Vol. 54, No. 2 (April), pp. 543-560.

Imai, Kosuke (2009). ``Statistical Analysis of Randomized Experiments with Nonignorable Missing Binary Outcomes: An Application to a Voting Experiment.'' Journal of the Royal Statistical Society, Series C (Applied Statistics), Vol. 58, No. 1 (February), pp. 83-104.

Ho, Daniel E., and Kosuke Imai (2008). ``Estimating Causal Effects of Ballot Order from a Randomized Natural Experiment: California Alphabet Lottery, 1978-2002.'' Public Opinion Quarterly, Vol. 72, No. 2 (Summer), pp. 216-240.

Imai, Kosuke (2008). ``Sharp Bounds on the Causal Effects in Randomized Experiments with ``Truncation-by-Death.'' Statistics & Probability Letters, Vol. 78, No. 2 (February), pp. 144-149.

Horiuchi, Yusaku, Kosuke Imai, and Naoko Taniguchi (2007). ``Designing and Analyzing Randomized Experiments: Application to a Japanese Election Survey Experiment.'' American Journal of Political Science, Vol. 51, No. 3 (July), pp. 669-687.

Ho, Daniel E., and Kosuke Imai (2006). ``Randomization Inference with Natural Experiments: An Analysis of Ballot Effects in the 2003 California Recall Election.'' Journal of the American Statistical Association, Vol. 101, No. 475 (September), pp. 888-900.