Abstract

Political scientists have long been concerned about the validity of survey measurements. Although many have studied classical measurement error in linear regression models where the error is assumed to arise completely at random, in a number of situations the error may be correlated with the outcome. We analyze the impact of differential measurement error on causal estimation. The proposed nonparametric identification analysis avoids arbitrary modeling decisions and formally characterizes the roles of different assumptions. We show the serious consequences of differential misclassification and offer a new sensitivity analysis that allows researchers to evaluate the robustness of their conclusions. Our methods are motivated by a field experiment on democratic deliberations, in which one set of estimates potentially suffers from differential misclassification. We show that an analysis ignoring differential measurement error may considerably overestimate the causal effects. This finding contrasts with the case of classical measurement error which always yields attenuation bias.

Replication Archive

https://doi.org/10.7910/DVN/TZOGL9

Related Papers

Blackwell, Matthew, Jacob R. Brown, Sophie Hill, Kosuke Imai, and Teppei Yamamoto (2025). ``Priming bias versus post-treatment bias in experimental designs.'' Political Analysis, Vol. 33, No. 4 (October), pp. 361-377. Winner of Political Analysis Editors' Choice Award.

Goplerud, Max, Kosuke Imai, Nicole E. Pashley (2025). ``Estimating Heterogeneous Causal Effects of High-Dimensional Treatments: Application to Conjoint Analysis.'' Annals of Applied Statistics, Vol. 19, No. 2 (June), pp. 866-888.

Ham, Dae Woong, Kosuke Imai, and Lucas Janson (2024). ``Using Machine Learning to Test Causal Hypotheses in Conjoint Analysis.'' Political Analysis, Vol. 32, No. 3 (July), pp. 329-344.

Johnson, Rebecca A., Tyler Simko, and Kosuke Imai (2024). ``A Summer Bridge Program for First-Generation Low-Income Students Stretches Academic Ambitions with No Adverse Impacts on GPA.'' Proceedings of the National Academy of Sciences, Vol. 121, No. 50, e2404924121.

de la Cuesta, Brandon, Naoki Egami, and Kosuke Imai (2022). ``Improving the External Validity of Conjoint Analysis: The Essential Role of Profile Distribution.'' Political Analysis, Vol. 30, No. 1 (January), pp. 19-45.

Egami, Naoki, and Kosuke Imai (2019). ``Causal Interaction in Factorial Experiments: Application to Conjoint Analysis.'' Journal of the American Statistical Association, Vol. 114, No. 526 (June), pp. 529-540.

Imai, Kosuke, and Zhichao Jiang (2018). ``A Sensitivity Analysis for Missing Outcomes Due to Truncation-by-Death under the Matched-Pairs Design.'' Statistics in Medicine, Vol. 37, No. 20 (September), pp. 2907-2922.

Imai, Kosuke, Dustin Tingley, and Teppei Yamamoto (2013). ``Experimental Designs for Identifying Causal Mechanisms.'' (with discussions) Journal of the Royal Statistical Society, Series A (Statistics in Society), Vol. 176, No. 1 (January), pp. 5-51. Read before the Royal Statistical Society in March, 2012.

Imai, Kosuke (2009). ``Statistical Analysis of Randomized Experiments with Nonignorable Missing Binary Outcomes: An Application to a Voting Experiment.'' Journal of the Royal Statistical Society, Series C (Applied Statistics), Vol. 58, No. 1 (February), pp. 83-104.

Ho, Daniel E., and Kosuke Imai (2008). ``Estimating Causal Effects of Ballot Order from a Randomized Natural Experiment: California Alphabet Lottery, 1978-2002.'' Public Opinion Quarterly, Vol. 72, No. 2 (Summer), pp. 216-240.

Imai, Kosuke (2008). ``Sharp Bounds on the Causal Effects in Randomized Experiments with ``Truncation-by-Death.'' Statistics & Probability Letters, Vol. 78, No. 2 (February), pp. 144-149.

Horiuchi, Yusaku, Kosuke Imai, and Naoko Taniguchi (2007). ``Designing and Analyzing Randomized Experiments: Application to a Japanese Election Survey Experiment.'' American Journal of Political Science, Vol. 51, No. 3 (July), pp. 669-687.

Ho, Daniel E., and Kosuke Imai (2006). ``Randomization Inference with Natural Experiments: An Analysis of Ballot Effects in the 2003 California Recall Election.'' Journal of the American Statistical Association, Vol. 101, No. 475 (September), pp. 888-900.