Research

In social science experiments, given fixed research budgets and/or limited numbers of participants, researchers need to optimize statistical efficiency. To this end, researchers often use stratification. However, conventional practices of stratification are agnostic with respect to the predictive relationship between the covariates and the potential outcomes, even though it is this predictive relationship that motivates stratification. Such practices are thus suboptimal when outcome data is available for some observations. This paper introduces an adaptive pairing and stratification procedure for running an experiment in batches. This approach builds upon recent work that demonstrates the theoretical optimality of pairing units based on the expected sum of potential outcomes. The method incorporates information about the relationship between covariates and potential outcomes when pairing or stratifying units. It uses data from earlier batches not just to inform pairing decisions but also to rematch observations across different batches without compromising the validity of inference. In experimental settings where sequential treatment assignment and outcome collection are feasible, this approach can improve the efficiency of treatment assignments relative to its alternatives. My simulations demonstrate such gains can be substantial.

Difference-in-differences (DiD) designs for estimating causal effects have grown in popularity throughout political science. It is common for DiD studies report their main results using a "dynamic" or "event study" two-way fixed effects (TWFE) regression. This regression combines estimates of average treatment effects for multiple post-treatment time periods alongside placebo tests of the main identifying assumption: parallel trends. Despite their ubiquity, there is little clear and consistent guidance in the discipline for how researchers should estimate dynamic treatment effects. This paper develops a novel decomposition of the dynamic TWFE regression coefficients in terms of their component 2x2 difference-in-differences comparisons in the style of Goodman-Bacon (2021). We use this decomposition to illustrate how bias can result from the incorrect specification of baseline time periods, the inclusion of units and time periods where all observations are treated, and heterogeneity in the dynamic treatment effects across different treatment timing groups. Our results provide additional intuition for the source of bias due to effect heterogeneity—what Sun and Abraham (2021) term "contamination bias"—by directly characterizing the contaminated 2x2 comparisons. We then provide a common framework for connecting the many proposed “heterogeneity-robust” estimators in the literature, noting that they vary primarily in which 2x2 comparisons they choose to include. Through a replication of three studies published in prominent political science journals, we conclude by showing how attentiveness to baseline selection and specification can alter findings.

Social scientists use survey experiments to study the effect of information on individual attitudes and behaviors. However, such experiments may fail to provide respondents with the information as intended. If the theorized mechanism is correct, noncompliance attenuates results, but noncompliance can also arise if the experiment exposes respondents to unintended information, affecting the substantive interpretation of results. In this note, we propose a diagnostic test and recommendations for treatment design that will help researchers evaluate theoretical mechanisms in survey experiments. This placebo test repurposes treatment-relevant manipulation checks to evaluate responses under control conditions. This approach offers a path toward more robust and more informative survey experiments.