The Unfulfilled Promise of Longitudinal Designs for Causal Inference

In longitudinal designs, deciding whether to adjust or not to adjust for² any given variable can be contradictory. We pay particular attention to the routine practice of adjusting for previous measurements. Intuitively, this approach should bolster causal inferences: previous measurements are often either predictive of or share common causes with both treatment and outcome, so pre-existing (or pre-treatment) stable differences in the outcome can be obviated by adjusting for such measures. But the causal assumptions underpinning valid inferences are largely unexamined and routinely overlooked in practice. What are the implications of adjusting for previous measurements when analyzing longitudinal data? In this section, we aim to delve into this inquiry by using typical scenarios encountered in longitudinal studies. As our illustrations will show, the implication of adjusting for previous measurements depends on the specific data-generating causal structures. Routinely adjusting for previous measurements can all too easily lead to incorrect causal inferences.

We consider a minimal example with longitudinal data in two waves. Suppose a researcher is interested in drawing inferences about the causal effect of being a victim of microaggression, such as being treated as irrelevant and invisible (non-randomized treatment $X$⁠) on the development of depression symptoms (outcome $Y$⁠). Variables recorded for each participant include relevant baseline time-invariant covariates, such as racial or ethnic identities, socioeconomic status, and unemployment (denoted collectively by $C$ for simplicity), the experience of invisibility at time 1 (⁠$X1$⁠), depression symptoms within the same time point or wave at time 1 (⁠$Y1$⁠), and a follow-up measure of depression symptoms after a delay (time 2; $Y2$⁠).

To illustrate the causal assumptions in this example, we use causal diagrams to visualize plausible data-generating scenarios which cannot be ruled out empirically without imposing additional restrictions using theoretical knowledge.³ In all causal diagrams, we denote participants’ experience of perceived invisibility by $X$ and depression symptoms by $Y$⁠, with subscripts denoting the measurement time point. Measured common causes, such as being a racial or ethnic minority, socioeconomic status, and unemployment, are jointly denoted by $C$⁠.⁴ We adopt the convention of using a round node to denote a hidden or unmeasured variable. For example, a hidden common cause of the contemporaneously measured $X1$ and $Y1$ is denoted simply by $D$⁠. Repeated outcome measurements are likely to be (auto)correlated due to hidden common causes or underlying processes; these are denoted by $U$⁠.⁵ Because $D$ and $U$ are unmeasured, they are ruled out from adjustment.⁶ Throughout this paper, we focus on the average (total) causal effect of $X1$ on $Y2$⁠. Therefore, we seek to close or block all non-causal paths with treatment $X1$ and outcome $Y2$ as the endpoints.

We acknowledge that adjusting for a previous outcome measurement can strengthen causal conclusions. We illustrate this point using two possible scenarios in Figure 1.

In Figure 1(a), within the same wave at time 1, experiencing invisibility (⁠$X1$⁠) is correlated with participants’ depression symptoms (⁠$Y1$⁠) due to a hidden common cause $D$⁠, such as being denied a promotion opportunity at work. Participants’ depression symptoms at time 1 (⁠$Y1$⁠) may have an autoregressive effect on their depression symptoms at time 2 (⁠$Y2$⁠). Here, we make a stringent assumption that this effect of $Y1$ on $Y2$ is unconfounded, as indicated by the absence of unmeasured common causes shared by $Y1$ and $Y2$⁠. Under this scenario, adjusting for all baseline covariates (e.g., $C$⁠) and the previous outcome measurement (⁠$Y1$⁠) suffices to block all non-causal paths linking $X1$ and $Y2$⁠. Therefore, adjusting for the previous measurement $Y1$ is necessary for valid causal inference.

We now turn to a different scenario. In Figure 1(b), we relax the causal assumptions regarding the depression symptoms at times 1 and 2. For example, suppose participants’ depression symptoms at time 1 (⁠$Y1$⁠) may not only affect their depression symptoms at time 2 (⁠$Y2$⁠), but they are simultaneously correlated due to unmeasured common causes $U$⁠, such as limited access to medical care.⁷ Another possible reason why $Y1$ and $Y2$ are (auto)correlated is that they are instantiations of the same latent process at two different times. But now, suppose that within the same wave at time 1, participants’ depression symptoms (⁠$Y1$⁠) make them less likely to be engaging conversation partners, which contributes to their risks of being treated as invisible (⁠$X1$⁠). This causal directionality – justifiable using theoretical knowledge and by measuring $Y1$ before $X1$ within the same wave – is indicated by the arrow from $Y1$ to $X1$⁠. Furthermore, suppose that this causal effect of $Y1$ and $X1$ is unconfounded (so that the hidden common cause $D$ can be ruled out). Under this scenario, adjusting for the previous measurement $Y1$ is necessary for valid causal inference.

Adjusting for a previous outcome measurement can be counterproductive and undermine causal conclusions when only a few minor alterations in the data-generating process are made. We illustrate this point using different scenarios in Figure 2.

In Figure 2(a), within the same wave at time 1, experiencing invisibility (⁠$X1$⁠) is correlated with participants’ depression symptoms (⁠$Y1$⁠) due to a hidden common cause $D$⁠, as in Figure 1(a). Here, we assume that an autoregressive effect of $Y1$ on $Y2$ can be precluded, as indicated by the absence of a $Y1→Y2$ arrow. Participants’ depression symptoms at time 1 (⁠$Y1$⁠) and at time 2 (⁠$Y2$⁠) are correlated merely due to unmeasured common cause(s) $U$⁠, such as limited access to medical care. Under this scenario, adjusting only for $C$ suffices to block all non-causal paths linking $X1$ and $Y2$⁠. Crucially, adjusting for the previous measurement $Y1$ is counterproductive: doing so opens a non-causal path (⁠$X1←D→Y1←U→Y2$⁠) and induces “collider (stratification) bias” (Cole et al., 2009; Elwert & Winship, 2014; Greenland, 2003; Griffith et al., 2020). Therefore, the previous outcome measurement $Y1$ should not be adjusted for (Foster, 2010; Morgan & Winship, 2015).

We now turn to a different scenario. In Figure 2(b), within the same wave at time 1, experiencing invisibility (⁠$X1$⁠) can now be justified to contribute to participants’ depression symptoms $(Y1$⁠), which in turn influences their depression symptoms at time 2 (⁠$Y2$⁠). The depression symptoms at both time points also share unmeasured common causes $U$⁠, such as limited access to medical care. Under this scenario, adjusting for the previous measurement $Y1$ is counterproductive: doing so not only changes the causal effect of interest (to the direct effect that bypasses $Y1$⁠; Elashoff, 1969) but also induces collider bias in the estimator (Schisterman et al., 2009). For related discussions, see Ananth and Schisterman (2017), Montgomery, Nyhan, and Torres (2018) and Pearl (2016).

Using the scenarios presented in Figures 1 and 2, we have explained how it can be clear-cut whether a previous outcome measurement $Y1$ should be adjusted for (Figure 1) or not adjusted for (Figure 2) when estimating the effect of $X1$ on $Y2$⁠. But things are rarely so simple. The causal diagrams up to this point represent stringent causal assumptions valid only in specific circumstances: respectively, an unconfounded effect of $Y1$ on $Y2$ (Figure 1(a)); an unconfounded effect of $Y1$ on $X1$ (Figure 1(b)); no autoregressive effect of $Y1$ on $Y2$ (Figure 2(a)); and an unconfounded effect of $X1$ on $Y1$ (Figure 2(b)). Relaxing these causal assumptions – which are empirically untestable in practice – in a longitudinal design can quickly lead to an unavoidable conundrum. We will elaborate on this next.

In Figure 3(a), within the same wave at time 1, experiencing invisibility (⁠$X1$⁠) is correlated with participants’ depression symptoms (⁠$Y1$⁠), similar to Figures 1(a) and 2(a). Furthermore, we impose fewer assumptions about participants’ depression symptoms at the two time points. Specifically, not only are participants’ depression symptoms at time 1 $(Y1$⁠) allowed to have an autoregressive effect on their depression symptoms at time 2 (⁠$Y2$⁠), but they are also allowed to simultaneously be autocorrelated due to unmeasured common causes $U$⁠, such as limited access to medical care – as in Figures 1(b) and 2(b). Under this scenario in Figure 3(a), $Y1$ adopts two conflicting roles simultaneously: It is a non-collider on one path (⁠$X1←D→Y1→Y2$⁠) and a collider on another path (⁠$X1←D→Y1←U→Y2$⁠).⁸ Hence, not adjusting for $Y1$ induces confounding bias, but adjusting for $Y1$ induces collider bias. Simply put, consistent estimation of the causal effect of $X1$ and $Y2$ requires simultaneously adjusting for $Y1$ and not adjusting for $Y1$ (this conundrum is discussed in, e.g., Pearl & Robins, 1995).⁹

This conundrum is not unique to Figure 3(a). In Figure 3(b), suppose within the same wave at time 1, experiencing invisibility (⁠$X1$⁠) can be justified to not only contribute to participants’ depression symptoms $(Y1$⁠) – as in Figure 2(b) – but they also share a hidden common cause – as in Figures 1(a) and 2(a). These causal relations are indicated by the $X1→Y1$ and $X1←D→Y1$ paths, respectively. Hence, for the same reasons as in Figure 2(b), $Y1$ should not be adjusted for. But $Y1$ is a non-collider on one path (⁠$X1←D→Y1→Y2$⁠) and should be adjusted for. Therefore, $Y1$ must simultaneously be adjusted for and not be adjusted for to avoid biases when estimating the total effect of $X1$ on $Y2$⁠.

Under this minimal scenario with just two waves, we explained why simple advice to adjust or not for a previous outcome measurement is credible only under strict assumptions about the data-generating process. Following a routine rule-of-thumb to either adjust or not adjust for a previous measurement can easily lead to inconsistent estimators in other scenarios. Critically, previous measurements can adopt conflicting roles simultaneously on different non-causal paths in longitudinal designs, leading to unavoidable biases.

We focused on simple probative scenarios with just two time points to ease the exposition. But the core issues we have raised apply more generally (exacerbated with more time points and other time-varying measured covariates) that encompass the scenarios described in this paper. Please see Appendix B for an illustration using an example with three waves.

We carried out a Monte Carlo simulation study to empirically demonstrate the biases arising from inappropriate adjustment for previous outcome measurements.¹⁰ We generated data according to each of the six scenarios shown in Figures 1, 2, and 3. We used a sample size of 10,000 to demonstrate that these structural biases were not due to chance associations and persist even in large samples. For simplicity, we assumed no covariates in $C$⁠. We used lavaan (Rosseel, 2012) to fit two regression models: one regressing $Y2$ on $X1$ and $Y1$ (hence adjusting for $Y1$⁠)¹¹, and another regressing $Y2$ on $X1$ only (hence not adjusting for $Y1$⁠). Our focus was on the (total) effect of $X1$ on $Y2$⁠. Results for 100 datasets are displayed in Table 1.

As shown in Table 1, adjusting for $Y1$ yielded unbiased estimates only under the data-generating scenarios depicted in Figure 1. Under the scenarios in Figure 2, adjusting for $Y1$ yielded severely biased estimates: the estimate could be either smaller or larger than the true effect. Finally, when data was generated under less restrictive assumptions, such as those depicted in Figure 3, adjusting or not adjusting for the previous outcome measurement led to incorrect causal inferences one way or the other. Under such scenarios, the estimate not only deviated from the true effect in magnitude but had opposing signs.

Causal structures in longitudinal data analysis are perpetually unspecified or unexamined in practice. Yet, as we demonstrated in the preceding sections, this step is essential to drawing valid causal conclusions. Routine analytic approaches, such as adjusting for previous measurements, are predicated on stringent causal assumptions. They are, therefore, prone to possibly severe bias when these assumptions are – unbeknownst to the researchers – violated.

The routine advice to improve causality using observational data is to adjust for confounders and merely avoid mediators and colliders. But as we have demonstrated in this paper, in longitudinal settings¹², this advice is inadequate and potentially misleading. Instead, we encourage researchers to look beyond common causes of treatment and outcome when analyzing longitudinal data. For example, researchers should consider possible common causes of the repeated outcome measurements (such as $U$ in Figure 3).

We, therefore, cannot offer blanket advice on whether or not to adjust for previous measurements because there is no one-size-fits-all panacea. Researchers seeking to reap the benefits of longitudinal designs for drawing causal conclusions should carefully construct – preferably at the initial stages of a research project – a causal diagram that best represents theoretical knowledge and the underlying data-generating process. For example, recent tutorials offer concrete advice on how to construct and justify a causal diagram in practice (Barnard-Mayers et al., 2022; Digitale et al., 2022; Ferguson et al., 2019; Grosz et al., 2020; Tennant et al., 2020). As shown in this paper, causal diagrams are an excellent research tool that lets researchers “draw their assumptions before their conclusions” (Hernán & Robins, 2020). They are especially beneficial for guiding analytic choices and improving the understanding of possible sources of structural biases.

With a defensible postulated causal diagram in hand, further examining many non-causal paths and determining whether each variable should or should not be adjusted for – as we have done in this paper – can be challenging and seemingly impossible in practice with many waves of data. Consequently, researchers may feel helpless and discouraged when faced with such a proposition. We encourage researchers to use the open-source and freely available DAGitty tool (Textor et al., 2017). The DAGitty tool facilitates the crucial task of clarifying and checking posited causal assumptions. To help researchers visualize their postulated causal diagrams using DAGitty, we have provided an example of Figure B1 at http://dagitty.net/mei2SaP. Researchers can use this as a starting point to modify and adapt the causal diagram for their unique substantive contexts. Crucially, researchers can then use DAGitty to automatically determine for a focal causal effect whether all non-causal associations can be eliminated by adjusting for a (minimal) subset of covariates. Researchers need not enumerate each path as we have done in this paper. This is achieved using the so-called “back-door criterion” that provides a set of sufficient graphical conditions for determining whether all non-causal paths (specifically “back-door” paths with an arrow pointing to treatment) linking treatment and outcome can be blocked by adjusting for a minimal set of variables (Pearl, 2009, chap. 3).¹³ Continuing our example in Figure B1, a minimal adjustment set for the effect of $X1$ on $Y2$ is ${C,U,Y1}$⁠; whereas a minimal adjustment set for the effect of $X2$ on $Y3$ is ${C,U,X1,Y1,Y2}$⁠.

We recommend that a best practice is to submit the posited causal diagram (e.g., using DAGitty), and selected covariates for confounding adjustment, for peer review as part of a Stage 1 Registered Report submission (Kiyonaga & Scimeca, 2019). This practice utilizes the collective substantive expert knowledge of editors and reviewers to fortify the defensibility of the postulated causal structure and the adequacy of the selected (and omitted) confounders. Using a carefully constructed and rigorously justified causal diagram that clearly and honestly explicates the causal assumptions – before data collection and analysis – can foster more principled causal inferences (Shpitser et al., 2021).

Finally, there are further complications in longitudinal data analysis we did not detail in this article. In psychology research, treatments are often time-varying: e.g., a person may experience invisibility at time 1 but not at time 2. Longitudinal confounders are similarly bound to be affected by earlier treatments. Treatment-dependent (variously termed post-treatment, time-varying, or treatment-induced) confounding poses severe threats to valid causal inferences of the effects of a time-varying treatment (Daniel et al., 2012; Thoemmes & Ong, 2015). Conventional estimation methods, such as a single regression model for the outcome given all treatments and covariates, cannot avoid undue adjustment for measured post-treatment confounders that induce spurious associations (Rosenbaum, 1984). In the presence of measured time-varying confounding, we recommend researchers utilize the well-established “g-methods” framework (where the “g” stands for “generalized”). G-methods have been recently introduced to the psychology literature (Loh & Ren, 2023a, 2023b, 2023c). This broad class of methods pioneered by James Robins has deep roots in causal inference research and is widely used in (bio)statistics, epidemiology, and medical sciences to assess time-varying treatment effects in longitudinal data when treatment-dependent confounding is present (Clare et al., 2018; Wijn et al., 2022).

Psychology researchers commonly use longitudinal data to answer causal questions. A widely adopted analytic approach is adjusting for previous measurements. But valid causal conclusions rely on stringent causal assumptions routinely unexamined and overlooked in practice. So should previous measurements be adjusted for? We demonstrate in this paper that the answer is nuanced and far from clear-cut. The advice of simply adjusting for common causes and merely avoiding adjusting for mediators and colliders must be revised, especially in longitudinal data. In particular, we highlight a conundrum in longitudinal designs: the routine analytic practice of adjusting for previous measurements can simultaneously eliminate and introduce non-causal associations, inadvertently leading to an inability to draw valid causal conclusions. We encourage researchers to make informed analytic decisions by conducting thoughtful examination and deliberate reflection of the causal assumptions. With this article, we hope to contribute to ongoing conversations on strengthening causal inferences from longitudinal data in psychological science.

We have no conflicts of interest to disclose.

Contributed to conception: WWL, DR Drafted and/or revised the article: WWL, DR Approved the submitted version for publication: WWL, DR

Wen Wei Loh was partially supported by the University Research Committee Regular Award of Emory University, Atlanta, GA.

Causal diagrams, also known as graphical causal models or causal Directed Acyclic Graphs (DAGs), are widely used to represent theorized causal relations and to establish a set of graphical rules sufficient for drawing valid causal inferences. This framework has been extensively introduced and explained elsewhere in the behavioral, health, and social sciences literature; see, e.g., Digitale et al. (2022), Elwert (2013), C. Glymour (2001), M. M. Glymour (2006), Grosz, Rohrer, and Thoemmes (2020), Hernán and Robins (2020), Lee (2012), Moerkerke, Loeys, and Vansteelandt (2015, Figure 2), Morgan and Winship (2015), Pearl (2012), Pearl, Glymour M., and Jewell (2016), and Rohrer (2018). In Table A1, we summarize basic graph-theoretic language relevant to discussing the issues raised in this article. In this paper, we will assume that a causal diagram can be substantively and defensibly justified as accurately representing the underlying data-generating processes based on established theoretical knowledge and rigorous experimental evidence. Researchers should further exploit information from the measurement of the variables in their study design to support the posited causal structure.¹⁴ For example, the timings can be used to establish temporal precedence and rule out reverse causation based on temporal-logical constraints, and the spacings can be optimized to allow sufficient time for the causal effects to manifest (Cinelli et al., 2022; Deffner et al., 2022; Tate, 2015; Vowels, 2023).

In this section, we explain how the unavoidable biases described above can quickly be exacerbated in longitudinal studies with repeatedly measured treatments and outcomes. To illustrate, we use a slightly more complex example with longitudinal data collected across three waves. Suppose that at time 1, a non-randomized treatment (⁠$X1$⁠), an initial measurement of the outcome (⁠$Y1$⁠), and baseline time-invariant covariates (⁠$C$⁠) are recorded. At time 2, the non-randomized treatment (⁠$X2$⁠) and the outcome (⁠$Y2$⁠) are recorded. At time 3, the outcome (⁠$Y3$⁠) is recorded. A causal diagram corresponding to such a setting is shown in Figure B1.

To simplify elucidating the challenges, we will focus on the lag one causal effects of treatment (e.g., $Xt$⁠) on the outcome at the next wave (e.g., $Yt+1$⁠, for $t=1,2$⁠). In the main text, we discussed the challenges of adjusting for the earlier outcome measurement $Y1$ when estimating the effect of a single treatment $X1$ on the later outcome $Y2$ in a setting with two waves. We now explain how these challenges escalate when the treatment is repeatedly measured by focusing on the effect of the intervening treatment (⁠$X2$⁠) on the final outcome (⁠$Y3$⁠).

First, note that the same arguments in the previous section can be applied here, with ${C,X2,Y2,Y3}$ taking the place of ${C,X1,Y1,Y2}$ in Figure 3(a), to realize that $Y2$ must simultaneously be adjusted and not adjusted for to block all non-causal paths linking $X2$ and $Y3$⁠. Therefore, we will inspect whether the initial treatment and outcome measurements (⁠$X1$ and $Y1$⁠) should or should not be adjusted for. Non-causal paths linking $X2$ and $Y3$ via either $X1$ or $Y1$ are displayed in Table B1. On any given path, $X1$ can be either a non-collider or a collider intersecting it; similarly, $Y1$ can be either a non-collider or a collider on a path intersecting it. In other words, $X1,Y1$⁠, and $Y2$ must each be simultaneously adjusted for and not adjusted for when targeting the effect of $X2$ on $Y3$⁠. In causal diagrammatic terminology, there is no subset of the measured variables ${C,X1,Y1,Y2}$ that, when adjusted for, suffices to block all non-causal paths linking $X2$ and $Y3$⁠. Therefore, the causal structure in Figure B1 rules out consistent and unbiased estimation of the $X2→Y3$ effect.