Code
install.packages("tableone")
install.packages("MatchIt")
install.packages("survey")Day 1 Practical: Potential Outcomes
Part 2 Bonus Exercises: Estimating Causal Effects
In this practical, we’ll get to grips with the basics of estimating causal effects using stratification, matching and propensity scores in a multivariate setting. We will need a number of R packages in these exercises. We assume you have completed or are familiar with the content of the univariate adjustment practical from today.
We will use a number of R packages in this practical, which you can install as follows
In this practical we will get some practice using some tools for causal effect estimation which have been inspired by the potential outcomes framework. To do this we will use data from Schafer and Kang (2008). We will reproduce only a small part of their analyses, but you can read up on even more ways to estimate causal effects from nonrandomized studies in their paper.
Schafer, J. L., and Kang, J. (2008). Average causal effects from nonrandomized studies: A practical guide and simulated example. Psychological methods, 13, 279-313. DOI: 10.1037/a0014268
Data and Research Question
The data are found in the datafile called SchaferKangData.dat which you can download from here. The dataset represents a (simulated) two-wave observational study on emotional distress and dieting behaviour in adolescent girls. Our research question is: What is the effect of dieting on emotional distress? We will apply several different methods we learned about in the lecture to answer this question. For null hypothesis tests, you can use a significance level of 0.05 throughout.
Checking Covariate Balance
The “naive” difference between those who did and did not diet would only be a valid estimate of the causal effect only under the assumption of exchangeability (i.e. no confounding). We know already that this is not a reasonable assumption based on the design of the study from which this data originates.
If the treatment assignment was randomized, then we would not expect to see systematic differences between the control and treated groups on our observed covariates. Observing such differences (for covariates that we believe to be potential confounders or causes/effect modifiers of the outcome) is a cause for concern.
Answers
We can check this in a number of ways. For instance, by plotting this histograms of the observed variables across dieting groups, for example:
Code
p1 <- hist(df[,"SLFWGHT"][df$DIET == 0], plot = F)
p2 <- hist(df[,"SLFWGHT"][df$DIET == 1], plot = F)
plot( p1, col=rgb(0,0,1,1/4), xlim=c(1,5), main = "Covariate distributions across groups",
xlab = "Self-rating of Weight") # first histogram
plot( p2, col=rgb(1,0,0,1/4), xlim=c(1,5), add=T) # second
abline(v = mean(df[,"SLFWGHT"][df$DIET == 0]), col = "blue")
abline(v = mean(df[,"SLFWGHT"][df$DIET == 1]), col = "red")
Alternatively, you can compute standardized mean differences (SMDs) between the two groups on any covariate(s) of interest. In epidemiology and health sciences, researchers often present a table of such SMD values, along with the means in each group of interest, as the first table of the results section, and so this is sometimes referred to as table 1. Let’s use the function CreateTableOne() from the package tableone to check this. Often, researchers use a rule of thumb that considers an SMD bigger than .1 to represent a worrying degree of imbalance between the covariate distributions. While we do not endorse these kinds of arbitrary cut-off values in general, it is common in the literature to see this or other values used as a reference.
Code
table1 <- tableone::CreateTableOne(vars=c("DISTR.1","BLACK", "NBHISP", "GRADE",
"SLFHLTH", "SLFWGHT", "WORKHARD", "GOODQUAL",
"PHYSFIT", "PROUD", "LIKESLF", "ACCEPTED",
"FEELLOVD"), strata="DIET", data=df,
test=FALSE)
print(table1, smd=TRUE) Stratified by DIET
0 1 SMD
n 4780 1220
DISTR.1 (mean (SD)) 0.62 (0.42) 0.71 (0.45) 0.214
BLACK (mean (SD)) 0.26 (0.44) 0.17 (0.38) 0.197
NBHISP (mean (SD)) 0.15 (0.35) 0.15 (0.36) 0.021
GRADE (mean (SD)) 9.16 (1.39) 9.37 (1.34) 0.152
SLFHLTH (mean (SD)) 2.20 (0.93) 2.35 (0.91) 0.171
SLFWGHT (mean (SD)) 3.19 (0.76) 3.84 (0.70) 0.895
WORKHARD (mean (SD)) 2.14 (0.91) 2.05 (0.85) 0.093
GOODQUAL (mean (SD)) 1.80 (0.67) 1.84 (0.71) 0.055
PHYSFIT (mean (SD)) 2.24 (0.93) 2.53 (0.93) 0.320
PROUD (mean (SD)) 1.76 (0.77) 1.86 (0.79) 0.126
LIKESLF (mean (SD)) 2.09 (0.99) 2.52 (1.06) 0.420
ACCEPTED (mean (SD)) 2.14 (1.00) 2.35 (1.06) 0.207
FEELLOVD (mean (SD)) 1.78 (0.83) 1.93 (0.90) 0.172Let’s take this result as an indication that the covariate distributions may be unbalanced across treatment groups.
Conditional Exchangeablity, Covariate Adjustment, and Propensity Scores
As we have seen, in observational settings the assumption of exchangeability is unlikely to hold. However, if we have gathered sufficient information on potential confounders, then we may be able to achieve conditional exchangeability (exchangeability within levels of the covariates). As such, to obtain an unbiased estimate of the causal effect, we must appropriately adjust for group differences in the covariates of interest.
Throughout the first three days of the course, we will be looking at a number of different methods for covariate adjustment. In the second practical you have in fact already practiced one type of covariate adjustment, known as stratification; in that context, we had a single covariate we wished to adjust for, and we did so by comparing treated and untreated individuals in each strata (i.e. level) of the covariate. Stratification, as a general strategy, becomes more difficult when we have many covariates, those covariates have many levels, and we have a relatively small sample size.
In this part of the practical we will introduce you to four concepts which are relevant for covariate adjustment:
- Matching, in which we try to create pairs of a treated and an untreated individual that have the same covariate values
- Propensity scores, a tool for summarizing information about multivariate confounding
- (Inverse Probability) Weighting, in which we use propensity scores to create a pseudo-population that is balanced on the covariates
Matching
Matching as a method of covariate adjustment is conceptually similiar to idea of stratification. The basic approach of exact matching is to search for people in both treatment groups that have the same score on all of the covariates you wish to adjust for (i.e., that have the same profile of covariate values). This can be thought of as an extreme version of stratification; in principle every possible combination of covariate values is a new strata. After matching, the matched treated and control groups are guaranteed to be balanced on their covariate values. A key downside of this adjustment approach is that we only include strata in our analysis for which there is at least one person in both groups that take on that combination of values. All other strata (i.e., all individuals who do not have an “exact” match in the other group) are dropped from the analysis.
To perform matching, we must first play close attention to what kind of causal effect we wish to estimate. When estimating the ATE we look for an eligible match for all individuals in the study.
Propensity Scores
In the lecture we introduced the concept of a propensity score, the probability of receiving treatment given the value of a set of covariates. Propensity scores are convenient tools to summarize information on a range of covariates, and they can be used to adjust for confounding in many different ways. Estimating a propensity score is relatively straightforward. In these exercises you will see how this can be done, and what important considerations are for a propensity score model.
You may notice that the syntax we used to fit the propensity score model is very similar to that we used with matchit(). In fact, behind the scenes, for certain specifications, matchit() uses propensity scores for matching; first, it fits a propensity score model using the matching covariates specified, then it matches on the propensity score not on the variable itself. So, you already have some experience with using propensity scores for covaraite adjustment with matching! We can even plot the distribution of propensity scores amongst matched and unmatched individuals:
Code
plot(match_nearest, type = "hist")
Recall that the PS is a sort of summary of multivariate information. From these histograms we can see that, although the distributions of covariates are different in the observed groups, after matching, the covariate distribution amongst the controls has been shifted onto the distribution of covariates amongst the treated.
Important to note here is that we must choose the PS model we wish to fit. In this situation we took a simple choice: modeling linear effects of all covariates. We may also choose to include, for instance, quadratic terms, interaction terms, or other non-linear effects of covariates. This is a disadvantage of PS methods: We must correctly model the treatment assignment model. The more data we have, the more flexibly we can model the PS!
For now, let’s see how else we can use propensity score models to estimate causal effects.
Inverse Probability Weighting
Inverse Propensity Weighting (IPW) or Inverse Probability of Treatment Weighting (IPTW) is a popular and flexible method of using propensity scores to account for unbalanced covariates. As with matching, we can use different variations of IPTW to estimate different causal effects (i.e., the ATE, ATT or ATC).
To estimate the ATE, the estimated propensity scores \(\hat{\pi}_i\) are used to determine the probability that an individual would have received the treatment that they in fact received:
for the treated (\(X_i=1\)) this is simply \(\hat{\pi}_i\)
for the non-treated (\(X_i=0\)) this is \(1-\hat{\pi}_i\).
Next, we can use these probabilities as weights, by taking their inverse: That way, a case that received a treatment that she was very likely to receive, will get a small weight, while a case that received a treatment that she was very unlikely to receive, will get a large weight. Thus, the inverse probability weight indicates the number of persons from the population that this person represents. For the treated, this weight is \(1/\hat{\pi}_i\); for the untreated, it is \(1/(1-\hat{\pi}_i)\). For this reason we refer to IPW as creating a “pseudo-population”.
We can compute our estimate of the IPTW-weighted ACE by using techniques which allow us to include weights directly. Typically, most glm() functions in R have a weights option. For this practical, we are going to use tools from the survey package.
Conclusion
In this practical we have examined two popular methods for covariate adjustment: Matching, and Inverse Probability of Treatment Weights. We considered matching directly on covariates, and matching using propensity scores. The goal of these techniques is to create two groups which are balanced on the values of relevant confounders.
In matching, this is done by creating pairs of a treated and an untreated person who have (almost) identical (propensity or covariate) scores, resulting in a smaller but balanced dataset. Matching is a form of conditioning on confounding variables. In inverse probability weighting, this is instead done by weighing each person’s observation by their inverse probability of received treatment, thereby creating a balanced pseudo-population. Notably, while this technique also adjusts for confounders, it does not condition on them. This distinction becomes important in more complex scenarios which we will see later in the week. When we use either approach, we should check whether it balances the covariates, for instance, by considering the standardized mean differences (for other options, see for instance: Austin, P. c. (2009). Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples. Statistics in Medicine, 28, 3093-3107. https://doi.org/10.1002/sim.3697).
Many other options exist for using propensity scores to mimic RCTs, with many of these surveyed and compared in Schafer & Kang (2008). We will continue to explore different methods of adjustment throughout the rest of the week. Each adjustment method has its own advantages and disadvantages, and rely on their own asssumptions.
It is important to always keep in mind that all the techniques discussed here were developed under the assumption of no unobserved confounding. This means that all relevant confounders should be included as covariates. Sensitivity analysis is a valuable approach to investigate how the effect of unobserved confounding can be further investigated (not covered in this course; for more information see for instance: Blackwell, M. (2013). A selection bias approach to sensitivity analysis for causal effects. Political Analysis, 22, 169-182. https://doi.org/10.1093/pan/mpt006).