Jason Abaluck writes me about masks and the Bangladesh RCT study
This is all him, no double indent though:
“As a regular reader of your blog and one of the PIs of the Bangladesh Mask RCT (now in press at Science), I was surprised to see your claim that, “With more data transparency, it does not seem to be holding up very well”:
- The article you linked claims, in agreement with our study, that our intervention led to a roughly 10% reduction in symptomatic seropositivity (going from 12% to 41% of the population masked). Taking this estimate at face value, going from no one masked to everyone masked would imply a considerably larger effect. Additionally:
-
- We see a similar – but more precisely estimated – proportionate reduction in Covid symptoms [95% CI: 7-17%] (pre-registered), corresponding to ~1,500 individuals with Covid symptoms prevented
- We see larger proportionate drops in symptomatic seropositivity and Covid in villages where mask-use increased by more (not pre-registered), with the effect size roughly matching our main result
The naïve linear IV estimate would be a 33% reduction in Covid from universal masking. People underwhelmed by the absolute number of cases prevented need to ask, what did you expect if masks are as effective as the observational literature suggests? I see our results as on the low end of these estimates, and this is precisely what we powered the study to detect.
- Let’s distinguish between:
-
- The absolute reduction in raw consenting symptomatic seropositives (20 cases prevented)
- The absolute reduction in the proportion of consenting symptomatic seropositives (0.08 percentage points, or 105 cases prevented)
- The relative reduction in the proportion of consenting symptomatic seropositives (9.5% in cases)
Ben Recht advocates analyzing a) – the difference in means not controlling for population. This is not the specification we pre-registered, as it will have less power due to random fluctuations in population (and indeed, the difference in raw symptomatic seropositives overlooks the fact that the treatment population was larger – there are more people possibly ill!). Fixating on this specification in lieu of our pre-registered one (for which we powered the study) is reverse p-hacking.
RE: b) vs. c), we find a result of almost identical significance in a linear model, suggesting the same proportionate reduction if we divide the coefficient by the base rate. We believe the relative reduction in c) is more externally valid, as it is difficult to write down a structural pandemic model where masks lead to an absolute reduction in Covid regardless of the base rate (and the absolute number in b) is a function of the consent rate in our study).
- It is certainly true that survey response bias is a potential concern. We have repeatedly acknowledged this shortcoming of any real-world RCT evaluating masks (that respondents cannot be blinded). The direction of the bias is unclear — individuals might be more attuned to symptoms in the treatment group. We conduct many robustness checks in the paper. We have now obtained funding to replicate the entire study and collect blood spots from symptomatic and non-symptomatic individuals to partially mitigate this bias (we will still need to check for balance in blood consent rates with respect to observables, as we do in the current study).
- We do not say that surgical masks work better than cloth masks. What we say is that the evidence in favor of surgical masks is more robust. We find an effect on symptomatic seropositivity regardless of whether we drop or impute missing values for non-consenters, while the effect of cloth masks on symptomatic seropositivity depends on how we do this imputation. We find robust effects on symptoms for both types of masks.
I agree with you that our study identifies only the medium-term impact of our intervention, and there are critically important policy questions about the long-term equilibrium impact of masking, as well as how the costs and benefits scale for people of different ages and vaccination statuses.”