Analyzing public health interventions undertaken a hundred (or more) years ago in Western countries can be important for understanding whether similar interventions in developing countries, if pursued today, would be effective. Given that restricting access to public health interventions from randomly chosen residents of a developing country would obviously be unethical, the analysis of historical data provides policy makers with credible estimates that would otherwise be difficult to obtain.
A 2005 paper by David M. Cutler and Grant Miller is an example of a highly influential study that used historical data to shed light on the effectiveness of public health interventions that are still used today. Specifically, these authors estimated the impact of clean water technologies – filtration and chlorination – on mortality in US cities at the turn of the 20th century. Two recent IZA discussion papers follow up on the Cutler and Miller study, providing new results and highlighting the importance of revisiting old questions with better data.
The original study sought to analyze whether clean water technologies explained the dramatic decline in mortality from 1900 to 1936. It was published in Demography, a leading journal for public health experts. Cutler and Miller found that water filtration and chlorination were responsible for 41 percent of the observed reduction in overall mortality during this period, and 59 percent of the observed reduction in child mortality.
Huge social rate of return?
The striking magnitudes of these estimates suggest, even under conservative assumptions, that there was a huge social rate of return to adopting clean water technologies. Cutler and Miller estimated that every dollar invested in water filtration would lead to 23 dollars of benefits in the form of improved health. The Cutler and Miller article has been, according to Google Scholar, cited more than 840 times, not only in academic journals but also in policy briefs and opinion pieces.
Accurately assessing the social rates of return to public policy interventions is of obvious importance. Practitioners and policymakers, especially in developing countries, face tight budget constraints and are under tremendous pressure to get the biggest bang for their buck. Different rates of social returns imply different mixes of optimal policy portfolios.
In a recent IZA discussion paper, IZA Fellows D. Mark Anderson (Montana State, NBER), Daniel I. Rees (UC Denver) and their co-author Kerwin Kofi Charles (U Chicago, NBER) examined the effects of several public health interventions that often accompanied the introduction of water filtration but have received much less attention in the literature. These interventions included the treatment of sewage before its discharge into lakes and rivers, requirements that milk meet strict bacteriological standards, and requirements that milk come from tuberculin-tested cows.
Unclear effects on infant mortality
Anderson notes that, because exclusive breastfeeding was not the norm in first decades of the 20th century, public health experts viewed cleaning up the milk supply as critically important. “Yet,” according to Anderson, “we know next to nothing about its effectiveness. Did cleaning up the milk supply reduce infant mortality? We were really curious.”
Their results were surprising. Consistent with Cutler and Miller, they found that filtering the municipal water supply sharply reduced mortality from typhoid. More broadly, however, they found little evidence in support of the argument that public health interventions drove the extraordinary reductions in infant and total mortality after 1900.
But why were Anderson, Charles and Rees’s results so different from Cutler and Miller’s? It turns out that there was a series of transcription errors in the data on infant mortality used by Cutler and Miller (twenty percent of their infant mortality counts were incorrectly transcribed). Correcting these errors reduced the estimated effect of water filtration on infant mortality by two-thirds. In addition, Anderson, Charles and Rees raised doubts about the method used to construct total death rates in cases where municipal population data was missing. Results on total mortality turned out to be very sensitive to the choice of method.
Productive exchange between the authors
Approaching the original authors with these new results sparked a vital and productive exchange. A first comment by Cutler and Miller on the new results can be found here. A rejoinder to this comment is now available as an IZA discussion paper. According to Anderson, “Cutler and Miller have been extraordinarily gracious and professional. Without their help and original data, we would not have been able to identify why our results were so different from theirs.”
This exchange between researchers highlights the importance of replicating existing, influential studies both from a policy perspective and as a cautionary note for future researchers. It also shows the importance of cooperation between the original and replicating authors. Without a willingness to share code and data, progress on these important questions would be much slower if not stalled.
Replications play an important role in building a robust base of empirical evidence. A previous IZA Newsroom post compared the dearth of replication studies to the “tragedy of the commons”: there is wide agreement that replications are useful, but most people count on others to conduct them. New incentives – through better publication possibilities (see e.g. IREE, the new International Journal for Re-Views in Empirical Economics) or specific funding supporting this type of research – have to be provided to raise the intrinsic value of replications, especially for early career researchers.
More on replication in social sciences in the IZA Newsroom: