Governments allocate a considerable portion of their budget to support basic research in various disciplines. Several European countries, following the US tradition, assign research funds through competitive grants like ANR grants in France and SNSF in Switzerland. Despite these significant investments in research, the evidence on their impact on scientific output or researchers’ careers is generally limited. This limitation could be due to the lack of data or assignment mechanisms that are suitable to pin down causal effects.
In a recent IZA Discussion Paper, Corinna Ghirelli, Enkelejda Havari, Elena Meroni, and Stefano Verzillo provide novel empirical evidence on the long-term effects of winning a European Research Council (ERC) grant. The main goal of the ERC, founded in 2007, is to stimulate scientific excellence by funding the very best creative researchers of any nationality and age to run projects in any EU country or associated member countries. ERC grantees have won prestigious international awards, including seven Nobel Prizes, four Fields Medals, and five Wolf Prizes. Furthermore, they have published 6,100 articles in top journals.
To find out whether obtaining an ERC grant has an impact on winners’ productivity, the authors used data on all ERC applicants from 2007 to 2013 and combined it with data from the Scopus database to track their publication records up to nine years after obtaining the grant. They also leveraged the fact that each year, applicants are evaluated and ranked by a panel of experts, depending on the type of grant (Starting or Advanced). A cut-off applies to the ranking list based on the budget available, and only the highest-ranked proposals are offered the grant until the call’s budget has been exhausted. This approach allowed the authors to firstly use a regression discontinuity design (RDD) to estimate the impact of an ERC grant by comparing the outcomes of winning and non-winning applicants around the cut-off.
Productivity effects of ERC grants vary by field
Using RDD, the authors find that overall, obtaining an ERC grant does not significantly improve researcher productivity, except for Physics in the Starting grants scheme, but it increases the probability of receiving other EU grants. This phenomenon is known as the “Matthew effect” in the literature.
Given that RDDs help identify a local effect (only observations around the cut-off point are used), the authors conducted a difference-in-differences (DID) analysis using the time series of bibliometric indicators available and widened the population of reference in this way. Using DID, they found that Advanced and Starting Grants increase research productivity and help applicants receive other EU grants in the nine years after grant assignment. These positive long-term effects on productivity and excellence hold for the fields of Chemistry, Universe and Earth Sciences, Institutions and Behaviors, Human Mind Studies, and Medicine.
Regarding the ability to obtain other EU funds, strong evidence in favor of the Matthew effect is found for winners in all disciplines and types of grants.
Finally, the authors split the ERC winners’ group into bottom-rank and top-rank and compared them with the non-winners to find that the results are positive and significant only for the second group. This result means that top-ranked winners are the ones driving the positive outcomes.
Selection mechanism may not be optimal for applicants at the margin
The findings of the study confirm that receiving ERC funds significantly improves standard bibliometric outcomes, such as scientific productivity, impact, and research funding, for top-ranked winners. However, the authors only found suggestive evidence of such an effect for winners who score near the funding threshold, with positive but not statistically significant coefficients, if any.
These results have practical implications for improving the management of ERC funds to increase their effectiveness as policy instruments. One possible solution could be to improve the selection process of the ERC project, which can be expensive. While this approach may be justified for top-ranked winners, it may not be necessary for those close to the threshold score since the authors found no evidence of a positive and significant effect of obtaining ERC funds for these applicants.
As a result, the standard selection mechanism based on peer-review may not be optimal for applicants with scores slightly above and below the funding threshold. The results of recently launched and ongoing projects that focus on partial-randomization of research funding may provide insights into the potential benefits and disadvantages of these new selection practices. This could help us better understand if this approach could be a way forward, at least for those applicants at the margin.