What are the impacts of Cochrane Systematic Reviews? An evaluation of the impact of the outputs of Cochrane Review Groups supported by NIHR

Abstract
Background
The last few decades have seen a growing emphasis on evidence-informed decision-making in health care. Systematic reviews, such as those produced by Cochrane, have been a key component of this movement. The National Institute for Health Research (NIHR) Systematic Review Programme currently supports 20 Cochrane Review Groups (CRGs) in the UK and it is important that this funding represents value for money.
Aims and objectives
The overall aim was to identify the impacts and likely impacts on health care, patient outcomes and value for money of Cochrane Reviews published by 20 NIHR-funded CRGs during the years 2007–11.
Design
We sent questionnaires to CRGs and review authors, undertook interviews with guideline developers (GDs) and used bibliometrics and documentary review to get an overview of CRG impact and to evaluate the impact of a sample of 60 Cochrane Reviews. The evaluation was guided by a framework with four categories (knowledge production, research targeting, informing policy development and impact on practice/services).
Results
A total of 3187 new and updated reviews were published on the Cochrane Database of Systematic Reviews between 2007 and 2011, 1502 (47%) of which were produced by the 20 CRGs funded by the NIHR. We found 40 examples where reviews appeared to have influenced primary research and reviews had contributed to the creation of new knowledge and stimulated debate. Twenty-seven of the 60 reviews had 100 or more citations in Google Scholar™ (Google, CA, USA). Overall, 483 systematic reviews had been cited in 247 sets of guidance. This included 62 sets of international guidance, 175 sets of national guidance (87 from the UK) and 10 examples of local guidance. Evidence from the interviews suggested that Cochrane Reviews often play an instrumental role in informing guidance, although reviews being a poor fit with guideline scope or methods, reviews being out of date and a lack of communication between CRGs and GDs were barriers to their use. Cochrane Reviews appeared to have led to a number of benefits to the health service including safer or more appropriate use of medication or other health technologies or the identification of new effective drugs or treatments. However, whether or not these changes were directly as a result of the Cochrane Review and not the result of subsequent clinical guidance was difficult to judge. Potential benefits of Cochrane Reviews included economic benefits through budget savings or the release of funds, improvements in clinical quality, the reduction in the use of unproven or unnecessary procedures and improvements in patient and carer experiences.
Conclusions
This study identified a number of impacts and likely impacts of Cochrane Reviews. The clearest impacts of Cochrane Reviews are on research targeting and health-care policy, with less evidence of a direct impact on clinical practice and the organisation and delivery of NHS services. Although it is important for researchers to consider how they might increase the influence of their work, such impacts are difficult to measure. More work is required to develop suitable methods for defining and quantifying the impact of research.
Funding
The NIHR Health Technology Assessment programme

Cochrane produces systematic reviews evaluating the effectiveness of health-care interventions. These reviews are produced by Cochrane Review Groups (CRGs) and published on The Cochrane Library. Twenty CRGs are based in the UK and receive funding from the National Institute for Health Research. The aim of our study was to assess the value of Cochrane Reviews produced by those groups by looking at the way they have been used by health-service providers, policy-makers, researchers and service users. For example, have they been used in developing guidance, such as that produced by the National Institute for Health and Care Excellence, or have they identified gaps in the evidence and led to new research? We used a mixture of methods: we sent questionnaires to CRGs and review authors, interviewed guideline developers, reviewed existing documents and looked at the number of times the reviews had been cited in other research papers or guidelines. We found evidence that policy-makers use Cochrane Reviews to inform clinical guidance both in the UK and around the world. However, policy-makers found reviews less helpful if they were out of date or if they had asked too narrow a question. There is evidence that Cochrane Reviews identify areas for new research and provide an important source of knowledge about the value of different treatments. Although there is some evidence that they influence the behaviour of health-care providers, and lead to benefits for the health service, these are more difficult to prove.