Trip Database Blog

Liberating the literature


systematic reviews

Are you a luddite?

Many will be familiar with my post A critique of the Cochrane Collaboration, it’s been the most viewed article ever published on this blog.  Continuing in the theme was Some additional thoughts on systematic reviews and more recently Evidence, hourglasses and uncertainty.

They all point to the current methods employed in systematic reviews (as exemplified by Cochrane) being a mess! As a type of summary, a few problems:

  • They can’t be relied upon to be accurate
  • They’re financially costly
  • They’re typically out of date
  •  Significant opportunity cost

More evidence that my perspective is correct is shown is the presentation Does access to clinical study reports from the European Medicines Agency reduce reporting bias? submitted to the Cochrane Colloquium in Vienna.  The conclusions:

Unpublished clinical study reports held by EMA may be a useful source to reduce outcome reporting bias.

It’s a testimony to Cochrane’s openness to allow such ‘dissent’ to be published.  It’s a dissenting view as the current Cochrane methods rely almost exclusively on published journal articles; unpublished clinical study reports – virtually unheard of. But that’s Cochrane; on one hand a business trying to maximise it’s business model but on the other a collection of individuals doing there best to improve methods.  As an outsider, I find this tension fascinating. Why? Because the best one can say for Cochrane’s methods (alongside most other SR producers) is that they are likely to produce ‘ball park’ accuracy – in most cases you simply cannot rely on the methods to produce results you can trust.  And this is where the tension comes, if they were being transparent, they would say ‘buy the Cochrane Library, most of the SRs are likely to be out of date and many are likely to be innaccurate’ which even I can see is not great for sales.

But this bring me nicely to diffusion of innovations – an area I studied for the PhD I never completed!  The Wikipedia article summarises it nicely by saying its “ a theory that seeks to explain how, why, and at what rate new ideas and technology spread through cultures.” The spread of an innovation is characterised as an S-shaped curve:

In relation to systematic review methods, where are you on this scale?  If, like me, you think there are serious problems by relying on published journal articles you’re probably at the innovator side of things.  If, however, you think current methods are great and there is no need to change you may well be a laggard.  But, the majority (those that are feeling uneasy) are likely to be in the middle.

Evidence, hourglasses and uncertainty

Long-term readers of this blog will know I struggle with many aspects of the systematic review process.  At the time of writing, my ‘A critique of the Cochrane Collaboration‘ has been viewed over 18,300 times and ‘Ultra-rapid reviews, first test results‘ nearly 10,000 times.

I believe the main justification given for conducting systematic reviews is to obtain a really accurate assessment of the effectiveness (or ‘worth’) of an intervention.  So, the thinking goes that spending 12-24 months is worth the cost (financial, opportunity, etc) due to the accuracy of the prediction it then gives.

My immediate response is that is demonstrably false. In my article ‘Some additional thoughts on systematic reviews‘ (just under 5,000 views) the evidence is clear that if you rely on published journal articles to ‘inform’ your systematic reviews (which is the case in the vast majority of systematic reviews) there is approximately a 50% chance that the effect size is likely to be out by over 10%.

But, even if we suspend being evidence-based and believe that systematic reviews can be relied upon to give us an accurate estimate of an effect size, is everything fine? I don’t think so and the image below illustrates my thinking.

It’s an hourglass!  At the top are all the unsynthesised trials, all floating around and the uncertainty is moderate.  Someone then spends 12-24 months pulling these together in a systematic review (likely of published trials and therefore ‘a bit dodgy’) and the certainty is reduced at the aperture of the hourglass.  But then, when you apply it to the real world of patient care, the uncertainty flares out again.  In the above example the intervention has a NNT of 6, so the intervention needs to be given to 6 people to obtain the desired outcome in 1 person.  Which is the 1 person?  Where’s the certainty?

If we were to spend significantly less time doing a review it might indicate a wider hourglass aperture (perhaps suggesting an NNT of 5-7).  In what situations does that matter?  I don’t think we’ve even started to explore these issues. In other words, when is it appropriate to spend 12-24 months on an systematic review and when is a significantly less resource intensive approach ‘ok’?

Is it irony that the reality is the type of review (systematic versus ‘rapid’) doesn’t alter the effectiveness of an intervention?  After all the compound remains the same, untroubled by the efforts of trialists.  Sorry, getting sociological there – must be time to sign off for now.

Evidence Live, systematic reviews and the US Air Force

I’m just back from the wonderful Evidence Live.  While I was away I saw this news story Is the West losing its edge on defence? and I was particularly drawn to the following passage:

The military have also contributed to their own misfortunes by conspiring with defence contractors to build ever more expensive weapons that can only be afforded in much smaller numbers than those they are supposed to replace.

Pierre Sprey, chief designer on the F-16 fighter noted the ruinous consequences of buying stealth aircraft at hundreds of millions of dollars a copy.

“It’s a triumph of the black arts of selling an airplane that doesn’t work,” he said.

This fits in very nicely with my perspective on systematic review methods, and was one of the main threads in my presentation on the future of evidence synthesis.  The current methods of systematic review production are costing way too much for what they deliver.  If you consider that the majority of systematic reviews rely on published trials they are inherently unreliable.

In the EBM world we’re buying F-16s…!

More to follow on this theme.

UPDATE: The wonderful Anne Marie Cunningham has pointed out (see comments) has pointed out that the consequence is of buying the really expensive stealth fighters (not F16s).  That’s a consequence of rushing a blog post so soon after a vigorous conference!  The point remains – purchasing too expensive planes has caused massive problems. 

A critique of the Cochrane Collaboration

What follows is a summary of a longer paper on some of the problems that the Cochrane Collaboration face.  It is based on the presentation I gave at Evidence Live 2013 entitled ‘Anarchism, Punk and EBM’. 

But to begin with I want to make it clear that I am fully supportive of systematic reviews and the reasons for doing them.  I also want to make it clear that this is not a criticism of the many thousands of volunteers who give their time freely to improve global healthcare.  I am in awe of their efforts.  My criticism is based on the fact that I feel that the current methods are unsustainable. 

Relevancy to clinical practice.

I have run a number of clinical question answering and between them have answered over 10,000 clinical questions.  It is very rare for a single systematic review to answer a question.  In an analysis of 358 dermatology questions only three could be answered by a single systematic review, so less than 1%.  Although we have only formally analysed dermatology there is little sense that many other areas do noticeably better, but there are some e.g. respiratory (but that would still answer fewer than 5% of the respiratory questions. In answering clinical questions I wish we had more systematic reviews that were useful for my work.  Should systematic reviews answer real questions?


On average a Cochrane systematic review takes 23 months from protocol to publication [1] and hundreds if not thousands of hours [2]. This causes problems with both production and subsequent updating of reviews.  Clearly, with a finite resource the longer a systematic review takes to produce the fewer you can do.

In 2009 only 39.8% of their systematic reviews were up to date (using Cochrane’s own definition of being updated within the past two years) while by 2012 it had dropped further to 35.8% [1]

These figures are slightly mis-leading as the number of systematic reviews has increased in that time.  In 2009 there were 3,958 active reviews and in 2012 that figure had risen to 4,905.  So, in 2009, of the 3,958 reviews, only 1,575 were up to date.  In 2012, of the 4,905 reviews, only 1,756 were up to date – an increase in up to date reviews of just 181 in three years.  Putting this another way, in 2009 there were 2,383 out of date systematic reviews and in 2012 this had risen to 3,149.
These figures are terrible and are made worse by the relatively recent increase in funding and spending Cochrane has enjoyed [3].

In the last seven years of financial figures the Cochrane Collaboration has spent in excess of £100 million and over the twenty years it has existed this is likely to be over £150 million – over a quarter of a billion US Dollars.  It is probably redundant to point out that this is a vast sum. [UPDATE: it has been pointed out that £150 million is actually not that much and could be seen as a pittance – I guess it depends on perspective].

As well as significant financial support Cochrane has the selfless support of 28,000 volunteers. Yet, the number of active systematic reviews is still modest.  This indicates that the current system is unsustainable and not fit for purpose.  The methodology, while reducing some bias, has resulted in a huge cost increase, not just financial but also opportunity cost. Ironically, the case of Tamiflu highlights that the methodology is flawed.


I do not wish to repeat the Tamiflu story here, for those interested there are numerous opportunities to find out more [4, 5].  In the latter reference, Tom Jefferson states:

“…I personally believe and my colleagues believe with me that Cochrane Reviews based on publications should really be a thing of the past…”

This is based on the fact that, when preparing the first Cochrane systematic review on neuraminidase inhibitors for preventing and treating influenza in healthy adults and children Tom and his team only relied on published journal articles [6].  This was subsequently found to miss large amounts of data, most of which was made available for the regulatory agencies e.g. EMA, FDA.  The updated, 2012, review [7] was a huge undertaking, even by Cochrane standards, but it was the only way Tom and his team felt they could obtain accurate estimations of the effect of neuraminidase inhibitors. 

But Tom is not alone in concerns about methodology, concerns with relying on aggregated trial data were made by Jack Cuzick, at Evidence Live 2013.  He made a general call for reviews to be based on individual patient data (IPD).

Both Tom and Jack feel that the current Cochrane methodology is not capable of making an accurate assessment of an interventions ‘worth’, albeit for different reasons.  The seriousness of this challenge should not be underestimated, it attacks at the very heart of the Cochrane Collaboration. 

Is there any hope?

In recent years there have been a number of articles that have suggested, to differing degrees, that doing things more quickly can give you the same or similar results to the Cochrane methodology.  I will highlight three:

1)    Can we rely on the best trial? A comparison of individual trials and systematic reviews [8].  In this paper the authors (including me) explored a random sample of Cochrane systematic reviews to see how often the largest randomised trial was in agreement with the subsequent meta-analysis.  This occurred in 81% of the meta-analyses examined and if the largest RCT was positive and significant it was around 95%.  In other words, using the largest RCT can give a broad hint as to the likely result of a subsequent meta-analysis.

2)    McMaster Premium LiteratUre Service (PLUS) performed well for identifying new studies for updated Cochrane reviews [9]. In this study the authors compared the performance of McMaster Premium LiteratUre Service (PLUS) and Clinical Queries (CQs) to that of the Cochrane Controlled Trials Register, MEDLINE, and EMBASE for locating studies added during an update of reviews. They concluded that PLUS included less than a quarter of the new studies in Cochrane updates, but most reviews appeared unaffected by the omission of these studies.  In other words, you do not necessarily need to get all articles to arrive at an accurate effect size (compared to the Cochrane systematic review).

3)    A pragmatic strategy for the review of clinical evidence [10].  In this paper the authors compared a research strategy based on the review of a selected number of core journals, with that derived by an SR in estimating the efficacy of treatments.  The authors concluded “We verified in a sample of SRs that the conclusion of a research strategy based on a pre-defined set of general and specialist medical journals is able to replicate almost all the clinical recommendations of a formal SR.”. Essentially, the same message as 2) above. 

The future

It is a very easy concept, the greater the cost (finance, time etc) of a systematic review the fewer systematic reviews within a fixed budget can be undertaken and kept updated.  Therefore, a major focus for Cochrane should be on reducing the cost per review.  Cochrane is full of incredibly talented people who appear to focus predominantly on reducing bias and random error.  This, to me, is a clear example of the laws of diminishing returns.  I would set the major challenge, for the next five years of Cochrane, to be – how to do a systematic review in a month (or less).

This side-steps the issue of regulatory data and/or IPD!

I see a future for Cochrane as having two types of systematic review: rapid systematic reviews undertaken in a significantly reduced timeframe, and a more costly systematic review that includes regulatory data and/or IPD.  If Cochrane can reduce the cost of a systematic review to around 10% of what it is now it means they can do ten times as many.  Or Cochrane might choose to do fewer than ten times as many rapid systematic reviews and leave any remaining resource to do the more costly systematic reviews.  The issues becomes (i) when can Cochrane ‘get away’ with a low-cost systematic review and (ii) when a high-cost review warranted.  These are questions requiring a research base to answer the questions, as well as being a question of values.

The argument has been made to me that there is a negative cost of doing a low-cost systematic review that might generate the ‘wrong’ answer.  While I appreciate this could be a scenario I would reply that while you’re busy doing one systematic review ‘correctly’ you are neglecting 5-10 rapid systematic reviews that might generate significantly higher benefits.  But, the lack of an evidence base is hampering our ability to address these questions.  This favours the status quo, which could actually be doing more harm than good.

Finally, I can’t help feeling the current direction of travel by Cochrane is taking us down a conceptual cul-de-sac [11]:

“Researchers in dominant paradigms tend to be very keen on procedure. They set up committees to define and police the rules of their paradigm, awarding grants and accolades to those who follow those rules. This entirely circular exercise works very well just after the establishment of a new paradigm, since building systematically on what has gone before is an efficient and effective route to scientific progress. But once new discoveries have stretched the paradigm to its limits, these same rules and procedures become counterproductive and constraining. That’s what I mean by conceptual cul-de-sacs.”

Bottom line: Systematic reviews are vitally important in practicing evidence-based healthcare.  Given that there is a finite funding ‘envelope’ it is imperative to maximise the number of systematic reviews that can be undertaken and to maximise relevancy to clinical practice.  This means significantly reducing the cost per review and improving the prioritisation process.

NOTE (04/09/2015): Since writing this article I have written a number of follow-up articles:


  1. The Cochrane Oversight Committee. Measuring the performance of The Cochrane Library. 2012
  2. Allen IE, Olkin I. Estimating time to conduct a meta-analysis from number of citations retrieved. JAMA. 1999 Aug 18;282(7):634-5.
  3. Cochrane Collaboration Annual Report & Financial Statements 2010/11
  4. Payne D. Tamiflu: the battle for secret drug data. BMJ 2012;345:e7303
  5. HAI Europe – Dr. Tom Jefferson on lack of access to Tamiflu clinical trials
  6. Jefferson TO, Demicheli V, Di Pietrantonj C, Jones M, Rivetti D. Neuraminidase inhibitors for preventing and treating influenza in healthy adults. Cochrane Database Syst Rev. 2006 Jul 19;(3):CD001265
  7. Jefferson T, Jones MA, Doshi P, Del Mar CB, Heneghan CJ, Hama R, Thompson MJ. Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database Syst Rev. 2012 Jan 18;1:CD008965. doi: 10.1002/14651858.CD008965.pub3
  8. Glasziou PP, Shepperd S, Brassey J. Can we rely on the best trial? A comparison of individual trials and systematic reviews. BMC Med Res Methodol. 2010 Mar 18;10:23. doi: 10.1186/1471-2288-10-23
  9. Hemens BJ, Haynes RB. McMaster Premium LiteratUre Service (PLUS) performed well for identifying new studies for updated Cochrane reviews. J Clin Epidemiol. 2012 Jan;65(1):62-72.e1
  10. Sagliocca L, De Masi S, Ferrigno L, Mele A, Traversa G. A pragmatic strategy for the review of clinical evidence. J Eval Clin Pract. 2013 Jan 15. doi: 10.1111/jep.1202
  11. Greenhalgh T. Why do we always end up here? Evidence-based medicine’s conceptual cul-de-sacs and some off-road alternative routes. J Prim Health Care. 2012 Jun 1;4(2):92-7.J Prim Health Care. 2012 Jun 1;4(2):92-7.

Blog at

Up ↑