Many will be familiar with my post A critique of the Cochrane Collaboration, it’s been the most viewed article ever published on this blog. Continuing in the theme was Some additional thoughts on systematic reviews and more recently Evidence, hourglasses and uncertainty.
They all point to the current methods employed in systematic reviews (as exemplified by Cochrane) being a mess! As a type of summary, a few problems:
- They can’t be relied upon to be accurate
- They’re financially costly
- They’re typically out of date
- Significant opportunity cost
More evidence that my perspective is correct is shown is the presentation Does access to clinical study reports from the European Medicines Agency reduce reporting bias? submitted to the Cochrane Colloquium in Vienna. The conclusions:
“Unpublished clinical study reports held by EMA may be a useful source to reduce outcome reporting bias.“
It’s a testimony to Cochrane’s openness to allow such ‘dissent’ to be published. It’s a dissenting view as the current Cochrane methods rely almost exclusively on published journal articles; unpublished clinical study reports – virtually unheard of. But that’s Cochrane; on one hand a business trying to maximise it’s business model but on the other a collection of individuals doing there best to improve methods. As an outsider, I find this tension fascinating. Why? Because the best one can say for Cochrane’s methods (alongside most other SR producers) is that they are likely to produce ‘ball park’ accuracy – in most cases you simply cannot rely on the methods to produce results you can trust. And this is where the tension comes, if they were being transparent, they would say ‘buy the Cochrane Library, most of the SRs are likely to be out of date and many are likely to be innaccurate’ which even I can see is not great for sales.
But this bring me nicely to diffusion of innovations – an area I studied for the PhD I never completed! The Wikipedia article summarises it nicely by saying its “..is a theory that seeks to explain how, why, and at what rate new ideas and technology spread through cultures.” The spread of an innovation is characterised as an S-shaped curve:
In relation to systematic review methods, where are you on this scale? If, like me, you think there are serious problems by relying on published journal articles you’re probably at the innovator side of things. If, however, you think current methods are great and there is no need to change you may well be a laggard. But, the majority (those that are feeling uneasy) are likely to be in the middle.
Trip Database Blog 
Recent Comments