Trip Database Blog

Conflict of interest declaration: Trip’s main aim is to help clinician’s answer their questions using the best available evidence.  As such we have worked, and continue to develop, techniques to hugely reduce the costs of doing systematic reviews.  See (Trip Rapid Reviews – systematic reviews in five minutes, Ultra-rapid reviews, first test results and Trip Rapid Review worked example – SSRIs and the management of hot flashes)

In my presentations to Evidence Live I was (constructively) critical of Cochrane.  This was distilled into two blog posts A critique of the Cochrane Collaboration and Some additional thoughts on systematic reviews.  In the first article I quoted Trish Greenhalgh:

“Researchers in dominant paradigms tend to be very keen on procedure. They set up committees to define and police the rules of their paradigm, awarding grants and accolades to those who follow those rules. This entirely circular exercise works very well just after the establishment of a new paradigm, since building systematically on what has gone before is an efficient and effective route to scientific progress. But once new discoveries have stretched the paradigm to its limits, these same rules and procedures become counterproductive and constraining. That’s what I mean by conceptual cul-de-sacs.”

I quoted Trish as I felt that Cochrane had come to dominate and lead the systematic review paradigm.  But one thing I didn’t write-up at the time and linked with Trish’s quote was my feeling that the methodological rigour and standards set by Cochrane was actually an economic barrier to entry for competitors.  The Wikipedia article on barriers to entry reports:

“In theories of competition in economics, barriers to entry, also known as barrier to entry, are obstacles that make it difficult to enter a given market. The term can refer to hindrances a firm faces in trying to enter a market or industry—such as government regulation and patents, or a large, established firm taking advantage of economies of scale—or those an individual faces in trying to gain entrance to a profession—such as education or licensing requirements.

Because barriers to entry protect incumbent firms and restrict competition in a market, they can contribute to distortionary prices. The existence of monopolies or market power is often aided by barriers to entry.”

Cochrane, due to their dominance, effectively set the standards of what’s deemed acceptable (irrespective of the significant evidence to the contrary – see the previous two blog posts for further information).  This effectively stifles competition. If systematic reviews could be done quickly and easily by anyone the business model of Cochrane would be severely compromised – I can see no other losers (except perhaps pharma).

Perhaps it is a coincidence that most changes to systematic review methods over the years appear to have more to do with increasing the methodological burden (by squeezing increasingly small amounts of bias out of the results) than with reducing the costs?

What has prompted the above post has been the announcement of the winner of the Nobel Prize for Economics. Jean Tirole has won for his work on market power and regulation.  The BBC reports:

“Many industries are dominated by a small number of large firms or a single monopoly,” the jury said of Mr Tirole’s work. “Left unregulated, such markets often produce socially undesirable results – prices higher than those motivated by costs, or unproductive firms that survive by blocking the entry of new and more productive ones.”

Now, that’s got to be a good link – EBM, Cochrane and the Nobel Prize for Economics!

But the point of the post, is not to moan at Cochrane, but to suggest that the systematic review ‘market’ is problematic and there appears to be little appetite to radically change things.  If we want to improve care we need more systematic reviews which means we need to innovate.  And by innovate I don’t mean small iterative improvements, more substantial changes are needed.

Perhaps we could start at first principles and ask why do we do systematic reviews in the first place?  I used to think it was to get an accurate assessment of effect size.  However, if you look at the evidence it’s fairly clear that systematic reviews – based on published trials – are pretty poor in this regard.  But if it’s not that, then why do we do them?  Once we can clearly articulate why we can perhaps better understand how to produce them more efficiently.

I’ve been involved in clinical uncertainties for many years.  I had the pleasure of helping create the DUETs database (UK Database of Uncertainties about the Effects of Treatments).  Around the same time Trip released the Tag Cloud of Clinical Uncertainty which was a great experiment. In all aspects of my professional life I like to highlight the importance of clinical uncertainty.  Although the term often unnerves people, I think they feel threatened by the notion.  But, it could be worse, the likes of Iain Chalmers and Muir Gray have often used the phrase clinical ignorance – which is far harsher.

My interest in uncertainties stems from my desire to improve the research procurement process.  The main drive with Trip is to help clinicians answer their clinical questions.  Without knowing the gaps in their knowledge and the evidence base how can you procure suitable primary research or even secondary, evidence synthesis?  Myself and my teams have answered over 10,000 clinical questions so we know how frequently the research base is lacking or not focused on clinical care.  So things need to improve and I think recording uncertainties is a great way to help.

I raise all this as, in my business planning, I’ve spoken to one of the UK’s largest research ‘agency’ and they are keen to work with Trip to better understand users questions and gaps in the evidence base.  So, for me, the answer is to try and capture the clinical questions our users have and when Trip has let them down.  I have a few ideas around this, including creating a PICO+ tool; a step-by-step tool to allow users to easily answer their clinical questions.  The user would start by adding the full question and then we would guide them through the PICO steps (e.g. what is the population, what is the intervention).  At the end they can tell us if their question has been answered.  If not, it’s got a good chance of being a clinical uncertainty. 

Seems like a plan!

Trip prides itself as a great tool for answering clinical questions. Over 80% of users find the information they need, all or most of the time.  But that’s still not perfect and one idea I keep coming back to is the ‘answer engine’.  The wonderful Muir Gray said, in relation to finding evidence, that three clicks was two clicks too many.  So, the challenge is, is there a way of getting answers based on a single mouse click?

The answer (or perhaps ‘My answer’) is the answer engine.

This would involve a system to try and understand the search and display a suitable answer.  So, if a user searches for minocycline and acne we can be fairly confident that they’re interested to know if minocycline is effective in treating acne.  Therefore, we could drop in the following answer:

Minocycline is an effective treatment for moderate to moderately-severe inflammatory acne vulgaris, but there is still no evidence that it is superior to other commonly-used therapies. This review found no reliable evidence to justify the reinstatement of its first-line use, even though the price-differential is less than it was 10 years ago. Concerns remain about its safety compared to other tetracyclines.

This has been taken from a recent Cochrane Systematic Review.  The normal search results would appear beneath the ‘answer’.  The user gets a great answer in one click.

There are a few issues with the above and one is scalability.  Parts of this can be automated but much of it will be manual.  Also, relying of sources – such as Cochrane – means it’s led by the evidence producers not the user.  So, the challenge is to have users supply answers.  Which leads to the Journal of Clinical Q&A.

The idea is to set up a brand new journal dedicated to answering real clinical questions.  Based, roughly on the BestBETs site it will follow a similar structure (e.g. Steroids in lateral epicondylitis).  The clinical bottom line will be pulled through into Trip to act as the ‘answer’ and then users can click to see the full article.

Peer-review is problematic on many levels and Richard Smith (former editor of the BMJ) has frequently criticises the current peer-review process (e.g. A woeful tale of the uselessness of peer review and Scrap peer review and beware of “top journals”).  But how can we improve on it?  I’m not sure we can but I’m open to help!  My current proposal is as follows:

• Each answer will be reviewed by an in-house team, a sanity check.  Those that seem reasonable will be released into the answer engine as an ‘answer pending approval’
• We would then ask the wider Trip community to read and rate the answer.  This would borrow from the F1000 approach which uses three classes: Approved, Approved with Reservations and Not Approved.  An article will be considered published when it reaches a certain approval threshold. Note, the F1000 approach is not without criticism (e.g. PubMed and F1000 Research — Unclear Standards Applied Unevenly), hence writing this article with the hope of obtaining help.

Trip would have a good answer and the person who uploaded it will obtain a citation.  The plan is to start slowly, see how it develops but the longer-term view would be to see the answers appear in Medline/PubMed – as currently happens with a large number of the articles in BestBETs.

The above is an idea, a work in progress.  I think there is every chance this can become a reality but a little help in refining the concept would be really good.

Trip is a wonderfully useful search engine, widely used and it has a great reputation and brand. We’re thinking we could build on this to create a formalised evidence service. 

An evidence service could undertake a number of roles to support users (probably organisations) for instance:

• Literature searches
• Critical appraisals
• Evidence reviews/synthesis
• Clinical Q&A
• Horizon scanning
• Etc

We have a network of highly skilled information experts who would undertake the work.  Due to our low overheads we could provide a very cost-effective service.

I have experience in the UK where there are a large number of organisations (e.g. CCGs) that do not have timely access to timely, robust evidence to support their decisions.  This is really problematic when introducing changes to the system; how can they be evidence-based with no evidence input?  I doubt UK is atypical in this respect.  Therefore, there is a real opportunity to improve care and improve our business!

If you’re interested in the service and want to help us develop our service then let me know.

Clickstream data is not widely known about.  In short it’s the analysis of users clicking on websites.  We’ve started exploring this and the clickstream we’re using is based on users clicking on particular search results. In short, if you do a search on Trip and click on documents number 2, 4 and 9 you’re effectively telling us that, for your intention, they’re connected.  In isolation it’s arguably meaningless, but over thousands of searches you start to see structure.  I’ve blogged about this previously (here, here and here) but now we’ve got more results.

Below is the largest continuous graph/map of connected documents – over 10,000 long (click on image to expand). 

Clinical trials are vital if we wish to improve healthcare, as such they are an important component of EBM.  However, trials are not straightforward and one major problem can be the recruitment of enough patients to ensure the trials have enough power.  There are numerous papers on the topic (well hundreds) and below are a sample:

• Recruitment Challenges in Clinical Trials for Different Diseases and Conditions
• Strategies for Recruitment and Retention of Participants in Clinical Trials
• Improving the recruitment activity of clinicians in randomised controlled trials: a systematic review
• Methods to improve recruitment to randomised controlled trials: Cochrane systematic review and meta-analysis
• Clinical trials: can technology solve the problem of low recruitment?

I raise this as Trip has had two conversations in a week with organisations involved with trial recruitment and both are very interested in working with Trip and our network of users.  Trip has around 100,000 registered health professionals, the vast majority will appreciate the need for clinical trials and therefore be sympathetic to the need to recruit patients.

So, the idea seems to be that if we are made aware of a trial, of say heart failure, we alert health professionals who have indicated an interest in heart failure (either through their registration or their search history) and are based geographically close.

Early days but it seems like a great idea.  As well as hopefully improving patient recruitment it could also help Trip’s finances – win:win.

Earlier this week I had the first planning meeting around turning Trip into a freemium service.  Much of the current site will remain free. The premium (paid for) service will have no restrictions and will have some enhancements.

So, here’s what we’re thinking:

Free service

• no advanced search
• limited export options
• no images, videos etc.

Premium service

• better export facilities
• improved meta-data
• advanced search available
• all content types available
• no adverts
• various discounts from evidence-style products
• ability to add (and share) reflective notes against each article
• introduce the answer engine

Costings is still fluid.  I intend to have an individual subscription and an institutional one.

Individual – My aim is to suggest a monthly charge of around a cup of Starbucks coffee but people will be at liberty to reduce (or increase) the amount they pay, although there would be a lower limit.

Institutional – Various levels based on the number of individual accounts available.  In addition we intend to introduce some analytics so the institution can see how the service is being used.

It’s still early days and if you have any opinions please let me know.

We have recently been in discussion with an organisation about potentially partnering with us.  This would have brought some financial security to the site as well as allowing me to further develop some of the products we’re currently working on.  These technologies being shared with the organisation. While nothing has officially been said I get the impression that it’s not a priority for them – so I need to move on.

I have come to the conclusion that I cannot rely on others to support Trip and therefore we need to strike out on a ‘Plan B’; which is a freemium version of Trip.  In a nutshell most of the current Trip remains free and certain enhancements are made to create a ‘paid for’ premium version.

The insecurity around funding is getting more than tiring – so something has to change.  We’ve got to work out the details (features etc) but my plan is to keep the price low – I quite like the idea of linking it to the local price of a Starbucks cappuccino (or similar).  The point being that if you value Trip you’ll surly pay a monthly fee that is the same as a Starbucks (I’d welcome feedback on this point).

If we can get a regular, secure income we can properly plan for Trip.  We can enhance and roll-out some truly great products.  The vision I have for Trip is really exciting and as such I could have gone and secured venture capital funding.  But that would have changed the nature of Trip, something I’m not particularly keen on – for obvious reasons.

In January this year I was a participant at a small gathering of EBMers in Oxford, exploring the notion of RealEBM.  It was jointly organised by Trish Greenhalgh and the Centre for Evidence Based Medicine.  The participant list was Trisha Greenhalgh, Jeremy Howick, Neal Maskrey, Druin Burch, Martin Burton, Hasok Chang, Paul Glasziou, Iona Heath, Carl Heneghan, Michael P Kelly, Richard Lehman, Huw Llewelyn, Margaret McCartney, Ruairidh Milne, Des Spence and myself.  Someone – certainly not me – called it the Evidence Based Medicine Renaissance Group.

A paper has been prepared and was recently published in the BMJ Evidence based medicine: a movement in crisis?

The preamble is:

Trisha Greenhalgh and colleagues argue that, although evidence based medicine has had many benefits, it has also had some negative unintended consequences.They offer a preliminary agenda for the movement’s renaissance, refocusing on providing useable evidence that can be combined with context and professional expertise so that individual patients get optimal treatment.

And the conclusion:

Much progress has been made and lives have been saved through the systematic collation, synthesis, and application of high quality empirical evidence. However, evidence based medicine has not resolved the problems it set out to address (especially evidence biases and the hidden hand of vested interests), which have become subtler and harder to detect. Furthermore, contemporary healthcare’s complex economic, political, technological and commercial context has tended to steer the evidence based agenda towards populations, statistics, risk, and spurious certainty. Despite lip service to shared decision making, patients can be left confused and even tyrannised when their clinical management is inappropriately driven by algorithmic protocols, top-down directives and population targets.

Such problems have led some to argue for the rejection of evidence based medicine as a failed model. Instead we argue for a return to the movement’s founding principles—to individualise evidence and share decisions through meaningful conversations in the context of a humanistic and professional clinician-patient relationship (box 2). To deliver this agenda, evidence based medicine’s many stakeholders—patients, clinicians, educators, producers and publishers of evidence, policy makers, research funders, and researchers from a range of academic disciplines—must work together. Many of the ideas in this paper are not new, and a number of cross sector campaigns with similar goals have already begun (box 3). We hope that our call for a campaign for real evidence based medicine will open up debate and invite readers to contribute (for example, by posting rapid responses on bmj.com).
I hope you’re interested and if you need an electronic copy, just drop me a line.