Trip Database Blog

TILT (Today I Learnt That) is a concept I still love.  I thought we’d done it right, used a panel of clinicians to pilot the idea, when that went well we built it.  I think it’s fair to say that it has failed, but after looking over it again after a few months of inaction I’m still enthusiastic about TILT.

The basic premise of TILT is that you record anything you’ve recently learnt (clinically).  As well as being a record of learning (useful for revalidation in the UK) it was also shared with the wider community.  The community could learn from your endeavours.  If you read someone else’s learning a simple click of a button added it to your own portfolio.  In many ways it’s like a twitter meets shared learning.

But why has it failed? I think the reasons are numerous:

• It was over-complicated!  KISS (Keep it simple stupid) – I should have known.  As well as the core concept (which we piloted) we added a few extra layers e.g. the ability to follow other users, the ability to create groups.  I also think the input fields could have been perceived as daunting.  Although optional we had/have fields for tags, time spent learning, reflections etc.  
• Availability of TILT.  Basically it was on the website and we also allowed people to automatically add content from twitter (by the addition of the #TILT hashtag).  But it needed to be easier for people to add TILTs – perhaps toolbars, bookmarklets, partnering with other sites to add TILT functionality. 
• Design.  While I don’t think it’s bad, I feel it could be significantly improved upon.
• Marketing.  Something TRIP isn’t great at and hence getting the word out didn’t really help.  In my naive mind I thought it’d be so good that word of mouth would see it diffuse.  However, this required it to be perfect which – in hindsight – it wasn’t.

To reiterate the concept it great and I still love it.  It’s a bottom-up form of learning – clinician’s read an academic paper (for instance) and distill what they’ve learnt into 2-3 sentences.  They only TILT when they’ve learnt something.  Brilliant.

To make it work I’m thinking the following would help:

• Simplify.  Remove lots of the non-core bits (tags, groups, following etc.) while the site grows and worry about these when the site gets to a size that makes it an issue!  But build up the critical mass required first!
• Availability. Better improve the integration with social media, create bookmarklets etc.  I also think re-writing the site in HTML5 would be great, making it work well on smartphones, tablets etc.
• As part of the HTML5 work I’d be really tempted to improve how the site works and how people can use it.
• Marketing – need I say anything about this!?

If anyone has any further thoughts feel free to share them!

UPDATE: I posted links to the blog on Twitter and the TRIP Facebook page and have received two comments so far:

• “Is TILT still there? Is it still a feature? I never understood what it was or how to use it?” – A good one this, our inability to communicate clearly what TILT is and what we’re trying to acheive.
• “Make login mandatory” – less clear this one as you need to login to add a TILT.  But I guess if you were always logged in it’d be easier to use.  The comment also links to being logged into TRIP.
• It is not clear what the value is for the user to use it. To my mind the value is one of being part of an altruistic community which makes learning easier.  Perhaps I’m too idealistic!

While systematic reviews remain the gold standard for synthesising evidence, they are typically costly and take months and months.  In a healthcare setting, both time and money are under heavy restraint – so what are the options?

I have been undertaking rapid reviews (in the form of clinical Q&As) for nearly 15 years e.g. ATTRACT.  Me and my various teams have answered well over 10,000 questions – the majority taking less than 4 hours.  So, there are clear difference between what we do and what a systematic review does. I have typically justified my outputs by not claiming to do a systematic review but to be transparent about what we do and also, the hope, that we would do better than an individual clinician.  In addition we have published all our answers on the web and many have been viewed over 5,000 times – most pass without comment.  I feel moderately reassured by this post-publication ‘peer review’.  In fact, we have only had one major alert where we clearly made a serious error.  It was around the time of the Cox-2 issues and we relied on pre-crisis documentation!

But, it’d be complacent to think our methods are perfect.  So, I recently asked the EBHC mail-list for any literature on the subject (rapid versus systematic reviews) and the results are below, if you know of any others then please let me know.

• Differences between systematic reviews and health technology assessments: a trade-off between the ideals of scientific rigor and the realities of policy making. Rotstein D et al. Int J Technol Assess Health Care. 2004 Spring;20(2):177-83.
• Rapid versus full systematic reviews: an inventory of current methods and practice in Health Technology Assessment. ASERNIP-S Report No. 60. 2007
• Health technology appraisal of interventional procedures: comparison of rapid and slow methods. Warren. J Health Serv Res Policy July 2007 12:142—146.
• Rapid versus full systematic reviews: validity in clinical practice? Watt et al. Anz j. Surg. 2008; 78: 1037–1040
• Rapid reviews versus full systematic reviews: an inventory of current methods and practice in health technology assessment. Watt A et al. Int J Technol Assess Health Care. 2008 Spring;24(2):133-9.
• Expediting systematic reviews: methods and implications of rapid reviews. Ganann et al. Implementation Science 2010, 5:56
• Evidence summaries: the evolution of a rapid review approach. Khangura et al. Systematic Reviews 2012, 1:10
• Rapid Evidence Assessment Toolkit index (Civil Service)
• Translating research evidence into action in daily practice. Otolaryngol Head Neck Surg. 2010 Jan;142(1):29-30.
• Evidence and guidelines in otorhinolaryngology: the merits of evidence-based case reports

The ability to search TRIP and then filter by publication type (e.g. systematic reviews, guidelines) is often cited as something that’s really positive about TRIP.  Aside from the addition of a few new categories there have been no real changes to it since it was introduced (probably ten years ago).

While it works really well, I’m wondering if it can be improved.  In the image below (a mock up, it’s not real) you’ll see a potential feature is displayed if a user clicks on the ‘Systematic Review’ filter – it allows you to select separate publications with the systematic review category.

Do people like this?

When you hear bad news sometimes you’ve got to get it out of your system.  We were in the early stages of being acquired but things were looking very positive as both sides appeared to want it to happen.  However, we heard today that it isn’t happening – so I’m flat.  Not for any personal reason based on me getting a big fat cheque (I never expected that).  What I thought was that with this new partner, giving us some money to invest, we could make a much bigger difference.

I’ve got a massive backlog of ideas I want to implement, all making TRIP so much better.  To introduce these needs investment, not much, but enough to do it properly. 

If you look at what TRIP has achieved (see this post & this one) with our amazingly low budget (around £25,000 per year) just think what we could achieve with a decent investment.

However, it’s not only the money, it’s help with other things – such as publicity – to help get the message of TRIP across.

So, my disappointment is that I really did feel I could have helped change the world.  For the time being that’s been taken from me – so flat as the proverbial pancake.

The TRIP Database was designed to help me answer questions for the ATTRACT service, a service I’m still involved in.  The ATTRACT methodology involves receiving a clinical question from a clinician and answering it, using the best available evidence, within 4-6 hours.  It’s clear to see that this is not a systematic review!  However, it’s unclear what the effects are so I’ve been doing some searching for evaluations of rapid reviews versus systematic reviews.  It’s early days in the review but a couple of interesting papers are:

• Expediting systematic reviews: methods and implications of rapid reviews
• Evidence summaries: the evolution of a rapid review approach

If you know of others please let me know.

Frequently, when reading around a subject I get side-tracked in thoughts and started thinking about carrying out rapid reviews from within TRIP.  It might work something like this:

1. The user starts by using the TRIP PICO search (click here to see it).
2. A user selects those articles that s/he wants to view.
3. These are then all opened up in new windows framed by a special TRIP frame (so we can do some clever stuff highlighted below).
4. The user can read the articles, highlight passages they want including in the review and ‘send’ them to a review builder on the TRIP site (by simply pressing a button)
5. After this process is carried out the user goes to the review builder, add a narrative to link the passages and TRIP publishes the reviews (after also adding references, etc)

We could even monitor new content added to TRIP and alert the original creator when new articles on the topic are published (arguably we could add them to the actual review as well in a ‘new evidence’ section!

It’d work for ATTRACT and I’m guessing it might work elsewhere…!

I read about an interface for PubMed that had created a proximity search.  I wasn’t sure what ‘proximity search’ was and I imagined it was a geographic proximity.  My thoughts were going down the geographic route due to my interest in geolocation and search in TRIP.   Currently, we take no account of geography in our search results so the UK, Canadian, USA etc guidelines all have similar weights.  However, I’ve thought for years (and probably blogged about it) that it makes sense if you’re searching from Australia, Australian guidelines appear higher up the results than others.

However, after some digging, it appears that it relates to proximity of search terms.  In other words only return documents where two search words are close.  I wasn’t convinced about this so asked our Facebook followers (click here for our Facebook page) and one comment said that it can be useful and gave the example of searching for drug abuse.  So, you could return documents where drug and abuse appear close by.  It made perfect sense to me, I hope it does to you.

But, my thoughts returned to the geographic proximity issue.  I think the following scenario sounds great:

TRIP works with librarians within organisations (e.g. hospitals, universities) to allow organisational accounts which allows 3 things to happen:

• The organisation can load their own documents (e.g. guidelines) for searching.
• The organisation uploads their link resolver (allowing easy resolution to full text documents)
• The organisation promotes their organisational TRIP account to those working/studying in the organisation so they can search the above local documents and/or easily link-out to full text documents.

So, two questions:

• How does the above scenario sound?
• Do people like the idea of proximity searching where people can find words that appear close to each other?

Really interesting questions for me and hopefully you!

I’m still going over the future direction of TRIP and it’s exciting.  We have one potential buyer that we’re talking with and that’s potentially a great move for TRIP – making it secure and allowing investment in the product.However, we cannot allow the potential purchase to divert us from planning for the future. 

As well as all the design and functionality issues mentioned before (see here) I’ve been thinking about developing a premium version of TRIP.  TRIP, as is, would remain free but we’d add functionality to a ‘paid for’ version.  I can see a number of possible enhancements that might make subscribing worthwhile, for instance:

• Development of the answer engine, possibly including content from other subscription services.
• Linking out to full-text.  If an organisation subscribed the library could add the link resolver details to the site and a user would then be able to seamlessly link-out to full-text.
• Allow the organisation to add their own content to the site (e.g. local guidelines, pathways etc).

If we don’t get bought I’m not sure where the money will come from – we’ll see!

Relatively early days of this year and it’s already proving interesting.

I’m still working on the answer engine idea (see here for last post on the topic) and have had the first meeting about that.  There are lots of issues to be resolved, but these appear fairly clear and none particularly worry me!

One topic that has concerned me is the relationship between TRIP and any answer engine we create.  Fortunately, that has clarified over the months and I can easily see a fit that allows a users to search TRIP and get a combined set of results – TRIP results and associated answers.  I’ve done some mock-ups and it fits seamlessly.  So, very excited. 

As ever money is the limiting factor but we may have some good news on that front in the near future.  But, due to this and commercial sensitivities around the answer engine I’m having to stop being quite so open and transparent – which is a real shame.  I think that helps explain why there have been few blog posts this year – even though it has been a really exciting start.

Those of you who know me or have got a feel for TRIP (via this blog or using the site) will know that our biggest motivation is allowing clinicians to get rapid answers to their clinical questions. The TRIP Database has been running for nearly 15 years and we’ve helped shape clinical search, spawning many similar search tools (perhaps the most recent being the eye-wateringly expensive (and soon to be rebranded) NHS Evidence). 
However, over the last few years my reservations about clinical search has grown.  To define that further, I mean clinical search for busy clinicians (as opposed to librarians, information specialists, academics etc).  Those that need a rapid answer to their clinical questions.

The current search paradigm is that users add 1-3 search terms, press ‘search’ and sift through the top 10-20 results to find the answer they’re looking for.  Below are a few problems with this (there are many more):

• We know that clinicians are typically poor searchers, so there’s a handicap from the start.
• The way clinician’s select which papers/documents to look at is problematic based on a number of factors.
• From our experience of Q&A (answering over 10,000 clinical questions) we know the average number of references to answer a question is over two.  So, that suggests that clinicians will need to open 2+ documents and read them to find the information they need – highly inefficient, especially if the document is long.

Using the example of NHS Evidence, they have a vast resource (£700,000+ for marketing alone.  Their marketing budget is nearly 30 times higher than the entire TRIP Database budget!), a good brand, a competent implementation, part of the NHS ‘family’ yet have embarrassingly low search stats.  I believe one of the main reasons is that search is not something that works for most clinicians.  It’s a paradigm, defined by Google and people seem happy to settle for it.  Is it any wonder that clinician’s main source of answers to their questions is to ask a colleague?  One obvious reason that clinicians ask other clinicians  (but there are clearly others) is that they get an answer – not 10-20 links that may answer their question.

So, to me, any solution is to go back to first principles – in this case a clinician with a clinical question.  What do they want? An answer.  I’m hoping that’s not controversial – it seems obvious to me. 

To reiterate, imagine you’re a busy clinician and have a clinical question, what would you prefer:

• A robust answer.
• A list of 10-20 results, any number of which may contain all or part of your answer.

Is it only me that sees this as a ‘no brainer’?

If you’re one of the people that thinks the latter – please contact me (jon.brassey@tripdatabase.com) as I’d love to understand your perspective better

Anyway, moving on to the notion of delivering an answer – this is where it gets complicated but also interesting.  I firmly believe that we shouldn’t shy away from a challenge, not one that’s so important as this.  With the experience gained in TRIP (with search and Q&A) I actually think that the issue is manageable.

Basically, I’m planning on building a system that will take a clinical question and deliver an answer. 

However, one thing for sure, it’s too big for TRIP to do it on it’s own.  So, we’ve started by getting together a small group of people representing organisations who have a vested interest in getting this right.  Those who share in the vision.  To me, the biggest challenge will be managing the disparate bodies – from the small to the very large.

If I can pull this off, we’ve got every chance of making an industry-wide change for the best.