Trip Database Blog

While doing some curation work on Trip I came across this paper “SSRIs for Hot Flashes: A Systematic Review and Meta-Analysis of Randomized Trials”, so I thought I would work through an example – to keep checking the system.

Step 1 – the search.  This was a bit fiddly!  I didn’t want to simply search for SSRI, so I added a few named SSRIs as well (citalopram, escitalopram, fluoxetine, paroxetine and sertraline).  I also wanted to search for hot flashes OR hot flushes.  A bit of mucking around with the syntax gave me the following:

(“hot flushes” or “hot flashes”) and (SSRIs or citalopram or escitalopram or fluoxetine or paroxetine or sertraline)

This returned 41 possible trials in Trip. 

Step 2 – the analysis. I then went through the titles and selected 13 articles which I felt matched the research question and pressed the ‘Analyse’ button which gave a score of +0.42 – this was after about 5 minutes (would have been quicker if I hadn’t messed up the search syntax).

Step 3 – checking.  Trying to improve the quality I went through the articles to check the various components (effect and sample size) and found I had to adjust three of them (mainly sample size) and this gave a new figure of +0.31.  NOTE: this step added an extra 3 minutes.

So, the final score is +0.31!

Now, time to check against the actual systematic review.  They only included 11 trials (so 2 less than we included – this raises an issue around the lack of quality assessment of our articles).  But, their conclusion:

“SSRI use is associated with modest improvement in the severity and frequency of hot flashes but can also be associated with the typical profile of SSRI adverse effects.”

So, both systems indicate that SSRIs are potentially useful in managing menopausal hot flushes/flashes.  So, on a broad level Trip Rapid Review got the same result as the systematic review.

One issue relating to harm to note, in the results they report “Adverse events did not differ from placebo” which is interesting as in the conclusion they report “…but can also be associated with the typical profile of SSRI adverse effects.”  Essentially, the systematic review didn’t find any particular harm issues, therefore I assume the authors are relying on external data.  I raise this as the Trip Rapid Review system is not – yet – trained to look for adverse events.  But, in this case, I imagine a user could look at any drug reference system (e.g. BNF) to highlight potential adverse events.  Being even more controversial they could even look at the adverse events list in Wikipedia.

So, same result using Trip Rapid Review in eight minutes, I shudder to think how long it took the seven authors of the systematic review – perhaps a year!?

So, what are the lessons learnt?  For me, the following:

We use minimal critical appraisal, our main effort (but the most important) is checking it’s a randomised trial or not.  In future we could easily adapt the system to alter scores based on them being appraised or not.  For instance, if the trial appears in Evidence Updates we assume it’s valid.

I still have issues of the number/final score.  We reported 0.31 but what does that mean?  I favour trying to assign various narratives based on the score, for instance:

• 1 >> 0.5 = Intervention is highly likely to be beneficial.
• 0.49 >> 0.25 = Intervention is likely to be beneficial.
• 0.24 >> -0.24 = Evidence is weak or ambiguous.
• -0.25 >> -0.49 = Intervention is unlikely to be beneficial.
• -0.5 >> -1 =  Intervention is highly unlikely to be beneficial.

But these could be modified based on the number of trials.  For instance scores based on multiple trials is likely to be more reliable than those based on a few.

Trip Rapid Reviews is probably a lot more than interesting!

Remember Trip Rapid Reviews is free and really easy to use – why not try it for yourself?

Wow, a systematic review in five minutes, surely not.

Well, behind the hyperbole, we’re not actually claiming that.  What we’re saying is that we’ve created a system to rapidly ‘examine’ multiple controlled trials and to attempt to approximate the effectiveness of an intervention.

So, how does it work?  It’s surprisingly straightforward:

1. You press the ‘Trip Rapid Review’ button and two search boxes appear, one is for the population (e.g. diabetes, acne) and the other is for the intervention (e.g. metformin, diet).  You enter the appropriate terms and press search. NOTE: The current system is optimised for placebo-controlled trials or trials versus ‘normal care’.
2. The user is then presented with a list of controlled trials which correspond to the search terms and the user selects those they feel are appropriate and then press the ‘Analyse’ button.
3. Our system then machine reads the abstracts and tries to understand if the article favours the intervention or not.  It also tries to ascertain the sample size.  It reads the article using two separate machine reading systems (decision tree and naive bayes) and if they both agree about the result we accept them but if they disagree we display the conclusion and ask that you manually tell us the result(s).  A positive trial scores +1, a negative trial scores -1.
4. Once we have all the information we perform a simple adjustment (based on sample size) so that a small trial is reduced by 75% (so scoring +0.25 or -0.25) and if it’s medium the adjustment is 50% (+0.5 or -0.5).  Large trials are kept at +1 or -1.  
5. All these scores are averaged out to give a figure, the overall score for the intervention.

In most situations this will take 5 minutes, but if a user locates lots of trials it’ll take a bit longer.

A really important message to get across is that our system has not been formally validated (although you may be interested in these in-house results)!  We have done lots of in-house testing (comparing our results with systematic reviews) but we’re committed to evaluating this approach, but until then use suitable caution.

One reason for releasing it – unvalidated – is to allow the Trip users to try it and see how they get on and if the results are plausible.  In most cases we think they will be, but we’re particularly interested in cases where our system fails – as these are the learning points for us.  So, if you try it, find it to not work – please let us know.

Another issue, relating to users trying it, is to try and understand what the numerical data means.  While an overall score of +1 indicates a really positive intervention and -1 a really negative, what does a score of 0.25, 0.35 or 0.45 indicate.  These are the things that require feedback to get right – so keep it coming.

We’re also hoping our approach will stimulate other people to improve on our system – to be inspired.  We already have plans for a version 2 with numerous improvements but we’d be delighted if other groups do even better – as long as they keep it free for others to use!!

But, the best way to understand the system is to try it for yourself, it’s free and really easy to use.  So, try it now and let me know how you get on!

One thing I’ve been working on recently has been an ultra-rapid review system, based on machine learning and some basic statistics.  In a nutshell can we take multiple abstracts, ‘read’ what they’re about and combine the results to give a ‘score’ for the intervention?  More importantly, will any score actually be meaningful?

Our latest version of the system is pretty robust and the significant amount of machine learning has proved beneficial.  So, to start testing it I thought I’d use real data.  To do this I took a relatively random selection of Cochrane Systematic Reviews, with these ‘notes’:

• I typically avoided classes of drugs, focusing on single interventions.
• Our system is optimised for placebo-controlled trials, so that guided my selection.
• I used recently released systematic reviews.

So, our system works in the following step-wise way:

1. Add search terms via two boxes: Population (e.g. diabetes, acne) and Intervention (e.g. antibiotics, zinc).
2. From the list of results, select (via a tick box) which articles are relevant to the search.
3. Press the ‘Analyse’ button and wait (about 5 seconds) for a result.  The result ranging from +1 to -1.  From that result we’ve started to think about assigning a narrative conclusion, as you’ll see from the table below.

So, the results are below, using ten Cochrane Systematic Reviews as a test

Those marked with a ~ means I’m unsure if they are right or not (possibly a shortcoming of the narrative system I’ve used).

But, two clearly wrong (the bottom two), one could argue that’s not too bad.  However, I did have a dig round to see why they might have been wrong and found that:

• The system analysed 17 trials, two were assessed wrong – so if re-assessed the score = 0.31 (likely to be effective)
• Analysed 3 trials, one was assessed wrong – so if re-assessed = 0.28 (likely to be effective)

I actually take this as a positive (the initial incorrectness, followed by the subsequent correctness!).  The current testing system is not the finished article, that should be available in 2-3 weeks.  This will improve on the above in two main ways:

• It will use two different types of machine learning to assess results (in this test we used a single type), making it easier to identify wrongly classified results.
• The system will make it much easier to edit our systems assessment of the scores.

In other words, the newer system will make it much easier to deal with the issues that caused the incorrect assessment of results.

In conclusion, this is early days and our first testing steps.  The results have been very encouraging and when our new system is out it’ll be even better.  But much more testing is required!

Oh yes, the time taken – if you’re interested, then scroll down.

With the exception of the second to last result they all took around 2-3 minutes.  The second to last one took approximately 5 minutes (as I had to scroll through around 55 results to select the 17 that we used). 

We have now released a new mobile interface for the Trip Database.

Previously, a user going to Trip from their mobile would have a disjointed experience!  However, we have no released an adapative interface.  This means our clever system can tell what size screen a user is using and adjust accordingly.  In reality this means that, on a mobile phone, we strip out a lot of functionality, leaving a stripped down searching experience. We have retained the ‘starring‘ feature as a few of our testers liked the idea of starring documents on their mobile to read later when they’re on a full screen.

Below is an image showing the homepage and results page via a mobile phone.  But why look, why not try it for yourself?  Either search for Trip Database on your phone’s search engine or navigate to http://www.tripdatabase.com

Over the last few months I have been working with the Terrier Team at the University of Glasgow.  It started with an email highlighting problems I perceived with search.  My main concern related to the intention behind searches, even common ones.  By intention I mean, what did the person need/hope to find when they searched. For instance, if you do a search for pain, the results Trip produces (as with other clinical search tools) are the same irrespective of the user involved.  Does an oncologist searching for pain really want the same information as a paediatrician?  Clearly not.  Similarly, a search for asthma will mean different things for a paediatrician compared with a primary care doctor or a nurse specialising in asthma. 

So, my work in Glasgow has been around exploring these issues and is starting to produce results and we’ll shortly be moving to a testing phase (so we will be asking for volunteers).  In short, there are three main areas of our work.

Anonymous users (those not signed in).  We’re planning on testing a technique called search results diversification.  Currently, when you issue a query to Trip, our search algorithm makes no attempt to factor-in intentions.  For a given search it returns results based on their text, quality and date.  If all of the documents focus on one specific intent, then that’s too bad.  So, a user might search for breast cancer and it might be that most of the results focus on screening.  If the person’s intention wasn’t related to screening the results are bad.  So, to overcome this you can use diversification.  This involves looking back at previous searches to estimate likely intentions.  For instance, if someone searches for DVT, what words are subsequently added to modify the search, to make it more focussed?  In this example the top five search reformulations are:

• dvt treatment
• dvt prophylaxis guidelines
• dvt d-dimer
• dvt cancer
• dvt diagnosis

We can use the above to help diversify the search results.  So, if a user does a search for DVT we effectively carry out six searches (the above five and DVT alone) and ensure there is a mix of top results covering as many intentions as possible. 

Known user (logged in). This is where it gets particularly interesting!  We have information on the user from their registration e.g. their profession, country and clinical areas of interest.  We also have other information based on their search history.  In effect, we build up a search profile.  With a search profile we can find similar users – the measure we’re calling ‘Clinicians Like Me’.  How does this help?  In a couple of ways:

• When someone searches we can use the diversification technique but instead of diversification across all searches conducted in Trip, it’ll be diversification based on the user themselves and clinician’s like them.  So, if you’re a oncologist the diversifications are likely to be different from a general population.  Using the first example, pain, and you’re an oncologist the diversifications will be based on how other oncologists have reformulated the search – making the results much more focussed.
• Boosting the scores of individual documents clicked on by similar users.  So, in the normal Trip search we might see oncologists are typically clicking on results 8 and 15 for a given search. Using this observation, we can subsequently boost these results for other oncologists when they next search. 

Highlighting new research. We believe that there are ways that we can more effectively highlight newly published research to our users.  We currently add around 5,000 new documents a month to Trip (500 secondary reviews and the bulk of the rest is from primary research), but our current email update approach is not well tailored to the breadth and diversity of our user base.  However, using the above techniques (learning from the users and similar users) we can start to make predictions as to their future information needs and hence better find the new documents you want to see.

All the above sounds wonderful, even magical.  But, Trip is about being evidence-based, so we need to generate evidence that this approach is worthwhile adopting.  To do this we need volunteers (health professionals only – sorry everyone else).  The above approach could dramatically improve the already wonderful Trip search.  So, if you’re a clinician please contact me (jon.brassey@tripdatabase.com) and I can tell you what’s involved – it really won’t be too onerous.

Last week I posted a review of Trip and since then I’ve sent the first set of emails to the advisory board.  This consisted of a list of ideas and issues I’m working on and asked for feedback.  I sent the emails yesterday but already some ideas are generating more excitement than others:

• Mobile interface.  Two comments sum this up – ‘The doctors are welded to their mobile phones’ and ‘just tried website on my Android and it was dreadful’.
• Patient interface. I’m working on this with a local university and their patient group.  The patients want to have better access to evidence, but need some support. So, create a ‘stripped down’ interface with features to make it easier to find and interpret the evidence.  I’m hoping we’ll work with Testing Treatments when we realise this feature.
• Answer engine. If someone searches for ‘acne and antibiotics’ we can infer they’re interested in answering the question ‘are antibiotics useful in acne?’. If so, why not drop in the answer (assuming a robust one is available)?
• Clustering. We may have a systematic review in our index in the BMJ. Critiques of the same article (DARE, EBM journal, Journal Club etc) may also appear. Instead of having multiple entries, just show the original with links to critiques.

 These are the early ‘leaders’ and liable to change.  But these are all really interesting issues with their own challenges.

These last few months have been hard work!

• Mid-March we released the Controlled trials filter in Trip.  500,000 trials, all easily searchable and incorporated into Trip.
• Also in mid-March I presented at EvidenceLive 2013, where I gave a talk under the interesting title ‘Anarchism, Punk and EBM’.  The broad thrust can be read in my A critique of the Cochrane Collaboration blog article. This has now been read nearly 4,000 times.
• In early April I set about recruiting for the new Trip Advisory Board. It’s a 20 strong group of clinicians and information specialists from around the world.  This group will become active in the near future and will help advise me on the way forward for Trip.
• I had some good fun creating the Trip Pinterest account. I view this as a simple way of posting pictures which I can easily link to!
• More recently I have been concentrating on the latest upgrade to Trip. The highlights being better access to full-text articles, a ‘developing world’ filter, integration of DynaMed and case reports and a host of other minor changes.  The full-text feature is a personal triumph as it has taken me so long to figure out how it’s done!  Successive surveys have shown better full-text access as a priority so, finally, to be able to help is wonderful.  As I write this, we have 199 institutions signed up and over 350 individuals have aligned themselves with their institution – not bad for 3(ish) days.  Currently, individuals from Barts Health NHS Trust, NHS Scotland, King’s College London, Academisch Medisch Centrum and University of British Columbia are the biggest institutional ‘subscribers’
• We’re often asked about promotional material and we’re very close to getting some leaflets produced.  On our Pinterest account you can see the final designs.

Not a bad two months for Trip.

But, there’s no resting and other plans are taking shape:

• Clinician similarity is something I blogged about in 2012.  Fairly quietly I’ve been working quite hard on this and have recently received funding to work with the University of Glasgow.  We’re hoping to have initial results of that in the next 2-3 months.
• Reporting even earlier than that will be another project I’ve been working on – near instant reviews.  Trip funded phase one and we received a grant to work on phase two.  This is really exciting as the phase one results were so promising.  At the end of this phase we should have a tool for people to experiment with.  The principles are sound, the technology looks good but I can’t help feeling acceptance will be the hard part!

The above are the two main projects I’m working on.  But that leaves future projects and this is where the advisory board will help.  A few example projects that I’m keen to explore:

• Patient interface – very excited by this
• Mobile interface
• Better publicity
• Creating of a decent business model, which may include a freemium Trip
• Improved full-text access, improve our initial offering
• Further improve the timeline experience
• And a handful more speculative/spectacular ideas

Happy days!

It’s only been a few days but already lots of people and institutions are taking advantage of our link-outs to full-text.  Today we went through the 150 institutions barrier, which astounds me, well it would if I wasn’t so frazzled adding them to our system!

But we have some early adopters and so far the top institutions (by way of users signing up) are:

• University of British Columbia and NHS Scotland – both with 6 users
• King’s College London – with 5
• NHS Wales, University Hospitals Coventry and Warwickshire NHS Trust Library and Knowledge Services, Academisch Medisch Centrum,Barts Health NHS Trust, Macquarie University, Bond University, Leiden University Medical Center, University of Otago – all with 4

We’ve released a bunch of upgrades to the site, some really powerful others simply useful!

The screengrab below (click on it to enlarge) highlights the major changes.

Full-text links: We’ve used two methods for this.  Firstly, we’ve started cross-checking our records with PubMed Central and linking accordingly.  Secondly, we’re working with institutions to allow the easy linking between Trip and the institutions full-text holdings.  For this to work a user needs to alter they profile (via the ‘Setting‘ button), about half-way down there are a series of drop-downs, select your institution from there and it should work straight away.  If your institution is not there then send us an email (jon.brassey@tripdatabase.com) and we’ll tell you the simple steps needed.

DynaMed integration: Click on the DynaMed tab and you’ll see the results.  Access to the actual content is only available for those with subscription access – alas we do not provide that!

Controlled trials database: This has actually been out for a while, but I’m including it here as it was planned with the rest of these changes and is a fairly recent addition.  Click here for further details.

Case Reports: Working with BioMedCentral’s Cases Database we’re really pleased to see this interesting collection added to the site.

Developing World Filter: Working with a slightly modified filter from a Norwegian Cochrane site we have created a specific and sensitive filter. If you would like to know the difference then email us via the email above.

Minor changes

• Ability to delete items from the timeline
• Move from eternal scrolling on timeline to pagination
• Number each result
• Ability to change password