Trip Database Blog

One thing I’ve been working on recently has been an ultra-rapid review system, based on machine learning and some basic statistics.  In a nutshell can we take multiple abstracts, ‘read’ what they’re about and combine the results to give a ‘score’ for the intervention?  More importantly, will any score actually be meaningful?

Our latest version of the system is pretty robust and the significant amount of machine learning has proved beneficial.  So, to start testing it I thought I’d use real data.  To do this I took a relatively random selection of Cochrane Systematic Reviews, with these ‘notes’:

• I typically avoided classes of drugs, focusing on single interventions.
• Our system is optimised for placebo-controlled trials, so that guided my selection.
• I used recently released systematic reviews.

So, our system works in the following step-wise way:

1. Add search terms via two boxes: Population (e.g. diabetes, acne) and Intervention (e.g. antibiotics, zinc).
2. From the list of results, select (via a tick box) which articles are relevant to the search.
3. Press the ‘Analyse’ button and wait (about 5 seconds) for a result.  The result ranging from +1 to -1.  From that result we’ve started to think about assigning a narrative conclusion, as you’ll see from the table below.

So, the results are below, using ten Cochrane Systematic Reviews as a test

Those marked with a ~ means I’m unsure if they are right or not (possibly a shortcoming of the narrative system I’ve used).

But, two clearly wrong (the bottom two), one could argue that’s not too bad.  However, I did have a dig round to see why they might have been wrong and found that:

• The system analysed 17 trials, two were assessed wrong – so if re-assessed the score = 0.31 (likely to be effective)
• Analysed 3 trials, one was assessed wrong – so if re-assessed = 0.28 (likely to be effective)

I actually take this as a positive (the initial incorrectness, followed by the subsequent correctness!).  The current testing system is not the finished article, that should be available in 2-3 weeks.  This will improve on the above in two main ways:

• It will use two different types of machine learning to assess results (in this test we used a single type), making it easier to identify wrongly classified results.
• The system will make it much easier to edit our systems assessment of the scores.

In other words, the newer system will make it much easier to deal with the issues that caused the incorrect assessment of results.

In conclusion, this is early days and our first testing steps.  The results have been very encouraging and when our new system is out it’ll be even better.  But much more testing is required!

Oh yes, the time taken – if you’re interested, then scroll down.

With the exception of the second to last result they all took around 2-3 minutes.  The second to last one took approximately 5 minutes (as I had to scroll through around 55 results to select the 17 that we used). 

We have now released a new mobile interface for the Trip Database.

Previously, a user going to Trip from their mobile would have a disjointed experience!  However, we have no released an adapative interface.  This means our clever system can tell what size screen a user is using and adjust accordingly.  In reality this means that, on a mobile phone, we strip out a lot of functionality, leaving a stripped down searching experience. We have retained the ‘starring‘ feature as a few of our testers liked the idea of starring documents on their mobile to read later when they’re on a full screen.

Below is an image showing the homepage and results page via a mobile phone.  But why look, why not try it for yourself?  Either search for Trip Database on your phone’s search engine or navigate to http://www.tripdatabase.com

Over the last few months I have been working with the Terrier Team at the University of Glasgow.  It started with an email highlighting problems I perceived with search.  My main concern related to the intention behind searches, even common ones.  By intention I mean, what did the person need/hope to find when they searched. For instance, if you do a search for pain, the results Trip produces (as with other clinical search tools) are the same irrespective of the user involved.  Does an oncologist searching for pain really want the same information as a paediatrician?  Clearly not.  Similarly, a search for asthma will mean different things for a paediatrician compared with a primary care doctor or a nurse specialising in asthma. 

So, my work in Glasgow has been around exploring these issues and is starting to produce results and we’ll shortly be moving to a testing phase (so we will be asking for volunteers).  In short, there are three main areas of our work.

Anonymous users (those not signed in).  We’re planning on testing a technique called search results diversification.  Currently, when you issue a query to Trip, our search algorithm makes no attempt to factor-in intentions.  For a given search it returns results based on their text, quality and date.  If all of the documents focus on one specific intent, then that’s too bad.  So, a user might search for breast cancer and it might be that most of the results focus on screening.  If the person’s intention wasn’t related to screening the results are bad.  So, to overcome this you can use diversification.  This involves looking back at previous searches to estimate likely intentions.  For instance, if someone searches for DVT, what words are subsequently added to modify the search, to make it more focussed?  In this example the top five search reformulations are:

• dvt treatment
• dvt prophylaxis guidelines
• dvt d-dimer
• dvt cancer
• dvt diagnosis

We can use the above to help diversify the search results.  So, if a user does a search for DVT we effectively carry out six searches (the above five and DVT alone) and ensure there is a mix of top results covering as many intentions as possible. 

Known user (logged in). This is where it gets particularly interesting!  We have information on the user from their registration e.g. their profession, country and clinical areas of interest.  We also have other information based on their search history.  In effect, we build up a search profile.  With a search profile we can find similar users – the measure we’re calling ‘Clinicians Like Me’.  How does this help?  In a couple of ways:

• When someone searches we can use the diversification technique but instead of diversification across all searches conducted in Trip, it’ll be diversification based on the user themselves and clinician’s like them.  So, if you’re a oncologist the diversifications are likely to be different from a general population.  Using the first example, pain, and you’re an oncologist the diversifications will be based on how other oncologists have reformulated the search – making the results much more focussed.
• Boosting the scores of individual documents clicked on by similar users.  So, in the normal Trip search we might see oncologists are typically clicking on results 8 and 15 for a given search. Using this observation, we can subsequently boost these results for other oncologists when they next search. 

Highlighting new research. We believe that there are ways that we can more effectively highlight newly published research to our users.  We currently add around 5,000 new documents a month to Trip (500 secondary reviews and the bulk of the rest is from primary research), but our current email update approach is not well tailored to the breadth and diversity of our user base.  However, using the above techniques (learning from the users and similar users) we can start to make predictions as to their future information needs and hence better find the new documents you want to see.

All the above sounds wonderful, even magical.  But, Trip is about being evidence-based, so we need to generate evidence that this approach is worthwhile adopting.  To do this we need volunteers (health professionals only – sorry everyone else).  The above approach could dramatically improve the already wonderful Trip search.  So, if you’re a clinician please contact me (jon.brassey@tripdatabase.com) and I can tell you what’s involved – it really won’t be too onerous.

Last week I posted a review of Trip and since then I’ve sent the first set of emails to the advisory board.  This consisted of a list of ideas and issues I’m working on and asked for feedback.  I sent the emails yesterday but already some ideas are generating more excitement than others:

• Mobile interface.  Two comments sum this up – ‘The doctors are welded to their mobile phones’ and ‘just tried website on my Android and it was dreadful’.
• Patient interface. I’m working on this with a local university and their patient group.  The patients want to have better access to evidence, but need some support. So, create a ‘stripped down’ interface with features to make it easier to find and interpret the evidence.  I’m hoping we’ll work with Testing Treatments when we realise this feature.
• Answer engine. If someone searches for ‘acne and antibiotics’ we can infer they’re interested in answering the question ‘are antibiotics useful in acne?’. If so, why not drop in the answer (assuming a robust one is available)?
• Clustering. We may have a systematic review in our index in the BMJ. Critiques of the same article (DARE, EBM journal, Journal Club etc) may also appear. Instead of having multiple entries, just show the original with links to critiques.

 These are the early ‘leaders’ and liable to change.  But these are all really interesting issues with their own challenges.

These last few months have been hard work!

• Mid-March we released the Controlled trials filter in Trip.  500,000 trials, all easily searchable and incorporated into Trip.
• Also in mid-March I presented at EvidenceLive 2013, where I gave a talk under the interesting title ‘Anarchism, Punk and EBM’.  The broad thrust can be read in my A critique of the Cochrane Collaboration blog article. This has now been read nearly 4,000 times.
• In early April I set about recruiting for the new Trip Advisory Board. It’s a 20 strong group of clinicians and information specialists from around the world.  This group will become active in the near future and will help advise me on the way forward for Trip.
• I had some good fun creating the Trip Pinterest account. I view this as a simple way of posting pictures which I can easily link to!
• More recently I have been concentrating on the latest upgrade to Trip. The highlights being better access to full-text articles, a ‘developing world’ filter, integration of DynaMed and case reports and a host of other minor changes.  The full-text feature is a personal triumph as it has taken me so long to figure out how it’s done!  Successive surveys have shown better full-text access as a priority so, finally, to be able to help is wonderful.  As I write this, we have 199 institutions signed up and over 350 individuals have aligned themselves with their institution – not bad for 3(ish) days.  Currently, individuals from Barts Health NHS Trust, NHS Scotland, King’s College London, Academisch Medisch Centrum and University of British Columbia are the biggest institutional ‘subscribers’
• We’re often asked about promotional material and we’re very close to getting some leaflets produced.  On our Pinterest account you can see the final designs.

Not a bad two months for Trip.

But, there’s no resting and other plans are taking shape:

• Clinician similarity is something I blogged about in 2012.  Fairly quietly I’ve been working quite hard on this and have recently received funding to work with the University of Glasgow.  We’re hoping to have initial results of that in the next 2-3 months.
• Reporting even earlier than that will be another project I’ve been working on – near instant reviews.  Trip funded phase one and we received a grant to work on phase two.  This is really exciting as the phase one results were so promising.  At the end of this phase we should have a tool for people to experiment with.  The principles are sound, the technology looks good but I can’t help feeling acceptance will be the hard part!

The above are the two main projects I’m working on.  But that leaves future projects and this is where the advisory board will help.  A few example projects that I’m keen to explore:

• Patient interface – very excited by this
• Mobile interface
• Better publicity
• Creating of a decent business model, which may include a freemium Trip
• Improved full-text access, improve our initial offering
• Further improve the timeline experience
• And a handful more speculative/spectacular ideas

Happy days!

It’s only been a few days but already lots of people and institutions are taking advantage of our link-outs to full-text.  Today we went through the 150 institutions barrier, which astounds me, well it would if I wasn’t so frazzled adding them to our system!

But we have some early adopters and so far the top institutions (by way of users signing up) are:

• University of British Columbia and NHS Scotland – both with 6 users
• King’s College London – with 5
• NHS Wales, University Hospitals Coventry and Warwickshire NHS Trust Library and Knowledge Services, Academisch Medisch Centrum,Barts Health NHS Trust, Macquarie University, Bond University, Leiden University Medical Center, University of Otago – all with 4

We’ve released a bunch of upgrades to the site, some really powerful others simply useful!

The screengrab below (click on it to enlarge) highlights the major changes.

Full-text links: We’ve used two methods for this.  Firstly, we’ve started cross-checking our records with PubMed Central and linking accordingly.  Secondly, we’re working with institutions to allow the easy linking between Trip and the institutions full-text holdings.  For this to work a user needs to alter they profile (via the ‘Setting‘ button), about half-way down there are a series of drop-downs, select your institution from there and it should work straight away.  If your institution is not there then send us an email (jon.brassey@tripdatabase.com) and we’ll tell you the simple steps needed.

DynaMed integration: Click on the DynaMed tab and you’ll see the results.  Access to the actual content is only available for those with subscription access – alas we do not provide that!

Controlled trials database: This has actually been out for a while, but I’m including it here as it was planned with the rest of these changes and is a fairly recent addition.  Click here for further details.

Case Reports: Working with BioMedCentral’s Cases Database we’re really pleased to see this interesting collection added to the site.

Developing World Filter: Working with a slightly modified filter from a Norwegian Cochrane site we have created a specific and sensitive filter. If you would like to know the difference then email us via the email above.

Minor changes

• Ability to delete items from the timeline
• Move from eternal scrolling on timeline to pagination
• Number each result
• Ability to change password

In February I posted an article discussing ‘the near instantaneous meta-analysis‘. In a nutshell – is there a way to very rapidly combine the results of multiple-trials?

Since then we have been quite busy working on this project, helped by some external expertise and a recent research grant.  Trip funded phase one, a proof of concept phase that allowed me to appreciate the challenges, limitations and opportunities that our approach presented.  The results were great and since then we have been awarded a grant to move forward to phase two.

Phase two will create a working model for people to use. This will be quite a simple solution and will only be aimed at synthesising placebo-controlled trials (more complex, comparator trials will form phase three).  The working model will work as follows:

1. User will use a modified search box telling us the condition and the intervention (e.g. acne and antibiotics).
2. Our system will search just the controlled trials portion of Trip to identify suitable trials.
3. We will then analyse these and present a score (more below on the scoring system).
4. We will then have an area that explains the results, how we arrived at them and the ability for the user to alter certain aspects.  This last bit is important as we’re relying on machines to ‘read’ the documents and extract pertinent information.  This is unlikely to be foolproof and while the system ‘learns’ it’ll need some feedback from users.  But, if the user does make alterations we will then re-analyse based on the updated information.

Quite simple really and steps 1-3 will take less than a second (we hope).

As for the score, that’s an interesting area and we’re sure it’ll change over time.  But the thinking at the moment is – what is most clinically useful?  After all, our audience for this will be practicing clinicians, not academics.  As such we’re thinking that an effect size is not particularly great/intuitive.  I really like the Clinical Evidence system for rating interventions e.g. ‘Likely to be beneficial’, ‘Unknown effectiveness’.

However, I’m also struck by the systems used by Amazon and TripAdvisor to rate articles.  A product/holiday is given an overall score but you can easily see how the score is arrived at.  When I use these I always look at the reasons people have given for 1 or 2 stars (ie people who have rated the item poor).  Whichever system we use we’ll make it very easy for users to differentiate good and bad aspects of an intervention.

Hopefully, phase two will be released in 6-8 weeks, probably not on broad release to start. This will be a gentle release, to a few people to start with.  This will allow us to alter the algorithms, allow for further machine learning etc.

I see this whole ‘instant review’ system taking a minimum of four phases.  Hopefully, if phase two works as well as we think it will, funding will follow to allow us to move to phase three.

In a meeting yesterday we discussed the concept of institutional ‘accounts’ with regard to our soon to be released full-text link-outs. In this scenario an institution (hospital, university etc) would give us some information to allow their users to link directly from Trip to the full-text.  For this to work the user needs to tell us they work for the particular institution.

But, once we have the information, can we be more useful?

Having all the evidence in Trip is great, but often local policies, guidelines etc are really important to consider. So, why not allow the user to search all the evidence in Trip but also be able to see if their institution has something to say on the issue.  It makes perfect sense to me.  But this is non-trivial and requires a number of issues to be dealt with:

• The institution needs to agree to this.
• How do we get local content into Trip.
• What format, web-based in easy but Word documents creates an extra level of complexity.
• Once we have the content in Trip, how do we maintain it being up-to-date?

The latter point is probably the biggest issue.  It’s fine to have initial enthusiasm to add content, but to go back every now and then and maintain up-to-dateness is a harder proposition. What happens if the enthusiast leaves?

That worry aside, doing a check every 3-6 months isn’t a big deal and we could arguably build some alert system saying ‘You have not checked your content for 3 months, please go and check now’.  I’m guessing we could even automate something that checks all the links every now and then.

One to ponder and to ask the advisory board about, when we get the latest upgrade out of the way.

UPDATE: One thought has struck me, why not create a spider to go and grab all the content on the site (the way Google does).  This would be no work for the institution but requires all documents to be on a public-facing website (so no intranet content).  Also, it grabs everything (e.g. see this MIT example) which might introduce some noise!