Trip Database Blog

Liberating the literature

Automated reviews – why?

In my previous post on our automated review system I concentrated on describing the product. This is all fine, but one quotation I’m trying to remember in my day-to-day life is:

So, why do what we’re doing?  The reasons are multiple and include:

  • Years ago I had a conversation with someone and I ended up sending them a link to a set of Trip search results.  They said that was fine, if he had a few hours to read them – which he didn’t.  I started to wonder if there was a way of automatically scanning the literature to allow people to really easily get a feel for a topic area.  So, you might want to see what interventions are useful for a given condition (perhaps the first line has failed).  Our system will automatically generate a list of interventions with an estimate of likely effectiveness.
  • The Trip Answer Engine is hugely popular and we can use the technology to boost the content.  So, it solves a problem for Trip (how to boost coverage) and it solves the problem of quick access to information for a specific clinical question.  So, a user might want to search for acne and lasers we can pull through an answer along the lines of “Lasers have been studied in 12 RCTs and have been broadly favourable” we can link to all the articles (if the user has the time) but we can also rank the intervention, saying something like “Lasers are ranked 3 out of 8 for interventions in acne“.  This seems incredibly useful to me.
  • Updating guidelines and reviews are problematic.  With our system you could ‘watch’ an intervention and get alerts when new evidence is generated.  Or, even more useful, when new research is published that contradicts the previous findings.
  • Rapid reviews are increasingly important and the automated system could form a core part of any semi-manual rapid review tool.
  • Intellectual challenge!

So, one product – multiple problems solved!


Automated reviews – very positive progress

NOTE: since publishing this we have a linked post Automated reviews – why?, which compliments this one!

This work has been supported via the KConnect project (a Horizon 2020 EU-funded project) and on Wednesday we had to present the work of the whole consortium at the EU in Luxembourg.  The response was overwhelmingly positive and so I wish to share a bit more of the work.

A slight bit of context; I have been involved in a number of automation projects and these were typically seen as methods to supplement the human methodology.  For instance speeding up risk of bias assessment via tools such as RobotReviewer. While these are great initiatives I wanted to explore what could be done fully automatically.

At Luxembourg we presented our ‘product’. The product is a system that automatically synthesises randomised controlled trials (and potentially systematic reviews).  It is not finished but we have a very good first effort.  I’m not sure whether to classify it as a ‘proof of concept’ or ‘alpha’ version – not sure it matters!

All the images below are based on asthma and I have deliberately blurred out the intervention names.  The reason being that we are still improving the results (more below) and I would hate to think people will be put off the system by making judgements based on a system that has yet to be optimised.

The first image is the current default result which is to present the interventions (y-axis) alphabetically while the likely effectiveness is presented on the x-axis. Note, as there are an awful lot of interventions for asthma we’ve can’t show them all – so we’re simply showing a single screen-grab, the actual graph is considerably taller!

To orientate you:

  • Each ‘blob’ represents a single intervention.  The size of the blob indicates the total population used in the trials.  So, a big blob indicates more participants in the various trials.
  • The horizontal positioning is based on our estimate of effectiveness (not effect size) and the further right it is the more effective the system estimates the intervention to be.  This is further indicated by the colours – green being better and red worse (traffic light colouring)!
  • Above the graph there is the ability to sort the interventions by number of trials, sample size, score etc.
  • The sample size refinement allows users to exclude trials that are below the size entered – as we know small trials tend to be less reliable.
  • The risk of bias allows you to automatically remove trials that are not considered low risk of bias – so another reliability measure.

This second graph shows the results arranged by effectiveness:

And, to reiterate, this is fully automatic and always up to date.  As new RCTs are published (PubMed only at present) they will be automatically added.  To me that’s incredibly cool.

While I would love to share this more widely there is still some work to do before I’m happy to open it up.  In internal testing we have identified a number of areas that could, realistically, be improved.  Nothing major, we’re just being sure we’ve done as much as we can – under the fully automatic banner – to ensure the biggest impact.

As far as I know, this is the first fully automated evidence synthesis/review tool. This is such a disruptive bit of technology people will need to be convinced of it’s worth and that will come with use and understanding.  David Moher wrote an editorial about the various synthesis methods being part of a family.  Is our technique the screaming baby of evidence synthesis, the eccentric uncle, the angry adolescent? You tell me….!

Automated reviews, something to show…

It’s all hands on deck here as we rush to get a robust prototype model of our automated review system ready for viewing by the EU (next week in Luxembourg).  Much of this work has been funded as part of our participation in the Horizon 2020 funded KConnect project (led by TUW, Vienna) and the EU like to see what they’re getting for their money.

Now we’ve had change to play with the data generated by our systems, two things are apparent:

  • This should, broadly, be a viable approach
  • Full-automation is still a way off (Note, we’re not attempting to reproduce manual systematic reviews).  We think the automation stage will get you ‘so far’ but it’ll still require a second level of ‘polish’ to increase the robustness.  We’re hoping this ‘polish’ stage will take no more than 5-10 minutes.

I’m going to share one image below:

In the image above we are showing a number of things, all generated automatically:

  • Each trial or systematic review is shown as a single ‘blob’.
  • Classification (x-axis) of trials based on perceived efficacy (does the drug work or not).
  • Sample size – bigger the ‘blob’ the bigger the trial.
  • Intervention name (y-axis).

What you’re not seeing is the fact that each trial is being automatically assessed for bias via RobotReviewer.

For next week we need to keep improving the data quality and – for each intervention – create a single estimate of effectiveness (our version of meta-analysis) and make it look nice!

King’s Fund will shortly be searchable on Trip

The King’s Fund collection of publications has been the most requested addition to our index over the years.  I’m very pleased to report that, as of the around the 20th September, their content will be searchable via Trip.

The King’s Fund is a UK think tank and is involved in work relating to the health system in England (mainly). It’s an important source of grey literature on policy and health systems here in the UK.

So, that’s great news.

But any other sources we should include?  If you know of any great sources of high-quality content that you think would improve Trip then PLEASE let us know.  Your suggestion could make Trip even better and result in more clinical questions being answered robustly!

Don’t be shy 🙂

Automated rapid review, we’re getting there

I recently gave an update on the progress of the system (see Automated rapid reviews).  In this I highlight the variables we’ll be able to automatically assess for each paper (RCT or systematic review):

  • P – population/disease
  • I – intervention
  • C – comparison (if there is one)
  • Sentiment – does the trial favour the intervention or not
  • Sample size – is this a large or small trial
  • Risk of Bias – via RobotReviewer, which is already on the site

As our systems processes all the articles we have to figure out how to create an output.  The outline brief I’ve suggested to our designers is:

Clearly I’m no designer! But I hope you get the picture!  The design works on two levels:

  • Top level – for a given condition each ‘blob’ will consist of a single intervention.  Size of blob will indicate the size of the evidence (based on sample size of trials), horizontal axis will represent the date of the first trial for that particular intervention, while the vertical axis will indicate likely effectiveness,
  • Second level – If a user clicks on a blob in the top level, this will be unpacked to break down each intervention in to the component trials.  Again, using similar plotting methods (sample size = size of blob, date of individual trial of horizontal access and effectiveness on the vertical).

It will look nicer and we’re exploring other visualisation techniques such as this one.

This needs to be ready by the end of September, so just over three weeks!

Automated rapid reviews

As part of the KConnect work (EU funded Horizon 2020 project) we have been doing a fair bit of work exploring the automatic extraction of various elements from RCTs and systematic reviews.  If we can automatically understand what a paper is about it can open up all sorts of avenues with regard search and evidence synthesis.

The KConnect output is virtually ready for Trip to use and it will allow us (with decent, but not perfect accuracy) the following elements from a RCT or systematic review:

  • P – population/disease
  • I – intervention
  • C – comparison (if there is one)
  • Sentiment – does the trial favour the intervention or not
  • Sample size – is this a large or small trial
  • Risk of Bias – via RobotReviewer, which is already on the site (see this post)

So, what can we do with this?  A few examples:

  • For a given condition we can identify all the trials in this area and what the interventions are.
  • We can rank the interventions on likely effectiveness
  • For a given intervention we can look at what conditions it’s been used it.
  • We could present graphic like Information is Beautiful’s Snake Oil for a given condition and/or intervention.
  • We can massively increase the coverage of our Answer Engine.

Also, all this will be fully automatic, as new trials are added to Trip they will get processed and added to the system.

We’ve got a few technical issues to go (integrating the various systems) but we are so close. You will have no idea how long I’ve fantasised about the system.  And, even though it won’t be perfect, it should stand as a very good proof of concept.

Cochrane records on Trip

Last month we reported repeated problems keeping our Cochrane records up to date and asked users to decide what we should do.  Overwhelmingly people said we should stop linking to Wiley’s Cochrane Library domain and link to PubMed.

We have now moved all the links over:

So, Trip now has all the Cochrane systematic reviews and, given our experience of PubMed, we’re confident we’ll continue to properly reflect Cochrane’s records from now on.

For those of you who miss the direct links to the Cochrane Library, you can still easily get there via PubMed:

Evidence Live: Community Rapid Review

The Community Rapid Review idea has been discussed for a while now and the final stage, before we move to production, is coming very soon.  Next week I will be running a workshop at Evidence Live on the idea.  It’ll be an interactive exploration of the thinking behind the idea and will hopefully see some final constructive criticism to guide the final product.

If you’re going to Evidence Live you can reserve a place via this link.

Medicines information coming to the Answer Engine

The Answer Engine started less than 6 months ago and has firmly established itself as a well-loved feature on Trip.  Currently, the answers are mainly linked to intervention efficacy style questions.  But, in around 4-6 weeks, we’ll be rolling out medicines information.  So, for a given drug we’ll allow users to easily see answers to questions about contraindications, warnings, interactions etc.  For instance:


Create a free website or blog at

Up ↑