Trip Database Blog

Liberating the literature


September 2017

Automated reviews – very positive progress

NOTE: since publishing this we have a linked post Automated reviews – why?, which compliments this one!

This work has been supported via the KConnect project (a Horizon 2020 EU-funded project) and on Wednesday we had to present the work of the whole consortium at the EU in Luxembourg.  The response was overwhelmingly positive and so I wish to share a bit more of the work.

A slight bit of context; I have been involved in a number of automation projects and these were typically seen as methods to supplement the human methodology.  For instance speeding up risk of bias assessment via tools such as RobotReviewer. While these are great initiatives I wanted to explore what could be done fully automatically.

At Luxembourg we presented our ‘product’. The product is a system that automatically synthesises randomised controlled trials (and potentially systematic reviews).  It is not finished but we have a very good first effort.  I’m not sure whether to classify it as a ‘proof of concept’ or ‘alpha’ version – not sure it matters!

All the images below are based on asthma and I have deliberately blurred out the intervention names.  The reason being that we are still improving the results (more below) and I would hate to think people will be put off the system by making judgements based on a system that has yet to be optimised.

The first image is the current default result which is to present the interventions (y-axis) alphabetically while the likely effectiveness is presented on the x-axis. Note, as there are an awful lot of interventions for asthma we’ve can’t show them all – so we’re simply showing a single screen-grab, the actual graph is considerably taller!

To orientate you:

  • Each ‘blob’ represents a single intervention.  The size of the blob indicates the total population used in the trials.  So, a big blob indicates more participants in the various trials.
  • The horizontal positioning is based on our estimate of effectiveness (not effect size) and the further right it is the more effective the system estimates the intervention to be.  This is further indicated by the colours – green being better and red worse (traffic light colouring)!
  • Above the graph there is the ability to sort the interventions by number of trials, sample size, score etc.
  • The sample size refinement allows users to exclude trials that are below the size entered – as we know small trials tend to be less reliable.
  • The risk of bias allows you to automatically remove trials that are not considered low risk of bias – so another reliability measure.

This second graph shows the results arranged by effectiveness:

And, to reiterate, this is fully automatic and always up to date.  As new RCTs are published (PubMed only at present) they will be automatically added.  To me that’s incredibly cool.

While I would love to share this more widely there is still some work to do before I’m happy to open it up.  In internal testing we have identified a number of areas that could, realistically, be improved.  Nothing major, we’re just being sure we’ve done as much as we can – under the fully automatic banner – to ensure the biggest impact.

As far as I know, this is the first fully automated evidence synthesis/review tool. This is such a disruptive bit of technology people will need to be convinced of it’s worth and that will come with use and understanding.  David Moher wrote an editorial about the various synthesis methods being part of a family.  Is our technique the screaming baby of evidence synthesis, the eccentric uncle, the angry adolescent? You tell me….!

Automated reviews, something to show…

It’s all hands on deck here as we rush to get a robust prototype model of our automated review system ready for viewing by the EU (next week in Luxembourg).  Much of this work has been funded as part of our participation in the Horizon 2020 funded KConnect project (led by TUW, Vienna) and the EU like to see what they’re getting for their money.

Now we’ve had change to play with the data generated by our systems, two things are apparent:

  • This should, broadly, be a viable approach
  • Full-automation is still a way off (Note, we’re not attempting to reproduce manual systematic reviews).  We think the automation stage will get you ‘so far’ but it’ll still require a second level of ‘polish’ to increase the robustness.  We’re hoping this ‘polish’ stage will take no more than 5-10 minutes.

I’m going to share one image below:

In the image above we are showing a number of things, all generated automatically:

  • Each trial or systematic review is shown as a single ‘blob’.
  • Classification (x-axis) of trials based on perceived efficacy (does the drug work or not).
  • Sample size – bigger the ‘blob’ the bigger the trial.
  • Intervention name (y-axis).

What you’re not seeing is the fact that each trial is being automatically assessed for bias via RobotReviewer.

For next week we need to keep improving the data quality and – for each intervention – create a single estimate of effectiveness (our version of meta-analysis) and make it look nice!

King’s Fund will shortly be searchable on Trip

The King’s Fund collection of publications has been the most requested addition to our index over the years.  I’m very pleased to report that, as of the around the 20th September, their content will be searchable via Trip.

The King’s Fund is a UK think tank and is involved in work relating to the health system in England (mainly). It’s an important source of grey literature on policy and health systems here in the UK.

So, that’s great news.

But any other sources we should include?  If you know of any great sources of high-quality content that you think would improve Trip then PLEASE let us know.  Your suggestion could make Trip even better and result in more clinical questions being answered robustly!

Don’t be shy 🙂

Blog at

Up ↑