Trip Database Blog

Liberating the literature

Trip, TripClick and the future of search

The Trip algorithm is great. To explain, the algorithm is the ‘behind the scenes’ way we order the results you see on the screen. As mentioned, it works great.

However, that’s not to mean it can’t be improved and we are currently working with a number of academics to try to use our data to improve search methods generally (not just Trip). We have an accompanying paper TripClick: The Log Files of a Large Health Web Search Engine. The idea is that, by using our clickstream data (what people search for, what they click on etc), machine learning techniques can be used to improve search results.

What’s particularly exciting is that we have created a competition, pitting different academic centres against each other, to see who returns the best results. Yesterday we had our first academic centre to report results:

I’m happy for a number of reasons, mainly:
  • The improvement over baseline was large
  • It was from a team headed by Prof Allan Hanbury at TU Wein, the wonderful lead of Trip’s Horizon 2020 work a few years back.

The competition is likely to run for months and after that it’s a question of taking stock and seeing how we can utilise the techniques within Trip.

If we can improve on our search results, even marginally, it’ll be a great result.

Guidelines, an exciting update

Trip prides itself on the comprehensive coverage of clinical guidelines; we’re not aware of any resource that comes close in this regard. Given the importance we have spent considerable resource trying to improve it further. One aspect has been our guideline grading project, which we hope to instigate sometime in the next 3-4 months.

More immediately, we are delighted to announce the first output from our project to mine PubMed for clinical guidelines. Up until this weekend all our guidelines were found via guideline repositories, professional website associations etc. This is great, but we’re aware that, for a number of reasons, this misses many guidelines. So, we have started a project to locate guidelines in PubMed and the first results are now available. We have focussed on national guidelines and have extracted guidelines from Japan, Poland, Brazil, France, Germany, Spain, Saudi Arabia Italy and many others.

Due to this work we have added just under 1,000 new guidelines with more to follow over the next few months.

What people are saying about the new site

We’ve had a fair bit of feedback on the new site so I thought I’d share it and respond to a few comments. And, as means of an update, we’re working on a ‘snagging’ list of things that need fixing. With a good wind, we’ll move the new site over by the end of May.

The vast majority of users found it easy or very easy to use and a similar number felt confident in using the new system although a few people fed back that they’ll need a bit of time to understand all the changes (BTW this key might help).

When we asked specifically what they liked, this was the sort of feedback:

  • Colours
  • Shift to the left-hand side of the ‘filter by evidence type’ – nice to see as it was a major worry for me!

Some specific comments being:

Pretty much all of it. Particularly the quality rating for the primary studies.”
“Beautiful and simple

When asked what they disliked:

  • ‘Nothing’ was, thankfully, a common response
  • Access to certain features (eg Latest and Greatest, LibKey) – these we’re dealing with in our ‘snagging list’
  • Evidence maps – this is not a feature we’re going to retain for now. It served a purpose, but it’s out of date, things have moved on and I’d want to reinstate it if we had the resource to do it properly!
  • Evidence pyramids – I feel your pain, it was something we gave up as part of the wider redesign ‘look and feel’

It’s not too late to give feedback yourself, please use this form.

And, finally, an important issue is that the new site has not crashed once πŸ™‚

Feedback on the new site please

Our new site is in beta version (try it here and we’re pretty close to working through the issues raised in beta-testing.

However, we want as much feedback as possible so please, after trying the new site, please complete this form to let us know what you think.

Guideline grading is getting there

We’ve been working on this for a while (eg Grading guidelines, an update). Well the grading has continued and we’ve now graded over 250 guideline producers covering the vast majority of the guideline we cover.

The grading is a score from 0-8, with 8 being the highest score (click here for an explanation of the scoring system) and the distribution is as follows:

Y-axis is the score while the X-axis is the number of producers scoring that particular score. So, around 50 producers score the maximum 8!

Nearly 50% score 7 or 8, which is encouraging. Lots also score 0 and this likely reflects the inaccessibility of methods for us to score. Their guideline might be great (seems unlikely) but if a user can’t see the methods how can they assess the ‘worth’ of the guideline?

Once we get the new site properly launched we can move on to introducing this πŸ™‚

The new site has arrived

After an enormous amount of work and testing we have released the new site in beta mode! Given the huge rewrite of code and irrespective of all the testing, we felt it was a big risk to simply replace the current/old site with the new one. So, for a short period of time they’ll run in parallel with a link from the top of the current/old site:

Alternatively you can simply go directly to the new site here:

This takes you to the homepage (this one being for Pro subscribers, free users get a green colour scheme):

And here is the Pro results page:

Where to start with the changes?

Possibly the biggest change has been the shift of the filter by evidence type from the right to the left of the results page. I was remarkably attached to the old way, but our designer convinced me of the shift.

Other than that the design has been cleaned up and there are some new icons:

Lots of other things to explore and click on (for instance the coloured ‘lozenges’ highlighting the evidence type – go on, click on one of them).

So, please go and try it now (via and let us know what you think via our feedback form. We’ve already received some feedback and they’re being worked on, thankfully most of them appear to be minor. Alternatively contact me directly:, it’s always great to hear from you!

Where have we been?

I can’t believe it, we’re heading towards the end of February and this is our first post of the year. The main reason being there has been no obvious news to reveal. However, we’ve been really busy behind the scenes with the new site. It is imminent, just not sure when exactly, we’re ironing out one outstanding issue!

When ready we will not replace the old site, we will run the two concurrently. Users going to the site will see the usual site with a link to the new site, with an invite to ‘try the beta’. Given the huge rewrite it just seems sensible to gently ease in to the new version.

Grading guidelines, an update

The original Grading guidelines post is nearly a year old and things have not moved smoothly – not only did Covid happen but also there was the rewrite of the website that took most of 2020! But we’ve not abandoned our wish to ‘grade’ guidelines to help our users with an indication of how ‘evidence-based’ the guidelines are. Our system scores each guideline producer (based on the system mentioned in the earlier post), with a maximum score of 8.

We have now graded around two-thirds of the guidelines and that figure is rapidly increasing. Our hope is to introduce the guideline system early in 2021 (but after the relaunch of the newly coded site)!

For interest we share a distribution of the guideline scores below:

28% get the maximum score of 8 with score of 0 (15%) and 7 (14%) being next most ‘popular’. So, many are being serious about producing evidence-based guidelines but also many seem seriously less inclined!! One clarification point is that a number of the zero scores are based on an inability to find any detail of the methodology employed. They may be great guidelines; they’re just not making is easy for us (or others) to find out!

Article networks, again

I love generating these network maps and I keep returning to these over the years.

The above is a map based on a sample of articles on Covid-19. Each article is represented by a node (grey circle) and the edges (lines of various colours) represent connections between them. The project was to explore ways of grouping documents with a view to speeding up evidence reviews. In this work the connections were (a) semantic and (b) citation.

What is clear is that the articles group around topics. But given the experimental nature of the work, the small sample and imperfect data I’m loathed to draw any firms conclusions but I am taking it as another endorsement of this approach, one I want to explore next year.

But how might such knowledge be useful? Here are a few and I’d be delighted to hear of any other suggestions:

  • Improve search 1 – if a user clicks on an article in a distinct cluster you can immediately highlight the closest other articles.
  • Improve search 2 – when someone searches you could highlight the distinct clusters and use it as a form of search refinement. So, using the above diagram, a user might have searched for Covid and we could highlight the three clusters.
  • Improve search 3 – a user might select 10 of the articles in a cluster but miss an article – we could flag this up.
  • Better intelligence – we could monitor the clusters and see when new articles become joined. We could then alert users who had interacted, previously, with the cluster.
  • Rapid reviews – we could highlight all the RCTs and/or systematic reviews in a cluster and start to extract value from each trial (e.g. risk of bias, sample size).

When we roll it out we will be able to include a third type of connection – clickstream data – which we’ve previously demonstrated to be incredibly powerful.Β It’s at times like this I wish we had a sizeable R&D budget


Blog at

Up ↑