Trip Database Blog

Liberating the literature



The evidence pyramid

A great explanation of the evidence pyramid (taken from this Walden University site):

We use this concept in Trip to help users navigate the results (helping indicate the likely reliability of the evidence):

The main place to see it is slightly to the right of the result.  Here you see the pyramid, a colour-banded representation of where the evidence lies in the pyramid and a phrase describing the content (e.g. systematic review, guideline etc). You’ll note the colour coding which can be found to the left of each result and it is carried through to the ‘Refine by’ area.

We’ve just had a large usability study of the site and this is what the report states:

  • Users from research or information management backgrounds understood the pyramid icon, but users who were new to it wondered what it was.
  • When they figured out what it was, or I talked them through it, they appreciated it and it appeared to add to their experience.
  • Recommend adding a mouse-over explanation or some kind of ‘Introducing the pyramid of evidence’ box somewhere, as part of onboarding or in help.

Bottom line: it’s useful but only when you know what it means!  With no understanding of the concept it’s just confusing.

I need to go and sit on the ‘naughty step’ and contemplate why I fell in to the trap of assuming users know all this sort of stuff (oh yes, and to fix it).



Top oncology articles

From the last month, the top ten most viewed oncology articles on Trip:

  1. Screening for breast cancer with mammography. Cochrane
  2. Interventions for the treatment of oral and oropharyngeal cancers: targeted therapy and immunotherapy. Cochrane
  3. A systematic review and economic evaluation of intraoperative tests [RD-100i one-step nucleic acid amplification (OSNA) system and Metasin test] for detecting sentinel lymph node metastases in breast cancer. NIHR HTA
  4. Aspirin use for the primary prevention of cardiovascular disease and colorectal cancer: U.S. Preventive Services Task Force recommendation statement
  5. Multiple myeloma. BMJ Best Practice
  6. Guideline Summary: Oncology evidence-based nutrition practice guideline. Academy of Nutrition and Dietetics
  7. Parental decision making about the HPV vaccine. Cancer Epidemiology & Biomarkers and Prevention
  8. Immunotherapy for advanced renal cell cancer. Cochrane
  9. Cancer and Rehabilitation (Treatment). eMedicine
  10. Cervical cancer and HPV. NICE CKS

Two stand out articles in the list for me – 7 and 9.  All the other articles are ‘secondary evidence’.

In the case of 7 it’s a primary research article and it’s from 2010. So, as well as being primary research, it’s also relatively old. This is significant as it potentially links in with our efforts to gather uncertainties (to improve research procurement – both primary and secondary research).  We’re always looking to analyse click patterns to see if we can unearth uncertainties but finding the rights signals is problematic.  The fact that a significant number of users have an information need met by an oldish article from down the evidence pyramid suggests it’s a ‘hot topic’. Perhaps one for a systematic review or further research?

Number 9 is an eTextbook – the lowest level of evidence (in the Trip hierarchy).  Again, lots of interest and users are not finding their information need met by higher quality research.

I’m excited by this ‘discovery’ one to discuss with my research funding colleagues!

Highlighting new research: the results

Last week I asked users from primary care to rate a series of new articles added to Trip.  We asked them to let us know which of 12 articles they thought looked interesting/important.  With 342 votes the results are in:

  1. Systematic literature review on effectiveness of self-management support interventions in patients with chronic conditions and low socio-economic status. EvidenceUpdate
  2. Shared decision making in primary care can reduce antibiotic prescribing. NIHR Dissemination Centre
  3. Multimorbidity and polypharmacy. NICE, advice
  4. Extended hours in primary care linked to reductions in minor A&E attendances. NIHR Dissemination Centre
  5. Antimicrobial stewardship: changing risk-related behaviours in the general population. NICE, Clinical Guideline
  6. Capsaicin 8% Patch in Painful Diabetic Peripheral Neuropathy: A Randomized, Double-Blind, Placebo-Controlled Study. EvidenceUpdates
  7. Support at Home: Interventions to Enhance Life in Dementia (SHIELD) evidence, development and evaluation of complex interventions. NIHR HTA
  8. Randomised controlled trial and economic analysis of an internet-based weight management programme: POWeR+ (Positive Online Weight Reduction). NIHR HTA
  9. Cutaneous melanoma. SIGN Guideline
  10. Nitric oxide breath analysis for the management of asthma. HTA Database
  11. Long-Term Continuous Ambulatory ECG Monitors and External Cardiac Loop Recorders for Cardiac Arrhythmia. Health Quality Ontario
  12. Canadian Youth Perceptions on Cannabis. Canadian Centre on Substance Abuse

Presented graphically:

Systematic literature review on effectiveness of self-management support interventions in patients with chronic conditions and low socio-economic status won by a single vote.

What was most interesting to me was that it still only obtained 13% of the votes (while those towards the bottom received only 4%).  So, only around 1 in 8 people voted for it.  I was expecting higher scores.  Perhaps that’s a reflection on the diversity of primary care and the diversity of our users.  Perhaps one to test in a different speciality!

Helping highlight the most important new documents on Trip: Primary care

This is an experiment to see if Trip users find it helpful and interesting to try and highlight which article, aimed at primary care, is of most use.  Below are a number of articles that we have identified as being high quality and suitable for primary care.  But the question is are they of use and/or interesting?  Can you look through the list and tick those that you think meet the criteria of being interesting or useful.

We’ll post the results in the very near future.  This will show if people enjoy rating articles and also if there are clear favourites!  And, if it works for primary care, we’ll explore other areas as well.

What we’re working on

The new upgrades have been well received and appear stable (phew).  But we’re not resting and we’re working on a  number of projects; the main ones being:

  • Learning to rank: The search algorithm decides what order the results appear in Trip.  Ours has been unchanged for many years and this shows how good it is.  However, technology has moved on over the years and one of the most exciting developments has been the technique known as learning to rank (LTR or L2R).  This uses machine learning to help understand what attributes of a result make them most likely to be clicked.  It then uses this to rank articles on the likelihood of being clicked.  This will be properly tested using a technique known as interleaving.  Timescale: We’re hoping to start testing in the next 2-3 months.
  • Community rapid review: Early days in the plans but there has been considerably enthusiasm from Trip users wanting to get involved.  An outline can be seen here while an important aspect, an educational environment, is also an important consideration (click here). Timescale: Finalised outline by July with system released by the end of the year.
  • Automated rapid reviews: Is it possible to create automatic reviews?  We think so and have started out exploring this as an option.  While not as detailed as the semi-manual community rapid review system it will still be really useful and in ways that many will not expect.  In addition, the methods used lend themselves to visualisations which will further enhance their impact. Timescale: By the end of the year, hopefully sooner.
  • Strategy: I have started reviewing the next steps of Trip with an external strategy expert.  It’s not been the most comfortable experiences (strategy is not my favoured environment), but I feel we’re making real progress. Timescale: Within 3-4 months.

At the same time we’ll continue to focus on the quality of the content and our new manual broken link system’s really useful in that respect.  We’re also working through our major providers of content to completely refresh their links. Last month was the National Guideline Clearinghouse and this month was Cochrane and CKS.

The above seems lovely an ambitious and we’re loving it!


What’s new for early 2017

We’re really excited by the release of the latest upgrade to the Trip Database.  There are a number of major upgrades to the site:

The Answer Engine: delivers answers to your clinical questions, quickly, with a single click!  For more information see this post for more details.



SmartSearch: an amazingly useful system to help speed up your search and to help users avoid missing important documents.  More information can be seen here.



Search suggestions: a pre-existing system that has been integrated much more intimately with your search experience.  Now appearing mid-way down the page; designed to prompt users to use it if the initial results aren’t focused enough.



Broken links: broken links are problematic for any search system.  We have an automated system which works reasonably well, but not perfectly.  Our new manual system should allow users to easily let us know if one of our links to a dead end!  Take our your frustration by clicking the link!



Other improvements: there are a number of other, minor changes:

  • We have improved the display and integration of the automatic assessment of bias via the RobotReviewer system.  For details of this automated system see this article.
  • We have tidied up the top of the results making it all much cleaner.


SmartSearch is an amazingly useful tool designed to help you avoid missing any important articles you may have missed. Or, it can be seen as a tool that speeds up your searching!

After you’ve searched our systems monitor the articles you’ve clicked to give us a better idea of your specific intention for this particular search.  If our system find articles it believes are relevant it pops an alert below the last article you clicked on:



Clicking on that reveals your suggested results:



SmartSearch is powered by our unique clickstream data.  In that respect SmartSearch is based on the collective wisdom of Trip users.  Or, to paraphrase the Isaac Newton quote, you’ll be ‘standing on the shoulders of giants‘.

Pro users get full access to this feature while free users use is restricted.  Users can upgrade here. Institutional subscriptions are also available (click here).

The Answer Engine

Many year ago Muir Gray stated – in relation to the access to evidence – that ‘Three clicks are two clicks too many’.  This statement inspired the invention of the Trip Answer Engine and we have delivered on Muir’s vision.  Answers to clinical questions, quickly, with a single click!

So, what is the answer engine?


The answer engine explores the search terms used to try to understand the intention.  In this case the likely interest of the user is ‘Is minocycline useful in the treatment of acne?‘.  Our system then finds the best available answer to that question; in this case a Cochrane Systematic Review.

Incredibly simple (conceptually) powered by some incredibly powerful/clever/magical systems behind the scenes!

Currently, the system is semi-automated and will answer around 1000 of the most common questions asked of Trip.  We hope to move to a fully automated system in mid-2017 and that will then answer hundreds of thousands of questions.


2016 – the impact of Trip

For the first time we have breached 5 millions minutes of usage on the site.  Granted a weird metric, but it just stands out!  As does the over 1 million separate sessions on the site.

Lots of big numbers, but what was the impact?  We were searched around 7.8 million times in 2016.  We estimate, from previous work, that 40.8% of searches help improve patient care (NOTE: see our post on why this figure might underestimate and/or overestimate the impact).

So, 40.8% of 7.8 million indicates the number of times Trip has improved patient care.  Therefore we can happily/proudly announce:

3,182,400 times!

That’s a huge number.


Blog at

Up ↑