Trip Database Blog

Systematic reviews are an important component of evidence-based medicine. Over the years we have attempted to support our users by finding as many systematic reviews as possible. Recently we have been lucky enough to work with a number of organisations and start-ups who have helped us find more. And, I’m delighted to say that, as of today, we have 564,350 systematic reviews in Trip.

We compared our coverage with a number of other databases, for example PubMed, and we consistently have more. To compare we used title searches (to overcome the differences between searching mechanisms between databases – something that shouldn’t affect title searches) and here are some examples:

Zinc

• 528 results for Trip
• 286 results for PubMed (using SR filter)
• 19 in the Cochrane Library

 Cancer screening

• 1890 for Trip,
• 646 for PubMed
• 19 for Cochrane

One advantage Trip has is that we also include health technology assessments (HTAs). These are often ‘grey’ and therefore don’t appear in most databases (which typically rely on journal publications).

One final thought, having more systematic reviews is something we’re pleased about, but it’s only part of the story. We introduced our guideline scoring system as many guidelines were not evidence-based and we want to help our users understand this fact. The same is true with systematic reviews, some are better than others. So, we’re restarting our work on automatically assessing the quality of systematic reviews. From our previous work (see here) we had a good system, not a great one. With the advent of LLMs we should be able to improve things considerably – watch this space.

We have now released the mis-spelling feature. It’s pretty simple:

Simply click on the suggested spelling and it repeats the search with the corrected term(s). It can handle multiple terms as well:

Many years ago Trip had a mis-spelling feature – helping users correct mis-spelt search terms. This post is from 2006 (I hope the quality of our blog posts have improved since then)! In it we highlight that hypertension and diabetes were most often mis-spelt and here are the dodgy spellings:

Clearly people struggle with spelling yet, for some unknown reason, we dispensed with this feature. No-one complained and therefore we haven’t tried to replace it. However, even though non-one has complained, we’re going to re-introduce this feature in the next month or so. I’m quite excited by this – another step in making Trip better.

At Trip we use filters to allow users to focus on the data they want. So, users can select to just see systematic reviews, guidelines, controlled trials etc:

A long-time user of Trip approached my today to suggest a Health Economics filter would be really useful. Do you agree?

The user suggested incorporating data from the Ideas/Repec database. A further thought might be to use a search filter/hedge to identify current articles in Trip that qualify as health economic. This is a substantial undertaking so I’m keen to understand if this is a good or bad idea – please take the poll and let me know:

Full-text is really important to our users and is one of the main benefits of Trip Pro. Historically, we have checked for full-text at the time of indexing only (indexing is the the process of taking the uploaded document and making it available to a user to search).

One realisation is that many documents are restricted when they are initially released and then become free full-text after 6-24 months. So, if we only check for full-text close to the time of release we miss those that subsequently turn open access.

So, we’ve introduced a re-sampling process that will periodically check documents in Trip to see if they now have free full-text access. This has been a huge success with a huge number of new full-texts identified. We can even quantity this:

• We have 4,244,009 articles with DOIs.
• We have 3,761,834 that link to full-text.
• Overall, 88.63% of articles with a DOI (typically PubMed articles) link to full-text.

This is spectacular!

Thank you for the many hundreds who took part in this survey, it has been really helpful and will definitely guide our future engagement with AI.

Overall, 51.4% of responders were health professionals, 31.8% information specialists, 9.8% academics, leaving 7% ‘other’!

We asked 4 questions, the first 3 being:

1. Automated Q&A system: Users can ask questions in free-text format. The system would generate answers using content exclusively from Trip, explicitly mentioning the strength of the evidence and including references. How desirable is this feature for you? Please rate it on a scale from 1 to 4, with 1 being not desirable and 4 being highly desirable.
2. Semi-automated evidence review system: Users can select a review topic, and our system will find the best available evidence, extract relevant content, and present it in an evidence table. The information would be summarised and automatically updated. How desirable is this feature for you? Please rate it on a scale from 1 to 4, with 1 being not desirable at all and 4 being highly desirable
3. Better results ordering: This system would allow users to perform their initial search and then they could provide additional context explaining the reason for their search. Based on this extra information, the search results would be re-ordered (using AI) to ensure the most relevant articles appear at the top. How desirable is this feature to you? Please rate it on a scale from 1 to 4, with 1 being not desirable at all and 4 being highly desirable.

Observations:

• All ideas were popular – which is good and bad!
• The questions could have been more discerning (linked to the above point). So, instead of asking about how desirable a feature we could have offset it with highlighting potential negative aspects of the approach!
• There was little difference between the groups of responders

Our 4th question took a slightly different format:

Focus on highest quality evidence: Currently Trip generates results from all evidence types, from the highest quality secondary evidence, through to journal articles and eTextbooks. Trip’s specialism is the higher-quality evidence and it might be the main reason you visit the site. To what extent would you want to use Trip to only see results from the highest quality evidence?

Again, very positive responses (y-axis = percentage) with little difference between types of users.

Free text responses were fascinating! The main issues being:

• Lots of concern about accuracy/hallucinations and having the ability to check responses
• Control – can any AI be optional
• Reproducibility
• Transparency
• Lots of very lovely comments about how people love Trip!
• A number of very interesting ideas for new developments…!

We are delighted with the above as they are very closely aligned with our own thinking. We have been working with LLMs for many months and have a reasonable level of experience. We have also tested a few ideas out and shortly we will be meeting to discuss which elements we will be taking forward. Watch this space!

If you are interested in adding a TRIP search box to your website then feel free to use the code below. This adds a great feature to your site and once there a search opens into a new window, meaning you don’t lose your users!

<form method="get" action="https://www.tripdatabase.com/search" style="border: 1px solid #9B82B4; font-family: Verdana, Helvetica, sans-serif; padding: 5px; width:200px">
<svg id="Layer_1" data-name="Layer 1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 165.17 104.41">
  <g>
    <path d="M59.36,9.66V6.2Q59.36.62,56,.62H3.34Q0,.62,0,6.2V9.66q0,5.56,3.34,5.57H21.05v61.9H38.31V15.23H56q3.34,0,3.34-5.57M78.13,21.79q-7.47,0-12.14,8.48a17,17,0,0,1-.39-1.79,17.79,17.79,0,0,0-.5-2.06c-.15-.41-.38-.93-.67-1.56a3.23,3.23,0,0,0-1.17-1.4A6.56,6.56,0,0,0,60.31,23a38.78,38.78,0,0,0-6.74.89q-4.51.9-4.51,2.68a32.78,32.78,0,0,0,.67,4,69.65,69.65,0,0,1,.67,11.71V77.13H66.88V41.87c2.23-3.72,5.19-5.58,8.91-5.58a11.35,11.35,0,0,1,2.89.45,11.28,11.28,0,0,0,2.34.44c1,0,1.79-1.17,2.45-3.51a22.09,22.09,0,0,0,1-5.86,6.75,6.75,0,0,0-1.4-4.18c-.93-1.22-2.58-1.84-5-1.84m81.25,8q-5.79-8-15-8a17.23,17.23,0,0,0-9,2.34,15.67,15.67,0,0,0-6,6,16.76,16.76,0,0,1-.39-1.68,19.62,19.62,0,0,0-.5-2,14.55,14.55,0,0,0-.67-1.62,3.23,3.23,0,0,0-1.17-1.4,6.53,6.53,0,0,0-3-.44,38.78,38.78,0,0,0-6.74.89q-4.52.9-4.51,2.68a32.78,32.78,0,0,0,.67,4,69.65,69.65,0,0,1,.67,11.71V99.49h16.48v-21a18.85,18.85,0,0,0,11.36,3.46q11.13,0,17.32-8.09t6.18-22.08q0-14-5.79-22M138.11,68.85a15.68,15.68,0,0,1-7.8-2.12V41.87q3.78-6.59,9.13-6.58,9,0,9,16.78T138.11,68.85M99.08,0H96c-4.46,0-6.68,1.19-6.68,3.57v8.11h16.48V3.57Q105.76,0,99.08,0m0,22.85H96c-4.46,0-6.68,1.19-6.68,3.57V77.13h16.48V26.42q0-3.57-6.68-3.57" fill="#533764" fill-rule="evenodd"></path>
    <path d="M37.33,77.53v.58Q37.33,82,31,82H27.26q-6.36,0-6.36-3.9v-.58Z" fill="#63c608" fill-rule="evenodd"></path>
    <path d="M67.19,77.53v.58q0,3.9-6.66,3.9h-3.1q-6.66,0-6.66-3.9v-.58Z" fill="#0e6cbb" fill-rule="evenodd"></path>
    <path d="M106,77.53v.92c0,2.38-2.22,3.56-6.65,3.56H96.25c-4.44,0-6.66-1.18-6.66-3.56v-.92Z" fill="#00a89d" fill-rule="evenodd"></path>
    <path d="M130.7,100.06v.9q0,3.45-6.55,3.45h-3.06q-6.56,0-6.55-3.45v-.9Z" fill="#eec82f" fill-rule="evenodd"></path>
    <path d="M106,11.83v.92c0,2.38-2.22,3.56-6.65,3.56H96.25c-4.44,0-6.66-1.18-6.66-3.56v-.92Z" fill="#ba390d" fill-rule="evenodd"></path>
  </g>
</svg><br />
<input type="text" name="criteria" style="width:150px" />
<input type="submit" value=" Go " />
</form>

We’ve been experimenting with AI for around a year and are confident enough with it to start to use it! However, a big uncertainty is how best to use it? By that I mean the technology is useless unless it supports our users! So, how best to support you?

We have created a short survey to help inform our decision making. Your opinion really matters and we really do listen. So, please, click here now to do the survey.

Thank you!

Trip’s search is sensitive! If a user searches for ‘measles’ and the term is mentioned one time in a document of over 100,000 words, it is still returned as a result. This is not normally an issue if there are lots of results. However, if you select a facet with relatively few results (e.g. UK guidelines) then these very low relevancy results appear.

A real example, if you do a search for measles and go into European guidelines, there are 12 results. These are results 7-12:

• ESPEN expert statements and practical guidance for nutritional management of individuals with sars-cov-2 infection
• Autologous haematopoietic stem cell transplantation and other cellular therapy in multiple sclerosis and immune-mediated neurological diseases
• Hepatitis E Virus Infection
• Autoimmune Hepatitis
• Guideline on the Diagnosis and Treatment of Sclerosing Diseases of the Skin
• ECCO-ESGAR Guideline for Diagnostic Assessment in IBD Part 1: Initial diagnosis, monitoring of known IBD, detection of complications

Result 7, ESPEN expert statements and practical guidance for nutritional management of individuals with sars-cov-2 infection mentions measles 4 times in over 8,000 words. To us, this article is not about measles!

This issue has been frustrating us, and some of our users, for years. But we’ve now made a big step forward in removing low relevancy results from Trip. This feature has been released today – for Pro users only. Repeating the search for measles we now get 6,681 results overall and 4 European guidelines, previously it was 7,764 results and 12 European guidelines.

Overall, there was a 14% reduction in total results, but in European guidelines the reduction was nearly 67%. This is to be expected as guidelines are typically much longer documents and therefore have more scope for mentions of low relevancy terms.

And, if you don’t like this you can revert to the full results by a link at the foot of the results page:

I suspect this feature will not be noticed by many but it should dramatically change the quality of results in some situations. I am delighted to see this feature, it has bothered me for many years.