Trip Database Blog

At Trip we use filters to allow users to focus on the data they want. So, users can select to just see systematic reviews, guidelines, controlled trials etc:

A long-time user of Trip approached my today to suggest a Health Economics filter would be really useful. Do you agree?

The user suggested incorporating data from the Ideas/Repec database. A further thought might be to use a search filter/hedge to identify current articles in Trip that qualify as health economic. This is a substantial undertaking so I’m keen to understand if this is a good or bad idea – please take the poll and let me know:

Full-text is really important to our users and is one of the main benefits of Trip Pro. Historically, we have checked for full-text at the time of indexing only (indexing is the the process of taking the uploaded document and making it available to a user to search).

One realisation is that many documents are restricted when they are initially released and then become free full-text after 6-24 months. So, if we only check for full-text close to the time of release we miss those that subsequently turn open access.

So, we’ve introduced a re-sampling process that will periodically check documents in Trip to see if they now have free full-text access. This has been a huge success with a huge number of new full-texts identified. We can even quantity this:

• We have 4,244,009 articles with DOIs.
• We have 3,761,834 that link to full-text.
• Overall, 88.63% of articles with a DOI (typically PubMed articles) link to full-text.

This is spectacular!

Thank you for the many hundreds who took part in this survey, it has been really helpful and will definitely guide our future engagement with AI.

Overall, 51.4% of responders were health professionals, 31.8% information specialists, 9.8% academics, leaving 7% ‘other’!

We asked 4 questions, the first 3 being:

1. Automated Q&A system: Users can ask questions in free-text format. The system would generate answers using content exclusively from Trip, explicitly mentioning the strength of the evidence and including references. How desirable is this feature for you? Please rate it on a scale from 1 to 4, with 1 being not desirable and 4 being highly desirable.
2. Semi-automated evidence review system: Users can select a review topic, and our system will find the best available evidence, extract relevant content, and present it in an evidence table. The information would be summarised and automatically updated. How desirable is this feature for you? Please rate it on a scale from 1 to 4, with 1 being not desirable at all and 4 being highly desirable
3. Better results ordering: This system would allow users to perform their initial search and then they could provide additional context explaining the reason for their search. Based on this extra information, the search results would be re-ordered (using AI) to ensure the most relevant articles appear at the top. How desirable is this feature to you? Please rate it on a scale from 1 to 4, with 1 being not desirable at all and 4 being highly desirable.

Observations:

• All ideas were popular – which is good and bad!
• The questions could have been more discerning (linked to the above point). So, instead of asking about how desirable a feature we could have offset it with highlighting potential negative aspects of the approach!
• There was little difference between the groups of responders

Our 4th question took a slightly different format:

Focus on highest quality evidence: Currently Trip generates results from all evidence types, from the highest quality secondary evidence, through to journal articles and eTextbooks. Trip’s specialism is the higher-quality evidence and it might be the main reason you visit the site. To what extent would you want to use Trip to only see results from the highest quality evidence?

Again, very positive responses (y-axis = percentage) with little difference between types of users.

Free text responses were fascinating! The main issues being:

• Lots of concern about accuracy/hallucinations and having the ability to check responses
• Control – can any AI be optional
• Reproducibility
• Transparency
• Lots of very lovely comments about how people love Trip!
• A number of very interesting ideas for new developments…!

We are delighted with the above as they are very closely aligned with our own thinking. We have been working with LLMs for many months and have a reasonable level of experience. We have also tested a few ideas out and shortly we will be meeting to discuss which elements we will be taking forward. Watch this space!