We had an email from a user asking if the search “do not resuscitate” (as a phrase) was possible. I was curious as – surely – it works. Unfortunately I was wrong. The issue being the use of a Boolean term within a phrase – in this case ‘not’. Via the advanced search I used this search:
And the system interpreted the result as:
Odd and, even though it’s a fringe example, it needed exploring. Well, we’ve fixed it – mostly! If you now do the search the results look like this:
But why do I say it’s fixed ‘mostly’? Well, another quirk of the system kicks in – in this case synonyms. In our system we have the following terms as synonyms:
DNR
DNAR
do not attempt resuscitation
do not resuscitate
So, when you do the search for “do not resuscitate” it also searches for the other synonyms as well, so you get results like this:
So, some “do not attempt resuscitation” in there. These are not the exact phrase the user searched for but there is no way to get round this unless we remove the synonyms in our system. It seems to me to be the lesser of two evils to allow these synonyms in the search. Feel free to tell me I’m wrong 🙂
Helping users keep up to date with the latest evidence is difficult and our latest project is our attempt to help. Starting with primary care we’ve released a ‘digest’ with summaries of the latest relevant evidence. To see the digest click here.
Here’s how it works:
We take a long list of articles we tag as ‘primary care’ as part of our regular updating of the evidence.
The long list is sent to an experienced clinician who specialises in primary care (in this case Chris, the general practitioner, who is the medical director of Trip) who selects around 20 articles he feels are most relevant and newsworthy.
These are then fed into an LLM with a prompt to summarise and give a clinical bottom line.
We then ask the LLM to write an editorial based on the included documents.
Finally the digest is published.
This approach is semi-automatic but it could feasibly be fully-automatic, and we’ll investigate that if the digest proves popular. If it does we’ll also expand into other clinical areas and, who knows, one day produce personalised digests.
Please take a look – click here – and let us know what you think.
There is a huge amount of research and evidence published daily, far too much to keep up with. This is an issue we, at Trip, have wrestled with for years. With the advent of LLMs we’re experimenting with a new approach.
Using Primary Care as a launch pad we’re creating a ‘Latest evidence’ review (or is it a digest?). Here are some screengrabs of our test:
Note the ‘September 2024’ which indicates it’ll be monthly. On the left hand side is an ‘editorial’ (LLM generated) and on the right is a list of articles we’re covering. If you scrolled down further you get:
We’re displaying a summary (LLM generated) and a link to the article.
Currently this is semi-automated and when we release it we’ll run it for a few months to see the reaction. If it’s favourable we’ll almost fully-automate it and make it available for multiple clinical areas e.g. oncology, cardiology, rheumatology. etc
Given the focus on quality at Trip we will only report high-quality evidence, much of which is ‘grey‘, hence not published in journals and therefore less likely to be seen. Given the lack of visibility for much of the content it makes this sort of promotion really important; let’s see how this approach is received by our users…!
Systematic reviews are an important component of evidence-based medicine. Over the years we have attempted to support our users by finding as many systematic reviews as possible. Recently we have been lucky enough to work with a number of organisations and start-ups who have helped us find more. And, I’m delighted to say that, as of today, we have 564,350 systematic reviews in Trip.
We compared our coverage with a number of other databases, for example PubMed, and we consistently have more. To compare we used title searches (to overcome the differences between searching mechanisms between databases – something that shouldn’t affect title searches) and here are some examples:
Zinc
528 results for Trip
286 results for PubMed (using SR filter)
19 in the Cochrane Library
Cancer screening
1890 for Trip,
646 for PubMed
19 for Cochrane
One advantage Trip has is that we also include health technology assessments (HTAs). These are often ‘grey’ and therefore don’t appear in most databases (which typically rely on journal publications).
One final thought, having more systematic reviews is something we’re pleased about, but it’s only part of the story. We introduced our guideline scoring system as many guidelines were not evidence-based and we want to help our users understand this fact. The same is true with systematic reviews, some are better than others. So, we’re restarting our work on automatically assessing the quality of systematic reviews. From our previous work (see here) we had a good system, not a great one. With the advent of LLMs we should be able to improve things considerably – watch this space.
Many years ago Trip had a mis-spelling feature – helping users correct mis-spelt search terms. This post is from 2006 (I hope the quality of our blog posts have improved since then)! In it we highlight that hypertension and diabetes were most often mis-spelt and here are the dodgy spellings:
Clearly people struggle with spelling yet, for some unknown reason, we dispensed with this feature. No-one complained and therefore we haven’t tried to replace it. However, even though non-one has complained, we’re going to re-introduce this feature in the next month or so. I’m quite excited by this – another step in making Trip better.
At Trip we use filters to allow users to focus on the data they want. So, users can select to just see systematic reviews, guidelines, controlled trials etc:
A long-time user of Trip approached my today to suggest a Health Economics filter would be really useful. Do you agree?
The user suggested incorporating data from the Ideas/Repec database. A further thought might be to use a search filter/hedge to identify current articles in Trip that qualify as health economic. This is a substantial undertaking so I’m keen to understand if this is a good or bad idea – please take the poll and let me know:
Full-text is really important to our users and is one of the main benefits of Trip Pro. Historically, we have checked for full-text at the time of indexing only (indexing is the the process of taking the uploaded document and making it available to a user to search).
One realisation is that many documents are restricted when they are initially released and then become free full-text after 6-24 months. So, if we only check for full-text close to the time of release we miss those that subsequently turn open access.
So, we’ve introduced a re-sampling process that will periodically check documents in Trip to see if they now have free full-text access. This has been a huge success with a huge number of new full-texts identified. We can even quantity this:
Thank you for the many hundreds who took part in this survey, it has been really helpful and will definitely guide our future engagement with AI.
Overall, 51.4% of responders were health professionals, 31.8% information specialists, 9.8% academics, leaving 7% ‘other’!
We asked 4 questions, the first 3 being:
Automated Q&A system: Users can ask questions in free-text format. The system would generate answers using content exclusively from Trip, explicitly mentioning the strength of the evidence and including references. How desirable is this feature for you? Please rate it on a scale from 1 to 4, with 1 being not desirable and 4 being highly desirable.
Semi-automated evidence review system: Users can select a review topic, and our system will find the best available evidence, extract relevant content, and present it in an evidence table. The information would be summarised and automatically updated. How desirable is this feature for you? Please rate it on a scale from 1 to 4, with 1 being not desirable at all and 4 being highly desirable
Better results ordering: This system would allow users to perform their initial search and then they could provide additional context explaining the reason for their search. Based on this extra information, the search results would be re-ordered (using AI) to ensure the most relevant articles appear at the top. How desirable is this feature to you? Please rate it on a scale from 1 to 4, with 1 being not desirable at all and 4 being highly desirable.
Observations:
All ideas were popular – which is good and bad!
The questions could have been more discerning (linked to the above point). So, instead of asking about how desirable a feature we could have offset it with highlighting potential negative aspects of the approach!
There was little difference between the groups of responders
Our 4th question took a slightly different format:
Focus on highest quality evidence: Currently Trip generates results from all evidence types, from the highest quality secondary evidence, through to journal articles and eTextbooks. Trip’s specialism is the higher-quality evidence and it might be the main reason you visit the site. To what extent would you want to use Trip to only see results from the highest quality evidence?
Again, very positive responses (y-axis = percentage) with little difference between types of users.
Free text responses were fascinating! The main issues being:
Lots of concern about accuracy/hallucinations and having the ability to check responses
Control – can any AI be optional
Reproducibility
Transparency
Lots of very lovely comments about how people love Trip!
A number of very interesting ideas for new developments…!
We are delighted with the above as they are very closely aligned with our own thinking. We have been working with LLMs for many months and have a reasonable level of experience. We have also tested a few ideas out and shortly we will be meeting to discuss which elements we will be taking forward. Watch this space!
Trip Database Blog 
Recent Comments