Trip Database Blog

I’ve been preparing a talk for ESSIR and it’s been great thinking through some information retrieval problems to talk about. Two of the are linked to quality and not quantity of evidence. If you do a search and find 1000 results, most people will find that 950 too many… So, it’s important that the top results are the best. Our algorithm attempts to do it and it does it pretty well, but it could be better.

I found myself looking at the search for prostate cancer screening which generated over 12,000 results. Of the top ten, 6 are ‘evidence based’ (ie the top green layer) while 4 are journal articles (these get boosted as they’re published in the last two years). This struck me as a bit odd as we emphasise the secondary evidence side of things… So, what if we did things differently:

• A search for prostate cancer screening, restricted to the title words and restricted to just secondary evidence = 119 results. If you add in ‘Key primary research that jumps to 167 results.
• As above but restrict to the last 5 years and it’s 35 results.
• A general search for prostate cancer screening, organised by relevancy, restricted to secondary evidence and the last 5 years = 1,352 results, restricted to the last 3 years and it’s 994 results.

In the above examples the search is very specific, removes the lower quality evidence and the user loses seeing some potential journal articles – which they can navigate to if they wish.

For information specialists – who typically want all results – we could allow that. But for busy clinicians we could allow some way of only returning a fraction of the results with a focus on higher quality evidence.

Discuss!

Amazingly, in around a fortnight, we have created a rudimentary ACQA system and the initial results are both good and bad! We’ve run through a number of questions and some have been pretty good and others pretty bad. An example of a good ACQA was Is brexpiprazole effective in schizophrenia? with our system generating the following answer:

“Brexpiprazole has shown effectiveness in treating schizophrenia based on a post hoc analysis of various short- and long-term studies. These studies demonstrated that brexpiprazole led to clinically relevant improvements in functioning for patients with schizophrenia in both short and long durations. Patients on brexpiprazole experienced greater improvements in Personal and Social Performance (PSP) scores and all PSP domains compared to those on placebo. The Global Assessment of Functioning (GAF) scale also indicated better functional remission for brexpiprazole-treated patients. These improvements were noticeable during the treatment of acute schizophrenia and generally persisted over a one-year period. However, low-dose brexpiprazole might not be as effective as standard-dose brexpiprazole in improving efficacy and safety for acute schizophrenia, potentially carrying a risk of weight gain. Additionally, a study highlighted that brexpiprazole treatment reduced impulsive behavior and associated brain activation in patients with schizophrenia. Overall, brexpiprazole appears to be effective in managing symptoms and improving functioning in schizophrenia.“

I’m not saying it’s perfect but pretty good for a first attempt. This answer was all referenced and mostly created from secondary evidence. However, a worse answer was generated for Is zolmitriptan effective in migraines?, which generated this answer:

“Zolmitriptan, a medication commonly used for migraines, belongs to the triptan class of drugs. It works by constricting blood vessels in the brain and reducing inflammation. Studies have shown its effectiveness in relieving migraine symptoms such as headache, nausea, and sensitivity to light and sound. It can provide relief within two hours of taking it. However, its effectiveness may vary and it may have side effects. Consultation with a healthcare professional is important. The provided text does not directly address the effectiveness of zolmitriptan in migraines.“

But understanding the process and why some answers are so poor have given us a load of learning, some easy to respond to but others less so. Currently, the biggest struggle is transforming the question into appropriate search terms. We’ll continue to use LLMs to try to solve this but an easier route might be to include the users more and make it semi-automatic. In many way I favour this, it gives users more understanding of the process and it’s less ‘black box’. It might add an extra step but the extra reassurance might be worth it.

Early days and all to play for!