Trip Database Blog

We have just started testing our automated Q&A system mentioned here. In short you ask a free-text question and it looks over Trip’s content to generate the best answer. Our medical director’s first tests were very encouraging so we’ve thrown it open to our AI Innovation Circle for additional testing. The initial questions from the ‘Circle’ being:

• Which decreases Migraines more: Magnesium, Zinc, Vitamin D or Omega-3?
• Which decreases Cluster Headaches more: Magnesium, Zinc, Vitamin D or Omega-3?
• Which combination of Magnesium, Zinc, Vitamin D or Omega-3 has been found to decrease headaches?
• How can cluster headaches be quickly stopped?
• What is the best treatment for a migraine?
• What is the QUICKEST treatment for a migraine?
• How does the combination of acetaminophen + caffiene + aspirin for migraine treatment compare to prescription treatments for migraine?
• How do corticosteroids compare to other treatments for preventing migraine recurrence?

We have sent these Qs to our system and have generated two answers per question, one using ChatGPT-4 and the other version 3.5 (ChatGPT describes the difference between the two as “ChatGPT-4 is an advanced iteration of OpenAI’s language model that offers improved understanding and generation of text, enhanced contextual awareness, and greater nuance in responses compared to its predecessor, ChatGPT-3.5.“). Each response takes around 10-15 seconds to generate and, really importantly, these answers are referenced.

We’ve sent the answers back to the people who asked them and we’re awaiting feedback. We know that some of them are suboptimal but we also know how we can improve things. That, ultimately, is the purpose of this proof of concept project – can a Q&A system work based on Trip’s content? Based on our work to date we are absolutely confident that we can make a robust system. Once this project is finished being tested we will take the learning and combine it with the learning from the second project (fully automated evidence reviews) and decide if/how to operationalise one or both of them…

2023 was a great year for Trip.

We continue to have a impressive impact on care around the globe, helping support millions of decisions with evidence-based content. This is Trip’s core function, as such, it’s ‘business as usual’. However, we have continued to improve the site as much as we can and 2023 was full of significant improvements.

First quarter of 2023: A real focus on quality improvement including the way we handle updates to content such as systematic reviews and ongoing clinical trials. We also started work on a major de-duplication effort and ending the quarter by refining our aspirations on quality.

Second quarter of 2023: This is where we really started to explore the potential of large language models (LLMs) such as ChatGPT. We also introduced scoring systems for guidelines and RCTs and a new guideline category for Europe. Finally, our de-duplication efforts saw 143,218 duplicate articles removed from Trip.

Third quarter of 2023: Our user survey results were published which is always interesting for us. We started exploring automated clinical Q&A using LLMs (see here and here). We also started our overhaul of the advanced search and started exploring some improvements to search relevancy.

Fourth quarter of 2023: We finished the year off strongly with the release on the new advanced search and the wonderful connected articles. We continued to push ahead with our LLM work with two main projects occupying our time (as well as a bunch of ideas bubbling away in the background). The two projects being a fully automated evidence review system and a clinical Q&A system. We also announced the creation of the AI Innovation Circle to help us to manage the AI potential/hype. Finally, we announced the main focus for the start of 2024: moving the Trip infrastructure to ‘the cloud’ and, unsurprisingly AI!

All this remarkable progress is supported by having a stable business model that has allowed us to remain independent and grow (as the site improves our income increases; a virtuous cycle)! While subscriptions to Trip Pro is the main source of income our bespoke rapid reviews have boomed in 2023. These are produced for a small group of selected organisations who appreciate our speed, evidence-based approach and cost-effectiveness. If you want to know more please email us.

Oh yes, don’t forget, we are still a ridiculously small organisation.

Another great boost for us is our users, they are wonderful and frequently crucial in helping us with surveys, feedback on new features etc. So, a big thank you from Trip to you.

2023 has proved immense, 2024 will hopefully be even bigger.

We import extra systematic reviews from a number of sources and we use special filters to identify them. It has come to our attention that one of the filters was too sensitive and therefore brought it a number of false positives ie tagged as systematic reviews and are clearly not!

So, we have started to clean these up and we’re currently removing 42,956 of them. Due to the method employed we’re also removing some true positives and therefore, once these are all removed, we will reimport these true positives! Convoluted, but the best way of getting this right.

I recently wrote about our plans for the next few months at Trip, this is already out of date! In the post I mentioned our work on automated reviews – this is still ongoing. However, we’ve also started an additional project – on Q&A.

Both projects are about generating full-automated POCs (proof of concepts). We’ve written about the automated reviews in the previous post. However, the Q&A work is a new one and will be based on retrieval augmented generation (RAG). One criticism of using LLMs (such as ChatGPT) is that they hallucinate. By using RAG you can, to a great extent, overcome these fears. RAG forces the LLM to obtain the answer from a predefined corpus of knowledge (as opposed to the knowledge within the LLM). In our case, for the POC, it will be using Trip’s clinical guidelines, systematic reviews and RCTs. If all goes to plan we should have a wonderful, evidence-based, Q&A system – one that can grade the answer based on the evidence used (taking into account risk of bias in RCTs, guideline scores etc).

Two final details:

• For both topics we are concentrating on the topic of migraine – a topic which isn’t too big or too small!
• We hope to be testing both of these in January

We’re quietly confident of both approaches but I suspect things will not be perfect!! As I said in my previous post on the topic – happy days!

Using the same mechanism as Connected Articles users can see related content for individual results on Trip.

Under each result on Trip you will see a related articles link:

If you hover over this link you’ll see the following:

If you click on this it’ll return the related content:

Nice and simple and it can be very useful 🙂

Now that we’ve got Connected Articles and Advanced search out and live we are working on two main areas of work:

Migrating to the cloud

Currently Trip sits on a server somewhere in the UK. However, this has some drawbacks and so we’re going to move our presence to the cloud. Given the complexity of Trip this is not easy but we can build it separately from the current Trip and then switchover when we’re happy it’s stable.

As part of this process we will be upgrading the underlying search system and rolling out ‘fixes’ to issues such as relevancy.

Artificial Intelligence

This is going to be interesting and there are two main avenues here:

• AI Innovation Circle: Even though we’ve hardly advertised this we’ve already had a great selection of volunteers. This group will help guide Trip in harnessing some of the amazing power that is now readily available.
• Automated reviews. Trip has a reasonably long tradition of exploring automated evidence reviews (e.g. Developing a fully automated evidence synthesis tool for identifying, assessing and collating the evidence) and more recently we’ve been experimenting with ChatGPT/LLMs (see More experiments with LLMs/ChatGPT). We’re about to take the experimentation into a proper working system. We’ll initially concentrate on a single condition and use that to learn before, hopefully, rolling out a something altogether more ambitious.

Happy Days 🙂

Advanced search is designed to give you much greater control of the search allowing more focussed results.

NOTE: Advanced search is a Pro only feature. If you’re fortunate to have access here follows a quick guide as to how to use it.

Above the search box on the homepage press the ‘Advanced’ tab and this will take you to a screen that looks like this:

There are 4 main elements:

• Search box – this is where you add you term or terms
• Where to search – this allows you to select where you want the search terms to occur. There are two options (1) Document text and (2) Title Only. The former is the default and if you want to search in the text of a document you can leave it alone. Also, note, that the document text refers to the text we have in our index, often this is full-text, but sometimes it might simply be the abstract
• Search type – this allows more Boolen functions and there are three options (1) All of these words – the default and effectively an ‘AND’ search (2) Any of these words and – an ‘OR’ search (3) The exact phrase – effectively places quotation marks around the added terms.
• Search – once you’ve created your search you press the search button.

See this example:

I have added the terms brexpiprazole Rexulti and selected ‘Any of these words’. I have left the ‘Where to search’ untouched. This effectively looks for any document in Trip that contains the word brexpiprazole OR Rexulti. After pressing ‘Search’ I see this:

As you will see it has created a line at the top of the search box, representing the first search. This is now #1. I will now do another search:

This time the search (#2) was a search for documents which have schizophrenia in the title. You can build up multiple lines of searching. The next aspect of the search is to combine the lines:

Here, I have selected the tick-box next to #1 and #2 and it has automatically populated the search box with the search. It defaults to an AND search but if you click on the OR or NOT buttons these are inserted e.g.

Reverting back to an AND search the results look like this:

The combination is now #3

A few things to note:

• You can manually add Boolean phrases, so you don’t need to use drop-downs or the ‘Boolean’ buttons. In other words you can simply type brexpiprazole OR Rexulti. Similar for phrase searching you can simply type “prostate cancer”.
• On the right-hand side of each search there is a bin symbol – this allows you to delete a single line of search. At the top there is a ‘Clear All’ link, this removes all the searches.
• To see the results for the most recent search simply scroll down. If you want to see a previous set of results, simply click on the hyperlinked number of results on the right hand side e.g. the 64, 22608 or 297.
• We default to showing the most recent 10-12 results. If you have done more and need to see them all, click on the ‘Show all’ link at the bottom of the list of searches.

Any questions or need for further clarification? Drop us a line: advanced.search@tripdatabase.com

We’re edging ever closer. As you can see from the latest screengrab we’re close:

The actual search works well just a few design issues we need to work out. But I’d say we’re 99% there 🙂

Trip is on the cutting edge, leveraging the transformative power of Artificial Intelligence, with a spotlight on sophisticated large language models (LLMs) like ChatGPT. You might have seen some of our recent forays: LLMs and Clickstream, LLMs again…. and More experiments with LLMs/ChatGPT.

As the AI landscape evolves at lightning speed, we recognise the significant potential of AI to help us deliver for our users. This is where the AI Innovation Circle comes into play – an informal group designed to strategically steer our AI journey to enhance user experience on Trip.

What will the AI Innovation Circle do?

• Harness AI’s vast capabilities to significantly elevate the Trip experience.
• Provide valuable insights to refine existing concepts, making them even more user-centric.
• Identify untapped opportunities where AI can drive innovation.
• Foster a collaborative learning environment to fuel collective expertise.
• Ensure Trip’s development are problem-led, not technology-led.

Engagement is not anticipated to be arduous and is likely to be managed via the convenience of email and the dynamic spark of occasional Zoom or Google Meets sessions.

Ready to be at the forefront of AI-powered transformation? Reach out to jon.brassey@tripdatabase.com to shape the future of Trip with us!

NOTE: The above was written by ChatGPT (including the group name) after I gave it a rough draft. Not 100% my sort of wording, but in the spirit of the technology I’m going to go with the flow!