Trip Database Blog

We launched our guideline scoring system two months ago; as a reminder this is what it looks like:

Since then we have been very busy added scores for another 50 guideline publishers, including:

• Ministry of Health, Malaysia
• Scandinavian Society of Anaesthesiology and Intensive Care
• World Society of Emergency Surgery
• British Society for Sexual Medicine
• European Association for the Study of the Liver
• European Psychiatric Association

We now have scores for almost all the guideline publishers we cover, nearly 300 in total!

A user alerted us to a set of poor results on Trip, in this case relating to a search of breast cancer. It took me a few seconds to realise what was wrong:

For a hot topic such as breast cancer you’d expect almost exclusively green results (signifying higher quality). And, one result I was looking for – a recent Cochrane systematic review – was at result #28. So, what was going on?

We figured out it was a synonyms issue. We had mammary as a synonym of breast, so our system saw effectively three terms in the title (breast, cancer and mammary) and felt these were really, really relevant.

We have now edited the synonyms and the results now look as we’d expect:

All green and there are two Cochrane reviews in the top 5.

As it happens this issue only affected a small number of results (it also affected colorectal cancer but to a lesser extent) but for a popular search, such as breast cancer, the impact would have been widespread.

We’re committed to making Trip better and improvements like this are significant steps forward.

User feedback is a cornerstone of Trip’s development and, over the years, it has proved invaluable.

To help us gather user feedback we have devised a short questionnaire. Please help us to make Trip better and take the survey.

CLICK HERE to participate.

At the end of last year I posted Connected/related articles to highlight some thinking about combining different connections between documents to help ensure users can quickly and find articles related to the documents they’ve already clicked on. In other words, as users click on documents of interest, we start to collate connected articles to present to the user to help them ensure they’ve not missed important documents.

I have had the pleasure of testing out our text version and it’s really really good. Currently its not had the design treatment but you can start to see the power. I did this search on Trip bisphosphonates prostate cancer and clicked on these results:

• Contemporary Population-Based Analysis of Bone Mineral Density Testing in Men Initiating Androgen Deprivation Therapy for Prostate Cancer
• Hypocalcaemia in patients with prostate cancer treated with a bisphosphonate or denosumab: prevention supports treatment completion
• The role of bisphosphonates or denosumab in light of the availability of new therapies for prostate cancer
• Use of bisphosphonates and other bone supportive agents in the management of prostate cancer-A UK perspective
• Bone Health in Patients with Prostate Cancer

The top four being from PubMed and the bottom one a Canadian guideline. Our connected articles outputted the following:

So a lot’s going on in the above screenshot, so some explanations around the score. Note, this is still in the testing phase so these weights/scores are liable to change. The following factors are shown:

1. N – number of results to display.
2. Clicks – this is based on our co-click data. If any of the 5 documents I clicked on have been co-clicked in a previous search session this is noted and added to the list of connected articles. We have currently weighted this by a factor of 3, as we feel this is a really important factor.
3. Rel – stands for related articles. This is only available, at present, for PubMed articles and we extract the related articles from the 5 documents clicked. These related articles are added to the list of connected articles.
4. Ref – references. We extract the references used from the 5 documents clicked. These referenced articles are added to the list of connected articles.
5. Cites – this looks to see if any of those 5 clicked documents been cited by other documents. These cited articles are added to the list of connected articles.
6. Incl – shows if the document is already in the Trip index.
7. Words – this explores if the connected articles contain the initial search terms in the document title. The more words the documents find in the title the closer we judge it to be to the initial search and therefore the user’s intentions.

So, from the above, points 2-5 are about identifying connected articles while the others are additional factors we’re using.

Here are the top 5 articles that our system generated with hyperlinks:

• Metastatic Prostate Cancer and the Bone: Significance and Therapeutic Options
• Harnessing the potential of therapeutic agents to safeguard bone health in prostate cancer
• Bisphosphonates for advanced prostate cancer
• The role of bisphosphonates or denosumab in light of the availability of new therapies for prostate cancer
• Hypocalcaemia in patients with prostate cancer treated with a bisphosphonate or denosumab: prevention supports treatment completion

Some observations:

• They are all highly relevant documents (although I note that one returned article was one we originally clicked on – this will be fixed before we released this).
• The connected articles tend to be older. This makes sense as the Trip algorithm favours newer articles so these tend to be shown first. So, connected helps unearth older, important, papers that a user may have missed.
• The 3rd article in a Cochrane systematic review. Interestingly not the most up-to-date version but it still unearths it and a user can quickly navigate to the latest version.
• One of the top 5 articles was not in the main Trip index (and in the larger list of 11 documents over half were not in Trip). That was the Cochrane review, we have the more up-to-date version. But it’s nice that the system can highlight possibly important papers outside of Trip.

I’m possibly biased but I’m really excited by the possibilities of this system to help users find the best articles they need. And remember, this is a test system and we have some fixes/improvements to roll out. Once we’ve done that we need to incorporate this into Trip and that highlights the next challenge – the user interface/design. We need to balance making it obvious to users yet not too intrusive. Given how good the system is I consider this a nice problem to have!

One of the developments strands of Trip is improving the quality of existing content or functionality. De-duplication was mentioned in a recent post on quality and we’re pleased to announce significant progress.

Given the complex nature of Trip and the variety of sources of content, we have generated a number of duplicate records – two (or more) examples of the same article. Often identical but sometimes a link to the abstract and another to the full-text. Having two copies of the same article is good for no-one and just adds ‘noise’ to the search results. To identify and remove these has proved to be a challenging piece of work but we’ve finished the work and identified a total of 143,218 duplicates and these are currently being removed from the index.

Are we now duplicate free? Invariably not, but we’ve probably got the vast majority. But, if you do spot one please let us know.

Up Next

As the de-duplication finishes our next quality issue is to remove articles, from PubMed, that contain no abstract. We never used to include them but with the new system it was overlooked so they’ve crept back in. PubMed articles with no abstract contain no/little actionable information so it adds ‘noise’ to the results and very little ‘signal’.

Hot on the heels of us releasing our guideline score we’re releasing our RCT score. We’ve been working with the wonderful RobotReviewer team for years now and one of their products is a Risk of Bias (RoB) score for RCTs. We introduced it in 2016 where we classified all the trials into categories of ‘low risk of bias’ and ‘high/unknown risk of bias’. When we recently re-wrote the site we did not immediately include the RoB score. In part this reflects that, since 2016, the thinking and technology has developed considerably. So, we’re very pleased to reintroduce it to the site.

The new score does not categorise the RoB into ‘low’ or ‘high/unknown’, it gives a score based on the likely RoB on a linear scale. We take that score and transform that into a graphic that is similar to that seen on the guideline score:

RCTs are important in the world of EBM and, as with guidelines, they are not all equally good! This score reflects the likelihood of bias and should help our users better make sense of the evidence base.

One relatively minor addition to our recent guideline enhancements has been the introduction of a new geographic guideline filter – ‘Europe’

Over the last few years we’ve been diligently identifying and adding guidelines from Europe so it made sense to add a new filter.

One final tweak was to make the ability to filter by geographic area a ‘Pro’ only feature.

The production of guidelines is a complex task and there are a multitude of methods, some more rigorous than others. While Trip places guidelines at the top of the evidence pyramid we need to recognise this is an over simplification . Our guideline score is designed to help our users understand how robust a guideline might be.

The guideline score has been a concept we’ve explored for a number of years (eg Quality and guidelines from 2019 and Grading guidelines from 2020) and involves us scoring each publisher (not individual guideline – see limitations below) based on 5 criteria:

• Do they publish their methodology? No = 0, Yes = 1, Yes and mention AGREE (or similar) = 2
• Do they use any evidence grading e.g. GRADE? No = 0, Yes = 2
• Do they undertake a systematic evidence search? Unsure/No = 0, Yes = 2
• Are they clear about funding? No = 0, Yes = 1
• Do they mention how they handle conflict of interest? No = 0, Yes = 1

The highest score being 8. Our work has shown that the above results give a very good approximations to the more formal methods, hence we’re using this simpler approach. And this is what it looks like:

Limitations

This approach has a number of issues, for instance:

• It is carried out at the publisher level and was done at a certain date. So, if we scored things in 2021 the scoring covers guidelines produced by the publisher in 2014 (say) and 2023. The methodology might well have changed between those dates. This is not reflected in our scoring.
• Linked with the above point, it assumes the guideline publisher uses the same methodology for all guidelines.
• Many of the lowest scoring producers do so due to the lack of publication of their methodologies making it impossible to properly score them, so our approach may underestimate the rigour of the methodology. If our approach encourages publishers to be more transparent then it’ll be a great result in itself!
• The scoring system uses 5 elements, it might benefit from more but we have to pragmatically balance rigour and resource.

Fascinating technology and we’re continuing to explore its use. One focus is on auto-summarising documents and so far it’s pretty good/interesting (with the large caveat that it can be problematic and hallucinate)! And there are two examples (below), generated from https://chat.openai.com/:

• Extract the main recommendations from a SIGN guideline
• Summarise the methods section of a Cochrane review

The first response seems reasonably good. However, the second one was problematic. I tried it a few times – more around the wording – to get it to ‘correctly’ summarise. A few times it suggested the document had nothing to do with cranberry juice and UTIs! Also, on one occasion it stuck to summarising the methods section whereas, in the example below, it went further!

Our first use case is to use it to summarise long documents into abstracts. Initially this would just be used by our search index and it’d allow us to modify our search to default to ‘title and abstract’. Currently our default is ‘title and body of the document’ which, for long documents, is particularly sensitive returning low relevancy documents. For instance, a document on measles, which is 100,000 words long mentions prostate cancer one time. This means it is returned for a search on prostate cancer! This is normally fine as it’d be low down in the results. However, if you filter the documents to a single area (eg UK guidelines) where there are few documents then suddenly the low relevancy results can be seen on the first page of results. A summary of the document would not include prostate cancer and would therefore not be returned in the results!

The second use case would be to allow users to see the summary, helping them decide if it was useful or not. There are issues around copyright to consider….

We’re having some fun with this technology and we’re happy to hear from you if you have any suggestions as to how Trip could use it to good effect!

Extract the main recommendations from a SIGN guideline

Prompt: Can you extract the background and main recommendations from this document https://www.sign.ac.uk/media/1987/sign-164-eating-disorders-v2.pdf

Response: Yes, I can extract the background and main recommendations from the document you provided.

Background:
The document is a guideline developed by the Scottish Intercollegiate Guidelines Network (SIGN) on the assessment and management of eating disorders. It provides recommendations based on the best available evidence for healthcare professionals involved in the care of individuals with eating disorders.

Main recommendations:
The document provides recommendations for the assessment, treatment, and management of eating disorders, including anorexia nervosa, bulimia nervosa, binge eating disorder, and other specified feeding or eating disorders. Here are some of the main recommendations:

Assessment: A thorough assessment of the individual’s physical, psychological, and social needs should be conducted as soon as possible. This includes obtaining a detailed history of the eating disorder, physical examination, and laboratory investigations.

Treatment: A multidisciplinary team approach is recommended for the treatment of eating disorders. The team should include a range of healthcare professionals, such as psychiatrists, dietitians, psychologists, and physicians.

Psychological interventions: Evidence-based psychological interventions, such as cognitive-behavioral therapy, interpersonal therapy, and family therapy, should be offered to individuals with eating disorders.

Nutritional management: Nutritional management should be tailored to the individual’s needs and preferences. A dietitian should be involved in the development and implementation of the nutritional plan.

Pharmacological interventions: Medication should only be used in conjunction with psychological and nutritional interventions. The use of medication should be carefully monitored, and the benefits and risks should be weighed.

Monitoring and follow-up: Individuals with eating disorders should receive ongoing monitoring and follow-up care. This includes regular physical and mental health assessments, as well as ongoing support from the multidisciplinary team.

Overall, the document emphasizes the importance of a collaborative, evidence-based approach to the assessment and management of eating disorders.

Summarise the methods section of a Cochrane review

Prompt: Can you summarise the methods section, as bullet points, from this document https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD001321.pub6/full

Response:

• This Cochrane systematic review aimed to evaluate the effectiveness of cranberry products in preventing urinary tract infections (UTIs) in people of all ages.
• The review included 28 randomized controlled trials (RCTs) involving a total of 4,473 participants, who were mostly women.
• The studies compared cranberry products (juice, capsules, tablets, or other formulations) to placebo, no treatment, or other interventions for preventing UTIs.
• The primary outcome was the incidence of symptomatic UTIs, defined as the presence of at least one urinary tract symptom (such as pain, burning, or frequency) and the presence of at least 10^5 colony-forming units (CFU) of bacteria per milliliter of urine.
• The secondary outcomes included the incidence of asymptomatic bacteriuria (the presence of bacteria in the urine without symptoms), the incidence of UTIs caused by specific bacteria (such as E. coli), adverse events, and compliance with the interventions.
• The review used standard Cochrane methods for searching, selecting, and assessing the quality of the studies, and for synthesizing the data using meta-analysis.
• The quality of the evidence was assessed using the GRADE approach, which takes into account factors such as study design, risk of bias, inconsistency, imprecision, and publication bias.
• The review found that cranberry products may slightly reduce the incidence of symptomatic UTIs compared to placebo or no treatment, but the effect was small and of uncertain clinical significance.
• The absolute reduction in UTIs ranged from 0% to 13%, depending on the type and dose of cranberry product, the population studied, and other factors.
• The review did not find evidence of a difference in the incidence of asymptomatic bacteriuria or adverse events between cranberry and control groups.
• The review concluded that cranberry products cannot be recommended as a routine prophylaxis for UTIs, but may be considered as a non-pharmacological option for some women who are motivated to try them and understand the limitations of the evidence.