While much of our attention is focused on AskTrip we are still working on other aspects of Trip and one thread is introducing a score for RCTs. We already have a risk of bias score but it has been increasingly difficult keeping this up to date. So, time to do our own in-house one. It will look like this:

It looks like our existing scores for guidelines and systematic reviews but will be powered by LLMs. And, as with the systematic review score, it will be based on the trial abstract.

Take this trial: Occupational therapy improves social participation of complex patients discharged from hospital: results of a powered randomized controlled trial. Using our prompts we were able to generate an overall score of ‘good’ (this is just our in-house terminology) and we generated these strengths and weaknesses:

Strengths

  • Clearly identified as a randomized controlled trial (RCT).
  • The aim follows the PICO format: Population (complex patients post-hospital discharge), Intervention (experimental occupational therapy plus standard rehab), Comparator (standard rehab alone), Outcome (social participation).
  • Participants are described with inclusion criteria and demographics.
  • Randomization is mentioned, indicating attempts to reduce selection bias.
  • The intervention and comparator are clearly described, specifying the type and duration.
  • Outcomes are reported with specific measures (Reintegration to Normal Living Index, COPM) and include statistical results (mean changes, confidence intervals, p-value).
  • Study is registered on ClinicalTrials.gov, suggesting transparency and planning.

Limitations

  • No mention of blinding, which could introduce performance or detection bias.
  • Sample size is relatively small (92 participants), which may limit generalizability.
  • No explicit mention of drop-out or loss-to-follow-up rates, which is important for assessing attrition bias.
  • Harms or side effects are not mentioned, which is critical for evaluating the safety of the intervention.
  • There is no information on funding or potential conflicts of interest, which is important for identifying bias.

When we roll this out, the above text will be available when a user clicks the question mark (far right on the image).

As per previous scores, the use of scores is not without criticism – but clearly we feel it’s worthwhile – here’s an old discuss on the topic.

This will be a gradual rollout starting with the most recent trials first.

At the top of this post I mentioned that this wasn’t directly linked to AskTrip, the reality is that it can be. In the medium term, we can factor in the quality of guidelines, systematic reviews and RCTs when answering a question. That would be rather special!