Trip Database Blog

I’m currently preparing a talk on standards in clinical question answering services. Even though I’ve been involved in Q&A for ten years it’s proving really hard. I’ve been scribbling notes down, trying to decide what is important. Some major issues:

• Transparency, you’re not doing a systematic review – so make that clear. I think the onus is on the Q&A service to ensure users are acutely aware of potential drawbacks in the service.
• Feedback, be it from the person who asked the question or from other readers of the answers (assuming things are web-based). We receive lots of feedback, but we could make it easier – perhaps something as straightforward as a digg style thumbs up or thumbs down.
• Process-standards. These are things such as length of answer, speed of answer, referencing of material etc. Not the most interesting!
• People skills. It’s fine to say that the person asking the question is competent (or even excellent) in searching Medline, but that certainly doesn’t make them good at answering questions! Understanding of the question is important before you consider a search. How can you make a standard around that?
• Quality control. Is there a robust system in place?

The above is not a complete list but some of the more memorable ones.

The biggest drawback I found is that there is very little research in this area on which to base standards! In the area of systematic reviews there are vast amounts of research on the actual process of conducting systematic reviews. In Q&A there is virtually nothing.

For me, and this notion hasn’t changed in ten years, is that we’re not trying to do a systematic review, we’re just trying to improve on what a clinician would do. I remember 8-9 years ago receiving some criticism from a civil servant suggesting what I did was negligent. I suggested to her that if our service is negligent then surely she was in providing Medline to clinicians who are poorly trained to search on them.

Although hard work it has proved very useful preparing for this talk, it’s an area I’ve not spent a great deal of thought on in the past. And there’s still over a week to go, so plenty of time for reflection.

Clinical Knowledge Summaries (CKS) was previously known as PRODIGY and produces some great guidelines. They have just released a redesigned website – it’ll take some getting used to!

I find the browsing of topic confusing but the search is good (powered by Google!). But my biggest concern is – once you find the correct guideline – how long it takes to find the appropriate section. They have split the screen into 3 sections, with the actual content squeezed into 1 section; before the content filled the screen. This effectively means each guideline is now around 3 times longer.

I was involved in the beta-testing and have been waiting for the release of the updated site. Perhaps the problem is that I’m an atypical user of the site, perhaps I’m just like most other people and find change unsettling. In 3 months time I may love it!

I’ve just answered the 5th question of the day and decided to check if we’ve had any feedback for the answers I posted earlier in the day. We received the following:

“I stumbled on to the service after searching the haematuria topic via google. The first time I tried you were not taking further questions so tried again and got a very helpful response. I think it is a BRILLIANT service and really useful for somebody like me (a GP) who would like to be evidence based in approaching clinical problems but often th eresaearch is just too much on top of the clinical work.

Serves a really imporatnt need. This is as important as anything else going on in the NHS now.”

I’ve been answering clinical questions for around ten years now and this sort of feedback only helps highlight the importance of clinical Q&A services. The Q&A services I run are answering hundreds of questions per month and are able to offer significant support to clinicians in accessing the evidence base. The services tend to be run with little money or other support. What does it take to have them taken seriously? I can only think that getting papers written in journals will help, but other than that I have few ideas. I still want to create a journal of clinical Q&A and may well get that off the ground after June. I see that having the following features:

• A formally written up Q&A. I’m not sure what this might include but the following seem sensible: the actual question, search methodology, articles found, narrative description of evidence and possibly a clinical bottom line.
• A resource review. This would highlight resources useful in answering clinical questions.
• Theory. There are many aspects of Q&A that lack a robust theoretical underpinning so papers exploring this would be helpful.

My slight concern is that the majority of the papers would be from me or close colleagues who work on the various Q&A services.

Due to the large volume of traffic we have temporarily removed the counts for each category on the results page. The counts should be re-instated by the end of the month using a new system to prevent the burden on the servers.

Since we removed the counts the speed of query-response has improved significantly.

Last month I posted a video of TRIP on YouTube. The quality was poor!

Well, a month later and we’ve posted it on our own server (click here to view). The quality is much enhanced and the text readable – hurrah!

I feel using short videos is a great way of showing people how the site works and is invariably more useful to a busy clinician than 2-3 sides of paper in a leaflet.

We’ll adopt this ‘technology’ in the future. For instance, the TRIPanswers site will be fairly groundbreaking (in the clinical world) so we’ll need to explain the broad principles to users – video seems ideal.

We receive lots of questions and we tend to be confident that we know what the user has requested. However, occasionally we receive questions which are less clear. In these situations we get back to the clinician and try and figure out why they’re asking the Q. We received this one:

How long should you stop a PPI for before doing helicobacter serology?

It seems a reasonable question, but it’s ambiguous. For instance is this pre- or post-eradication?

It transpires that it was post-eradication. Over the course of the exchange I highlighted that serology is not recommend post-eradication and that a breath test should be used. Therefore, we end up with the ‘proper’ question:

How long should a patient avoid PPIs prior to a H.pylori, post-eradication, breath test?

It hasn’t changed a great deal but the differences are crucial!

This week’s BMJ carried a news article on the most popular Cochrane systematic reviews (click here, although you need a subscription!). Aside, from suggesting it was a slow week for news, they highlighted geographic variations in popular reviews.

I responded to the article highlighting (click here) that the most popular Cochrane reviews – as viewed by TRIP users – was radically different.

I feel the main reason is that if a user goes to search Cochrane, they’ll get a restricted cohort of evidence to look through. In other words users might be more likely to view an article that is ‘almost’ what they were after.

TRIP, which contains a much greater range of secondary reviews and other evidence, means that users can be more discerning in their selection of articles. Further evidence of this might be that TRIP users only venture to Cochrane 2.6% of the time. So 97.4% of the time they feel their evidence/information needs are met elsewhere.

One of the most frequent comments we get back from the users of the NLH Q&A Service is the speed of our responses. With significant changes likely to happen in the next 3-6 months I thought it was important to record a few process outcomes. This will provide a worthwhile comparison once the changes have bedded in.

Feedback

In our feedback forms (sent with each answer) we ask a number of questions, the two are most important relate to how they rate the service and whether they would use it again. Below are the two questions we ask with the results underneath.

1) Are you likely to use NLH query answering service again in the future?

• Definitely Not – 0%
• Unlikely – 0%
• Maybe – 4%
• Likely – 8%
• Very likely – 88%

2) Overall, how would you rate the NLH Q&A Service:

• Poor – 0%
• Below Average – 0%
• Average – 4%
• Good – 13%
• Excellent – 83%

It’s worth pointing out that the vast majority of the relatively poor responses coincide with when there is no real evidence. I’m assuming there is a feeling that we haven’t answered the question.

Timeliness

We pride ourselves in the high-level of service we provide. As highlighted above we receive very positive feedback from the users of the service. An important aspect of the Q&A service is getting the answer to the clinician in a timely manner. Using the same feedback as above 100% of users report getting their answer back quickly enough. But we’ve analysed the actual turnaround times for the last 6 months:

• less than 6 hours – 20.2%
• 6-12 hours – 43.4%
• 12-24 hours – 16.4%
• 1-2 days – 15.8%
• 2-5 days – 3.6%
• 5+ days – 0.6%

The above figures are based on time we receive the question to time of posting the answer.

So what we’ve created, and I hope continues, is a very quick, very well received Q&A service.

It’s been a busy day. We’re going to be ‘staff light’ next week so I’ve put in a fair few hours today. Of the 7 questions attempted few could be answered with robust evidence; a few examples:

What is the difference between soluble and insoluble fibre? How does the soluble form reduce cholesterol.
The first part was fairly straightforward, while the second part was conjecture that doesn’t appear to have moved on in a number of years. I suppose that it doesn’t matter greatly as long as there is an effect, but even so…

What current evidence for best practice is there in the conservative podiatric management of paediatric pes cavus?
Cochrane reviewed this topic in 2007 (not restricted to children) and found no real evidence, apart from one area. We did find a 2008 retrospective review of cases in children, but that’s hardly the most robust evidence!

In minor surgery following abscess incision and drainage, is packing necessary and if so how long and how frequently should the abscess be packed?
This was a good one! Why? Because the published evidence appeared to clash with our medical directors experience!!! His view was that the research was of low quality. But then we get down to the ‘nitty gritty’ of evidence-based practice – what is the role of experience?

In the absence of robust evidence, when there is only poor-quality research, where does a doctor’s (or nurse’s) experience fit into things? I often feel that ‘evidence-based’ practice judges experience too harshly. Perhaps not judges it; sneers might be a better word.

I still feel that DUETs has some promise in highlighting gaps in the research evidence. DUETs is attempting to record gaps in the evidence (relating to therapeutics) highlighted from research recommendations and clinical Q&As. The idea being to inform/enhance the research procurement process. In other words, if you know that lots of people are interested in a particular clinical question and the evidence-base is poor – procure some research!

As part of our TRIPanswers (to be launched July/August 2008) we’re taking a different angle at highlighting the research gaps – with our ‘tag cloud of clinical uncertainty‘. Ours is a more pragmatic approach. Perhaps the biggest differences will be what constitutes uncertainty (we’ll take a less academic approach) and what can be covered (we’ll cover everything, not just therapeutics). Our pragmatic approach will allow us to ‘cover’ more ground and we should launch with between 500-1,500 uncertainties.

If either (or both) approaches makes an impact it’ll have been worthwhile.