Tuesday, December 16, 2014

Creating a Q&A environment in Trip

For those of you who've followed this blog for a while will see that I'm always revisiting the answer engine concept, most recently two months ago. A month before that I mentioned it in the context of a a Journal of Clinical Q&A

This all stems from my belief that Trip is a wonderful tool to answer clinical questions but a also belief that it could be even better!  After all, it was the reason I started it in the first place - to help me answer clinical questions via the ATTRACT Q&A serviceSurveys have shown that many clinicians agree, with over 70% of questions, supporting clinical care, are helped by using Trip.

Recapping briefly on the answer engine and the Journal of Clinical Q&A:

  • The answer engine will try to predict questions from the search terms and insert an answer above the search results.  Users will get an answer in one click.
  • Journal of Clinical Q&A is a journal idea - radically different from any other journal.  It will be a structured answer to a clinical question, posted on the site (and helping populate the answer engine) which will be peer-reviewed and given a citation.
So far, fairly radical and fairly good.

Now, another variable to consider - the PICO search system.  In the forthcoming upgrade we'll be enhancing this feature in the premium version.  It will be more guided than the existing version and it could work like this:
  1. Users types in their full-text question.
  2. Users then select the PICO elements from the question.
  3. Users view relevant results.
  4. Users are given the option to write up an answer. If they write up the answer we will show them the articles they've looked at and they can indicate which were useful (and thereby form the reference list).
  5. They can choose to keep it private or share it - feeding the answer engine.
Another powerful component for a Q&A environment, what could go wrong (I ask tentatively!)?

Friday, November 21, 2014

Professions in Trip Profile

When you register with Trip you are asked to select your profession, the current list is shown below:

The above 9 options are simply not good enough as around a quarter of users select the 'Other' option (and I can't imagine these users are made to feel particularly special!).  Also, as we want to offer increasingly personalised information, the more granular the detail we have on a person the better.  So, in our recent surveys I asked people to tell us their profession and from that I have come up with a more comprehensive list:

Academic researcher
Dental - other
Doctor/physician - other
Doctor/physician - primary care
Doctor/physician - secondary care
Librarian/Information specialist
Medical laboratory scientist
Nurse lecturer
Nurse practitioner
Nurse, clinical specialist
Public health professional
Physical therapist/physiotherapist
Physician assistant
Speech and language

Expect to see the changes in early 2015

Thursday, November 13, 2014

Communicating the evidence 'types'

Those who use Trip will possibly have noticed small thumbnails to the right of each search result (see image 1 below).  The idea is that they are a small screenshot of the actual page which people can rollover to see a preview of the actual result  They are problematic as it's currently broken so we only have screenshots for around half of them.  Also, they are moderately resource intensive.

So, we need to decide to fix them or remove them or replace them with something else - hence this post.

One idea I've got is to use the space to give additional information to users to help them understand the evidence they're looking at.  For instance, we could use it to give a clearer idea of the likely strength of evidence.  We currently do this via the use of colour flashes but unfortunately many people miss this.  The colour flashes link the individual article to the colours used in the filter section (so green indicates higher quality evidence etc.).  Below are some images that are an attempt to show what it might look like.  I'd appreciate you looking at them (click on the image to enlarge it) and then go to this survey to let us know what you think.  There are only 4 questions so it shouldn't take long.

Thank you in advance.

Friday, October 31, 2014

Strange results

Barb, one of our volunteers on our Twitter accounts, commented a while ago about seeing some strange results on Trip, so I asked her to send any news ones she found to help me understand what was going on.  She was looking for new articles in Trip that are returned for the search 'immunisations'.  Many were fine but a few weren't, for instance:

  • Multiple sclerosis: management of multiple sclerosis in primary and secondary care
  • Health visiting
  • Economic Evaluation of Complex Health System Interventions: A Discussion Paper
  • British Guideline on the management of asthma
  • Developing and Evaluating Methods for Record Linkage and Reducing Bias in Patient Registries

Now, these are not specifically about immunisations but they'll all make reference to it.  For instance the top result has the following:

1.4.2 Be aware that live vaccinations may be contraindicated in people with MS who are being treated with disease-modifying therapies.

We return all the results that match the search terms (and/or synonyms).  However, our algorithm is designed to emphasise those results which are more relevant.  So, ordinarily, if you do a search with lots of results the relatively irrelevant results don't appear (well they do, but not till way down the results).  However, if you look for things with few results (perhaps an unusual condition or you heavily restrict the results) you are more likely to see 'strange' results.

So, what can we do? I see three options:
  1. Leave it 'as is' and hope people don't get put off by the occasional result they find strange.
  2. We allow users to set a relevancy cut-off themselves.  Each search result gets a score from 0 to 1 (with 1 being very relevant) and every result that matches the search term gets at least 0.0001 and therefore can be shown in the results.  We could give users a 'slider' to allow them to chose what cut-off they want, So some might chose 0.1 while others might chose 0.3.
  3. We effectively borrow a concept from PubMed's Clinical Queries which has a narrow and broad search.  The narrow search returns fewer results, they're more relevant but you may miss a few (it's a specific search) while the broad search gets more results but more irrelevant results (it's a sensitive search).  So, in effect, Trip currently does a highly sensitive search.  You can see the effects in PubMed for a broad and narrow search for prostate cancer screening:
 My 'gut' instinct is the third option.  We, at Trip, experiment to try and arrive at a reasonable relevancy cut-off which is introduced by default on all searches. On the result's page we highlight that the search is narrow and to make it broad simply press a button.

Feedback please and thank you - again - Barb for the input :-)

Sunday, October 26, 2014

The Trip Answer Engine (again)

With the move to the next upgrade - and a freemium Trip the notion of the Answer Engine appears again. I've talked about the Answer Engine for at least 4 years but previously I've never had the conviction that it'll work.  The idea is great: infer a question from the search terms and show 'the' answer.

It's because I like the idea so much I keep coming back to it. I've done a mock-up of how it might look.

I'm waiting to hear from one publisher about using their content. If they agree that I can re-use their content it'll be thousands of Q&As ready to go and I'll be ready to commit to getting it off the ground.

I'm also talking to other publishers about their willingness to participate. We get Q&As and they get their content in a prime position on Trip, a win:win in my book! Other than that I'll be undertaking another user survey and will ask then if people want to volunteer to add a few Q&As. If everything falls into place we'll have a reasonable chance of making it work!

Friday, October 17, 2014

Survey 2014 - initial results

Trip users are amazing - in less than 48 hours of releasing the survey we had 1,0001 responses, at which stage SurveyMonkey closed the survey saying we'd reached the limit!  Apologies if you feel your voice hasn't been heard, if that is the case email me (jon.brassey@tripdatabase.com), I'd love to hear from you. Given your generosity of time I thought I'd share the initial results highlights...

The top 5 professions represented in the survey
  • Doctor - secondary or tertiary care
  • Doctor - primary care
  • Librarian
  • Other
  • Researcher/scientist
75% of respondents have been using Trip for more than a year with 35% using it for longer than 3 years.

I asked about the most important features relating to our content and these are the top 6 responses (those that were highlighted by more than 30% of the respondents):
  • Largest single searchable collection of ‘evidence-based’ content
  • Largest global collection of clinical guidelines
  • Many more systematic reviews than Cochrane
  • Content is from around the globe, for example USA, UK, Canada, Australia, New Zealand, France, Germany, Japan, Singapore, South Africa
  • Selected collection of PubMed’s leading clinical journals
  • Database of over 500,000 clinical trials
I also asked if there were many surprises - and there were lots of responses.  The main one being the lack of awareness of our image and video collections.  We clearly need to work hard on getting that message out.

I asked about the most important key features of Trip, the following are all those that polled over 30%:

  • Easy filtering of results to restrict to evidence types e.g. systematic reviews, guidelines
  • Monthly alert of new evidence linked to your interests
  • PICO search interface
  • Order the results by quality, relevance or date
  • Easy/Friendly interface with no steep learning curve
  • Advanced search interface
  • Colour coding scheme to make it easier to highlight high quality evidence
Our users seem keen to be alerted to clinical trials, jobs, conferences and books (most polled over 50% approval).

We asked about a Trip Evidence Service and most thought it was a good idea.  However, only 11% thought they would be able to find the money within their organisation.  But I'm encouraged as 11% is still high, given our large user base.

Most people appeared to be broadly supportive/understanding of our need to move to a freemium business model.

I listed a number of possible new premium features and those that polled greater than 20% (only the top 3 were higher than 30%):
  • Add in additional full-text articles
  • Creation of an 'Answer Engine' giving you instant answers to your clinical questions
  • PICO+. Based on the popular PICO search make it more user friendly and powerful
  • A 'Help' feature so if you can't find what you need you can ask the wider Trip community
  • Providing education points based on your time using Trip
  • Improved emails highlighting evidence that is more likely to be useful to you
  • Introduce a 'People who looked at this article, also looked at these articles' features to highlight related articles
  • Improved export of records
Due to us using colour we asked about colourblindness and 3.2% said they were colourblind.  I've no idea how that compares to the wider population.  nearly 30% of the users reported "I am not colour blind and I was not aware that you used colour to help highlight the quality of the results".  So, another communication challenge for us.

Finally, in looking through the 'Any other comments' section I was completely overwhelmed by the messages of love and support.  Knowing that makes my work so much easier.

Tuesday, October 14, 2014

Survey time

We are planning to make significant changes to Trip in early 2015.

An important aspect of this is better understanding our users; how they use Trip and what features they value.  In addition we're keen to explore attitudes to various proposed changes.

This is a really important survey so can you please take 5-10 minutes to go through the 14 questions.

Click here to take part in the survey

Thank you.