Trip Database Blog

This is one of the most important challenges facing Trip, one which I hope I can rely on your help.

How to help people use Trip to capture and publish evidence collections!?  Collections of evidence already exist and are typically time consuming.  Four examples:

• ATTRACT – this is part of my NHS work.  Our team receives questions and finds appropriate evidence with which to answer it.  Relatively unstructured.
• ATTRACT CME – a good example might be this review of obesity.  In this example it’s a mixed collection of the latest evidence and background information.
• BestBETs – these are reviews, based on questions arriving in emergency medicine, that tackle a single question.  In many ways these are similar to ATTRACT but are more structured.
• Cochrane systematic reviews – these are highly structured collections of clinical trails. 

Non-health related collections are important and available, a few examples:

• Pinterest – see this example of Liverpool Football Club.
• Amazon’s wish list – users can create a collections of items they’d like to buy

Summary: collections are everywhere and are clearly useful.

I see two issues in relation to Trip:

1. Would Trip users like to make collections?
2. If they do, what might it look like?

I think the answer to 1. is ‘yes, assuming you can make it a rewarding and easy exercise‘. 

But 2. is really problematic; how to create a product that looks great, is easy to use and facilitates the production of robust and useful reviews?  We can do a few clever things such as making it easy to group articles together, auto-reference and even suggest related articles.  But you’re still left with the core problem – the middle of the collection – the actual content (sandwiched between title and references)?

I like the visual impact of something like pinterest (see another example from Doctors Without Borders).  Highly visual, so engaging.  The downside being there’s not much space for text.   But again, I could see us allowing a user to pull in their documents of interest, annotating each article with the key point and then pulling it together with a summary and/or clinical bottom line.

At the top of the post I said this was the most important challenges to Trip, I believe it and I also believe if we get it right we will have created something hugely useful. 

So, if you read this and have any suggestions, no matter how silly/random you may feel they are, please let me know (via comment below or emailing me – jon.brassey@tripdatabase.com).  Often it just takes a few novel thoughts to unblock the creative process.  This perspective is exemplified by a comment I received at a Trip training session where a user said they would love to be able to ‘tag’ an article (or articles) saying these helped her answer a particular question.  In other words, she wanted to group articles together around answering a clinical question.  That simple request started all this thinking…!

In Trip our search algorithm (the magic that decides which order articles appear on the results page) is made up of three main components:

• Publication score – the higher quality the publication (think Cochrane, NICE, AHRQ) the higher the score.
• Year score – a document from 2012 scores more highly than a document from 2011.
• Text score – this analyses documents and assigns a score based on location of matches (e.g. if the search term appears in the title it scores more highly than if it only appears in the body of the text).

These separate scores are combined and the article with the highest score appears at the top and the rest of the results appear in descending score order. This typically works very well but there can be problems.  If a document scores lowly on one component and high on two others it can appear quite highly in the results.  This is typically not a problem expect, I think, in the case of text relevancy.

When someone does a search on Trip we retrieve every document that mentions the search term(s) and each of these documents are given a text score.  If we have a big document that mentions the search term once it will still be found and still get a score, even though it is obvious that the document isn’t really about the subject.

So, what I’m thinking of doing is introducing a relevancy cut-off. If someone searches on Trip and the search generates a large number of results (say over 100) we introduce a text score cut-off.  This text relevancy score would still be quite low but enough to remove the really irrelevant results.  For example the text relevancy score ranges from 1 to 0.  In my mind the cut-off might be at around 0.1. 

Now, the issue with this is that the results are now being restricted, which I know makes many uncomfortable.  I think this depends on reason for searching Trip.  If you’re a busy clinician wanting to just get really quick results it’d be no big deal.  However, if you’re an information specialist wanting to ensure you’ve checked everything – it’d be seen less favourably.

Therefore, the compromise might be some sort of button/warning that says something like ‘We have removed all articles Trip considers of low relevance to the search, click here to show all results’.  I’d like to think that’s the best of both worlds.

One thing that struck me recently was that we don’t have a filter for RCTs in Trip.  Given the importance of these it seems remiss of us.  So, why not create one?  Well, I now plan to in early 2013.  We can build this by scraping content from PubMed using a suitable RCT filter (the current count is 438,900 trials) and I hope to work with Mendeley to highlight even more.  So, we should have a RCT collection of over 500,000 trials – which is hugely impressive.

While I was working through this idea I saw an email about the lack of systematic reviews suitable for the developing world (low and middle income countries, or LMIC).  As you may know we have done some work in this area (see our crowdsourcing initiative) but that’s never really taken off. 

So, a thought occurred, why not use a filter for LMIC content?  A filter is a series of terms designed to highlight focussed content.  We’ll use one for identifying RCTs (see here for further details) and it looks for terms/phrases such as randomised in the title.  There are a number of validated filters for RCTs, which is great.  A link from the Norwegian Satellite of the Cochrane Effective Practice and Organisation of Care Group highlights a LMIC filter.  It’s unvalidated but looks like a great start.

So, pulling these two tales together:

• We create a wonderful RCT database
• We can tag RCTs if they’re suitable for LMIC
• We can tag the rest of the Trip content as being suitable for LMIC

This is massive!

In March I addressed this issue (see here) but I still keep coming back to the notion of having a freemium version of Trip.  As Wikipedia defines it:

“Freemium is a business model by which a product or service (typically a digital offering such as software, media, games or web services) is provided free of charge, but a premium is charged for advanced features, functionality, or virtual goods.”

Freemium has many advantages, namely security (if people sign up) and of keeping the site – mainly – free. But what would it look like?

I’m currently working on a few benefits to people signing-up:

• Discounts of various ‘evidence-based’ products (events, courses etc).
• I’m talking with a book publisher about getting pretty reasonable discounts on their books.
• Removal of adverts, so a subscriber would not have to see them.

While these are pretty decent, I’m not sure we’ve reached the stage where membership might be compelling!  Anyone got any other ideas?

Also, pricing would need considering.  I was thinking of £2 per month, which is relatively low (in richer parts of the globe). 

The fact that I’ve revisited the topic after 6 months shows two things:

1. It’s still a model I’m interested in.
2. It’s not clear-cut, otherwise I would have done it by now!

This week I did a couple of training sessions in London on using Trip.  It’s great doing these as you really connect with the user, they highlight problems that need fixing and also throw in some ideas.  One person mentioned something that immediately struck me as being useful and fitted with a broad, but vague, theme I’ve been thinking about for years. One of the librarians asked if she could indicate, in Trip, if a document helped answer a question.

In other words, she went to Trip with a Q, for instance, “Is vitamin D2 better than vitamin D3 in vitamin D deficiency?”. She might search and find that a document answers the Q. She would then indicate to Trip that that particular document answered the Q she had.

It’s a simple concept but operationalising it is more complex (more below) but also – fundamentally – would it be widely used?  Clearly, if we create an engaging, easy to use system, it’s more likely to be used..!

Below are some thoughts on the topic, which I hope will resonate with people.  The process might look something like this:

• A user comes to Trip and they use the site as normal (but logged in).
• At some stage we ‘highlight’ the tagging feature.  Something like ‘if Trip has helped you answer a Q, let us know and share’.
• If they activate this we show them their session history (? use the timeline) and they indicate (via tickboxes) which articles they used (as often a Q will require multiple articles to answer it) and then tag all these with ‘These answered my question on X’.
• We could probably allow users to write a bit of text, to pull it together, give a bottom line etc.  This might well mean we need to allow comments, so people can respond.

The next issue is what to do with this?  A further few thoughts:

• Activity is recorded in their timeline.  So, a user does a Q&A this is highlighted in their timeline.
• This collection has it’s own separate page.  So, a user can point to each eg tripdatabase.com/qa/vitd2ord3
• Each user has their own Q&A page, which lists all the answers they’ve done.
• How do we index/use this?  Do we add it to Trip answers and this is then searchable? This seems reasonable – but a slight worry about potential bias but that can be mitigated by warning text.
• On each individual result (for an article in Trip, not a Q&A) we indicate, somehow, that it has been used to answer a question.  In other words a user searches Trip normally and in the results it’s highlighted if a user has ‘tagged’ the article to say it has been useful.

Right, lots of thoughts above – highlighting the issues I’m wrestling with.  It also gives a glimpse of how I work.  This is an early stage idea which needs conversations between me and users (and our techie and designer). 

After this stage – and with your help – I’ll try and get our designer to mock this up so we can better explain the concept and make it more tangible.  How long that takes is another issue!

The timeline on Trip captures all your activity on the site, recording your search terms and articles viewed.  An extension of this is the ‘star’ feature.  This allows you to highlight articles that you think are particularly ‘notable’.  To ‘star’ an article you simply press the star to the left of a particular result (remember you should be logged in).

At any stage you can look back at your starred articles via a link at the top of the page called ‘Starred items’. 

You can also restrict any search you carry out to only show items you’ve starred.  You do this via the ‘Further refinements’ section on the right-hand side of the results page (for interest, there is also the ability to restrict search results to those you’ve previously looked at).

I’ve also created a screencast for further information – click here to view.

Using Trip can be highly educational – searching, reading articles, reflecting etc.

Many professionals (doctors, nurses etc.) can be required to keep a record of their educational activity throughout the year, to demonstrate that they are keeping ‘up to date’ with the latest research.

Trip has two main ways of supporting this:

• The timeline – this is a record of activity on Trip (search terms, articles viewed), this can easily be exported as a PDF for inclusion in any portfolio of learning.
• Reflective learning – more in-depth than the timeline the reflective learning tool allows clinicians to easily record any reflections they have while reading an article.  This is accessed via the CPD/CME button under each result.  Again, all CPD/CME activity is easily exported as a PDF file.

To help understand how to use Trip and the different types of support we offer we’ve produced a brief screencast, which can be viewed here.

So far the PICO search has been one of the most heavily praised features on the new Trip.  But, we received the following comment:

I noticed my search was translated as follows: 8 results for “(title:ischaemic stroke)(title:CT perfusion scan)(non CT perfusion)”, by relevance

If so, I would not feel confident that I had not missed out on other pertinent papers….

This is a really important point, how does PICO search work?

At the heart of the PICO search is something called contingency searching.  With the normal Trip search you get all the results that match your search terms but with the PICO search we aim to just show a limited number of highly focused results.  To achieve this our first search is for all the PICO elements as title only searches.  If there are too few results we then make the final search term a ‘title and text’ search and repeat the search and if that too has too few results we make the penultimate term a ‘title and text’ and we repeat that until we get a manageable number of results.  All these repeated searches are done in the background; from a user’s perspective it’s a single search. 

So, in response to the last point raised by the user, it’s not an exhaustive search and should not really be used for a timely ‘gather all’ search. It’s designed to help users, who are in a rush, get a really manageable set of results to help answer their clinical query.  It does that rather well.

A ‘screengrab’ showing the PICO search is below (click to make larger). We’ve also made a screencast to demonstrate PICO in action – click here to view that.

At the end of the summer 2011 we asked users about how they used Trip, what they liked, what they didn’t like and how they would like Trip to develop.  The main set of results can be seen here.  These results, combined with my own views, independent feedback from users and the contents of the wonderful book Search User Interfaces spurred Trip on with the latest redesign.

Add in the following elements:

• The ever wonderful Phil, our main developer. Superlatives fail to describe his wonderful work on the site (Click here to see his LinkedIn profile)
• Reuben, (introduced to us by Phil), his work has been so exciting and it’s been great having a fresh pair of eyes on the site/problems we face.
• An fair amount of investment, both financial and time from the Trip team (myself and Chris)
• Those that donated to Trip earlier on this yeat
• The beta-testers – thank you for your work.

I’ve described the main updates in this blog post but the only real way to appreciate the site and the breadth of changes is to go and use it – go now!

Another way is to watch this brief screencast I’ve produced (which can be viewed in a larger format here).

http://www.screenr.com/embed/mCj8