Trip Database Blog

After the last post (New: Controlled Trials in Trip) we got the following comment:

Will you make the information about the PubMed filters for your controlled trials available so we can get an idea how comprehensive your database is. Will you also compare your results with those listed in the central database of controlled trials in the cochrane library? 

This seems entirely reasonable, so the first part of the comment, the filters:

Julie Glanville suggested 4 different filters, all with different sensitivity and specificity):

1. (randomized controlled trial[Publication Type]) OR ((randomized[Title/Abstract] OR randomised[Title/Abstract] OR placebo*[ti]) and (controlled[Title/Abstract] OR trial[Title/Abstract]))
2. (randomized controlled trial[Publication Type]) OR ((randomized[Title/Abstract] OR randomised[Title/Abstract] OR placebo*[tiab]) and (controlled[Title/Abstract] OR trial[Title/Abstract]))
3. (randomized controlled trial[Publication Type]) OR ((randomized[TI] OR randomised[TI] OR placebo*[ti]) OR (controlled[TI] OR trial[Ti]))
4. (randomized controlled trial[Publication Type]) OR ((randomized[Title/Abstract] OR randomised[Title/Abstract] OR placebo*[tiab]) OR (controlled[Title/Abstract] OR trial[Title/Abstract]))

I tried these all out in PubMed and got the following numbers of identified trials for each filter:

1. 419575
2. 434984
3. 438900
4. 921118

The 4th, being so different from the first three seemed easy to ignore while the other three, all being within 10% was reassuring.  So, I decided to go for number 3.  Testing revealed some false positives but nothing too scary!

With regard the second part of the comment, comparing the results with CENTRAL.  I’d be delighted for someone else to, but we don’t have the resource or the knowledge to do so!

Today we released a new refine option in Trip, one for Controlled Trials (mainly RCTs).

After help with filters from Julie Glanville we have grabbed trials from PubMed and Mendeley and this has resulted in approximately 500,000 trials being added to Trip (too see the filter used, click here).  Give the nature of filters used to highlight controlled trials there is a compromise between sensitivity and specificity. Over the next few months we’ll work to improve the quality and also the quantity of trials.

In testing, I’ve used the feature extensively and it’s worked really well.  It really is a powerful addition to Trip.  To use it yourself, simply go to Trip and search as you would normally and simply press the ‘Controlled Trials’ link/button in the refine area on the right hand side of the search results.

It’s been nearly a month since my last post, which reflects how busy we are at the moment. The main effort is actually around reviews and combining articles to help answer questions.  This is taking two separate routes, but the potential overlap is clear.

The first route is a review wizard. This would be a step-by-step way of searching Trip followed by a way of capturing all the articles that are of interest and allowing the user to collate these in a ‘beautiful’ format.  People use Trip to review topics all the time.  So, if we can help that process it’s got to be a good thing.

The second route is altogether more ambitious, the near instantaneous meta-analysis. I’m working with a few people to explore a technique I’ve discovered that will allow for near systematic review quality results within ten minutes.  Sounds ambitious?  This has the potential to be massive, turning the productive of high-quality evidence on it’s head.  Currently, it take 1,000 hours, two years and between £20-100,000 to do a systematic review.  Surely, taking ten minutes and little cost and you’ve got something close to a systematic review would be a wonderful breakthrough?  So, I’m aiming high with this one.  It may well come to nothing, but if you don’t try you’ve got no chance.  Also, if I fail I’ll post my failing(s) on the blog and elsewhere and hopefully people can learn from my mistakes and push it through.  I shouldn’t be negative as I’m really optimistic on this one

The timeline on Trip captures all your activity on the site, recording your search terms and articles viewed.  An extension of this is the ‘star’ feature.  This allows you to highlight articles that you think are particularly ‘notable’.  To ‘star’ an article you simply press the star to the left of a particular result (remember you should be logged in). In the image below (click to enlarge) you can see the stars higlighted next to each article.

At any stage you can look back at your starred articles via a link at the top of the page called ‘Starred items’ (also highlighted).

You can also restrict any search you carry out to only show items you’ve starred.  You do this via the ‘Further refinements’ section on the right-hand side of the results page (for interest, there is also the ability to restrict search results to those you’ve previously looked at).

I’ve also created a screencast for further information – click here to view.

NOTE: This is a slight expansion of an earlier post (from 2012) but it’s an important feature we want to help users understand.

What started out as a minor upgrade has turned into something altogether more substantial. I posted much of the detail a few weeks ago, but as we start work things develop.  We’re still hoping to get the upgrade out by the end of February (depending on testing) and the main new features will be:

• RCT filter.  It struck me, given their prominence in the ‘evidence based’ world, as strange that we didn’t have an RCT filter.  So, why not have one and why not make it wonderful.  So, we’re going to grab RCTs from multiple sources and hopefully launch with at least 500,000 RCTs making it one of the biggest RCT databases. Invariably the largest FREE RCT database.
• Full-text.  The ability to better link to full-text has been a major request from clinician users of Trip.  So, we’re going to make it much easier for users to navigate from the primary research articles to full-text (we currently just point to abstracts).  We plan to do this in two ways:
1. Better integration with PubMed Central, the full-text sibling of PubMed.
2. Working with organisations to allow users to link to their institutions full-text collections.
• LMIC (Low and Middle Income Countries) filter.  We’ve worked on this idea in the past but this takes a new approach.  We’ll be using the LMIC filter highlighted by the Norwegian Satellite of the Cochrane Effective Practice and Organisation of Care Group (see here). It’s not validated but it needs to be put out there, tested and then – hopefully – improved upon.  It should make the identification of evidence for LMICs much easier.
• DynaMed. This is not certain, but we’re hopeful, that users of DynaMed will be able to search Trip and see DynaMed content in their search results.
• Case Reports. Perhaps at the lower end of the evidence spectrum, we’ll be introducing case reports from the really interesting Cases Database.
• Low relevancy cut-off. A search in Trip returns ALL results that match a search query – even if the search term is only mentioned once in a ten thousand word document. I would consider that document as having low relevancy to the search.  So, we’re going to remove all articles with a low relevancy score.  Users, if they want can reintroduce them, can do so with minimal effort!

Fingers crossed that testing goes well.

Related to that is a brief survey we’re doing mainly around how to position the full-text offering.  Six questions, five minutes. Please do it here.

Today I instructed Phil to start work on the latest upgrade to Trip.  This will see the following new features:

• RCT filter.  On Trip we have a systematic review filter but it’s a bit odd, given the prominence in the EBM world, that we have no RCT filter. However, in order to do this we have to build a decent collection of RCTs which we’re also having to do.  We’ll mostly grab content from PubMed but we’re also exploring other options/databases.  We’re hoping to have at least 500,000 RCTs by the time we launch.
• Full-text link outs.  In our user surveys the desire to link to full-text has been a consistent request and, at last, we are able to act on this.  We are using two main techniques:
• PubMed Central (PMC) is a repository of full-text, freely available articles organised by the National Library of Medicine (who also supply PubMed).  We currently link to PubMed for all our abstracts to primary research articles.  So, we’re going to cross-reference our PubMed holdings with PMC and link to the full text on PMC (if it exists).
• Institutional holdings.  Many users work in institutions that have paid for full-text.  So, we will offer the ability for institutions to work with us to allow their users to seamlessly link with full-text articles the institution has purchased.
• LMIC. This stands for Low and Middle Income Countries and we want to add a specific filter to identify evidence suitable for such areas.  
• Relevancy filter.  A search in Trip returns ALL results that match a search query – even if the search term is only mentioned once in a ten thousand word document. We would consider that document as having low relevancy to the search. We will remove these low relevancy results. If there are lots of results (say over 100) we remove all results with very low relevancy. If we get over 250 results we remove all results with low relevancy. All this would be undoable (ie show all results) at the push of a button.

Hopefully these will be released by the end of February.

I have just finished Ben’s latest book – Bad Pharma.  I cannot recommend it highly enough, it is a brilliant book.  It is written in 6 chapters (and an afterword) :

• Missing Data
• Where Do New Drugs Come From
• Bad Regulators
• Bad Trials
• Bigger, Simpler Trials
• Marketing
• Better Data (afterword)

I found the ‘Missing Data’ and ‘Bad Regulators’ the most chilling. 

If you’re reading this blog you’re probably interested in robust evidence in which case you really should read Ben’s book.If you’re still not convinced, have a look at Ben’s talk at TEDMED 2012.

This is one of the most important challenges facing Trip, one which I hope I can rely on your help.

How to help people use Trip to capture and publish evidence collections!?  Collections of evidence already exist and are typically time consuming.  Four examples:

• ATTRACT – this is part of my NHS work.  Our team receives questions and finds appropriate evidence with which to answer it.  Relatively unstructured.
• ATTRACT CME – a good example might be this review of obesity.  In this example it’s a mixed collection of the latest evidence and background information.
• BestBETs – these are reviews, based on questions arriving in emergency medicine, that tackle a single question.  In many ways these are similar to ATTRACT but are more structured.
• Cochrane systematic reviews – these are highly structured collections of clinical trails. 

Non-health related collections are important and available, a few examples:

• Pinterest – see this example of Liverpool Football Club.
• Amazon’s wish list – users can create a collections of items they’d like to buy

Summary: collections are everywhere and are clearly useful.

I see two issues in relation to Trip:

1. Would Trip users like to make collections?
2. If they do, what might it look like?

I think the answer to 1. is ‘yes, assuming you can make it a rewarding and easy exercise‘. 

But 2. is really problematic; how to create a product that looks great, is easy to use and facilitates the production of robust and useful reviews?  We can do a few clever things such as making it easy to group articles together, auto-reference and even suggest related articles.  But you’re still left with the core problem – the middle of the collection – the actual content (sandwiched between title and references)?

I like the visual impact of something like pinterest (see another example from Doctors Without Borders).  Highly visual, so engaging.  The downside being there’s not much space for text.   But again, I could see us allowing a user to pull in their documents of interest, annotating each article with the key point and then pulling it together with a summary and/or clinical bottom line.

At the top of the post I said this was the most important challenges to Trip, I believe it and I also believe if we get it right we will have created something hugely useful. 

So, if you read this and have any suggestions, no matter how silly/random you may feel they are, please let me know (via comment below or emailing me – jon.brassey@tripdatabase.com).  Often it just takes a few novel thoughts to unblock the creative process.  This perspective is exemplified by a comment I received at a Trip training session where a user said they would love to be able to ‘tag’ an article (or articles) saying these helped her answer a particular question.  In other words, she wanted to group articles together around answering a clinical question.  That simple request started all this thinking…!

In Trip our search algorithm (the magic that decides which order articles appear on the results page) is made up of three main components:

• Publication score – the higher quality the publication (think Cochrane, NICE, AHRQ) the higher the score.
• Year score – a document from 2012 scores more highly than a document from 2011.
• Text score – this analyses documents and assigns a score based on location of matches (e.g. if the search term appears in the title it scores more highly than if it only appears in the body of the text).

These separate scores are combined and the article with the highest score appears at the top and the rest of the results appear in descending score order. This typically works very well but there can be problems.  If a document scores lowly on one component and high on two others it can appear quite highly in the results.  This is typically not a problem expect, I think, in the case of text relevancy.

When someone does a search on Trip we retrieve every document that mentions the search term(s) and each of these documents are given a text score.  If we have a big document that mentions the search term once it will still be found and still get a score, even though it is obvious that the document isn’t really about the subject.

So, what I’m thinking of doing is introducing a relevancy cut-off. If someone searches on Trip and the search generates a large number of results (say over 100) we introduce a text score cut-off.  This text relevancy score would still be quite low but enough to remove the really irrelevant results.  For example the text relevancy score ranges from 1 to 0.  In my mind the cut-off might be at around 0.1. 

Now, the issue with this is that the results are now being restricted, which I know makes many uncomfortable.  I think this depends on reason for searching Trip.  If you’re a busy clinician wanting to just get really quick results it’d be no big deal.  However, if you’re an information specialist wanting to ensure you’ve checked everything – it’d be seen less favourably.

Therefore, the compromise might be some sort of button/warning that says something like ‘We have removed all articles Trip considers of low relevance to the search, click here to show all results’.  I’d like to think that’s the best of both worlds.