Trip Database Blog

In February I posted an article discussing ‘the near instantaneous meta-analysis‘. In a nutshell – is there a way to very rapidly combine the results of multiple-trials?

Since then we have been quite busy working on this project, helped by some external expertise and a recent research grant.  Trip funded phase one, a proof of concept phase that allowed me to appreciate the challenges, limitations and opportunities that our approach presented.  The results were great and since then we have been awarded a grant to move forward to phase two.

Phase two will create a working model for people to use. This will be quite a simple solution and will only be aimed at synthesising placebo-controlled trials (more complex, comparator trials will form phase three).  The working model will work as follows:

1. User will use a modified search box telling us the condition and the intervention (e.g. acne and antibiotics).
2. Our system will search just the controlled trials portion of Trip to identify suitable trials.
3. We will then analyse these and present a score (more below on the scoring system).
4. We will then have an area that explains the results, how we arrived at them and the ability for the user to alter certain aspects.  This last bit is important as we’re relying on machines to ‘read’ the documents and extract pertinent information.  This is unlikely to be foolproof and while the system ‘learns’ it’ll need some feedback from users.  But, if the user does make alterations we will then re-analyse based on the updated information.

Quite simple really and steps 1-3 will take less than a second (we hope).

As for the score, that’s an interesting area and we’re sure it’ll change over time.  But the thinking at the moment is – what is most clinically useful?  After all, our audience for this will be practicing clinicians, not academics.  As such we’re thinking that an effect size is not particularly great/intuitive.  I really like the Clinical Evidence system for rating interventions e.g. ‘Likely to be beneficial’, ‘Unknown effectiveness’.

However, I’m also struck by the systems used by Amazon and TripAdvisor to rate articles.  A product/holiday is given an overall score but you can easily see how the score is arrived at.  When I use these I always look at the reasons people have given for 1 or 2 stars (ie people who have rated the item poor).  Whichever system we use we’ll make it very easy for users to differentiate good and bad aspects of an intervention.

Hopefully, phase two will be released in 6-8 weeks, probably not on broad release to start. This will be a gentle release, to a few people to start with.  This will allow us to alter the algorithms, allow for further machine learning etc.

I see this whole ‘instant review’ system taking a minimum of four phases.  Hopefully, if phase two works as well as we think it will, funding will follow to allow us to move to phase three.

In a meeting yesterday we discussed the concept of institutional ‘accounts’ with regard to our soon to be released full-text link-outs. In this scenario an institution (hospital, university etc) would give us some information to allow their users to link directly from Trip to the full-text.  For this to work the user needs to tell us they work for the particular institution.

But, once we have the information, can we be more useful?

Having all the evidence in Trip is great, but often local policies, guidelines etc are really important to consider. So, why not allow the user to search all the evidence in Trip but also be able to see if their institution has something to say on the issue.  It makes perfect sense to me.  But this is non-trivial and requires a number of issues to be dealt with:

• The institution needs to agree to this.
• How do we get local content into Trip.
• What format, web-based in easy but Word documents creates an extra level of complexity.
• Once we have the content in Trip, how do we maintain it being up-to-date?

The latter point is probably the biggest issue.  It’s fine to have initial enthusiasm to add content, but to go back every now and then and maintain up-to-dateness is a harder proposition. What happens if the enthusiast leaves?

That worry aside, doing a check every 3-6 months isn’t a big deal and we could arguably build some alert system saying ‘You have not checked your content for 3 months, please go and check now’.  I’m guessing we could even automate something that checks all the links every now and then.

One to ponder and to ask the advisory board about, when we get the latest upgrade out of the way.

UPDATE: One thought has struck me, why not create a spider to go and grab all the content on the site (the way Google does).  This would be no work for the institution but requires all documents to be on a public-facing website (so no intranet content).  Also, it grabs everything (e.g. see this MIT example) which might introduce some noise!

I’ve been writing on this blogs for years – since 2006. Since 2008 we’ve been tracking the page views (how often people read an article) and that it over 111,000 times.

Most articles get around 200-500 page views, some fewer and a handful many more.  Our recent critique of Cochrane has been viewed (at the time of writing) 3,068 times, it’s our second most viewed article (Using TRIP to help identify content suitable for resource poor settings has been viewed 5,483 times).

These figures seem high, but are they?  The technique I’ve come up with, to answer the question, is to compare the figures to individual articles in the BMJ.  Each BMJ article has a handy article metrics tab making this easy.  So, comparing to the Cochrane critique I found 5 articles published on either the 3rd or 4th April (the Cochrane critique was published on the 7th).

• Effect of lower sodium intake on health: systematic review and meta-analyses – viewed 5854 times (excluding PDFs)
• Effect of increased potassium intake on cardiovascular risk factors and disease: systematic review and meta-analyses – viewed 9036 times
• When to remeasure cardiovascular risk in untreated people at low and intermediate risk: observational study – 2445 times
• Effect of longer term modest salt reduction on blood pressure: Cochrane systematic review and meta-analysis of randomised trials – 8352 times
• Predictive value of S-100β protein for prognosis in patients with moderate and severe traumatic brain injury: systematic review and meta-analysis – 2017 times

On average they are viewed 5,540 times – about 80% higher than our Cochrane article.  But, the Cochrane article had higher figures than 2 of the 5 BMJ articles, which impresses me.

In conclusion, I think our readership figures can be pretty good, even when compared to one of the world’s top medical journals. 

For many secondary research articles (e.g. guidelines and systematic review) we already link to the full text.  However, for primary research we typically link out to the abstract on PubMed.  In our users surveys, linking to full-text has consistently been the  top of the list of  ‘wants’ for new features.

Well, after a significant amount of effort we have solved it and, as of mid-May, we will significantly boost our links to full-text articles.  This will take happen in two ways:

• We have started to cross-reference our PubMed articles to see if they already appear in PubMed Central (the full-text equivalent of PubMed).
• Institutional holdings.  If you work for an institution that purchases full-text journals, we can now (in most cases) link directly from Trip to your institutions full text.  

In the case of the institutional holding we require contact with someone who knows about link resolvers and this will typically be someone from the library.  So, if you want better access to full-text I recommend popping into the library. It’s probably best if you ask them to email me directly: jon.brassey@tripdatabase.com

This feature will be rolled out, with a host of others, in mid-May. 

In 2009 we started an advisory board, mentioned in this blog.  The wording is as relevant now as it was then:

TRIP has grown and matured as a site considerably over the years and this change has been particularly rapid in the last 12 months. The next upgrade will mark a significant improvement and it’s a momentum I wish to maintain. To help TRIP in this process I’ve decided to set-up an advisory board.

The TRIP advisory board will be an informal network of clinicians, information specialists and techies and I would expect it to serve the following functions:

• Respond and advise on ideas generated within TRIP
• To suggest ideas for new features/improvements on TRIP
• To generally be an extended pair of ears to highlight new technologies, opportunities for TRIP.

We set up the board using a website called Ning and it worked pretty well, but then they started charging and I let things drift.

Basically, I want to set it up again as there are some big decisions coming up and I would love to feel I can ask really dedicated and enthusiastic Trip user’s opinions.  Equally importantly is to create a ‘space’ where board members can feed back honest opinions and suggestions.

So, volunteers (unfortunately, it’s unpaid) would be nice.

Also, suggesting for the best mechanism to communicate would be good.  Ning seemed pretty good as I could post out to everyone and people responded, allowing everyone to see the responses.  Alternatively, I could simply email questions out and link to survey monkey etc.Perhaps that can be the first question for volunteers.  So, if you’d like to be part of the board, let me know via jon.brassey@tripdatabase.com

In an increasingly inter-connected world it is often useful to share content.  The easier it is to share content the more likely a user is to to do it.  At Trip we’ve got a really easy system to share our great results via email, Twitter and Facebook.

Simply click on the ‘Share this’ button (top image) and then select which method you want to use (email, Twitter or Facebook). 

After the last post (New: Controlled Trials in Trip) we got the following comment:

Will you make the information about the PubMed filters for your controlled trials available so we can get an idea how comprehensive your database is. Will you also compare your results with those listed in the central database of controlled trials in the cochrane library? 

This seems entirely reasonable, so the first part of the comment, the filters:

Julie Glanville suggested 4 different filters, all with different sensitivity and specificity):

1. (randomized controlled trial[Publication Type]) OR ((randomized[Title/Abstract] OR randomised[Title/Abstract] OR placebo*[ti]) and (controlled[Title/Abstract] OR trial[Title/Abstract]))
2. (randomized controlled trial[Publication Type]) OR ((randomized[Title/Abstract] OR randomised[Title/Abstract] OR placebo*[tiab]) and (controlled[Title/Abstract] OR trial[Title/Abstract]))
3. (randomized controlled trial[Publication Type]) OR ((randomized[TI] OR randomised[TI] OR placebo*[ti]) OR (controlled[TI] OR trial[Ti]))
4. (randomized controlled trial[Publication Type]) OR ((randomized[Title/Abstract] OR randomised[Title/Abstract] OR placebo*[tiab]) OR (controlled[Title/Abstract] OR trial[Title/Abstract]))

I tried these all out in PubMed and got the following numbers of identified trials for each filter:

1. 419575
2. 434984
3. 438900
4. 921118

The 4th, being so different from the first three seemed easy to ignore while the other three, all being within 10% was reassuring.  So, I decided to go for number 3.  Testing revealed some false positives but nothing too scary!

With regard the second part of the comment, comparing the results with CENTRAL.  I’d be delighted for someone else to, but we don’t have the resource or the knowledge to do so!

Today we released a new refine option in Trip, one for Controlled Trials (mainly RCTs).

After help with filters from Julie Glanville we have grabbed trials from PubMed and Mendeley and this has resulted in approximately 500,000 trials being added to Trip (too see the filter used, click here).  Give the nature of filters used to highlight controlled trials there is a compromise between sensitivity and specificity. Over the next few months we’ll work to improve the quality and also the quantity of trials.

In testing, I’ve used the feature extensively and it’s worked really well.  It really is a powerful addition to Trip.  To use it yourself, simply go to Trip and search as you would normally and simply press the ‘Controlled Trials’ link/button in the refine area on the right hand side of the search results.

It’s been nearly a month since my last post, which reflects how busy we are at the moment. The main effort is actually around reviews and combining articles to help answer questions.  This is taking two separate routes, but the potential overlap is clear.

The first route is a review wizard. This would be a step-by-step way of searching Trip followed by a way of capturing all the articles that are of interest and allowing the user to collate these in a ‘beautiful’ format.  People use Trip to review topics all the time.  So, if we can help that process it’s got to be a good thing.

The second route is altogether more ambitious, the near instantaneous meta-analysis. I’m working with a few people to explore a technique I’ve discovered that will allow for near systematic review quality results within ten minutes.  Sounds ambitious?  This has the potential to be massive, turning the productive of high-quality evidence on it’s head.  Currently, it take 1,000 hours, two years and between £20-100,000 to do a systematic review.  Surely, taking ten minutes and little cost and you’ve got something close to a systematic review would be a wonderful breakthrough?  So, I’m aiming high with this one.  It may well come to nothing, but if you don’t try you’ve got no chance.  Also, if I fail I’ll post my failing(s) on the blog and elsewhere and hopefully people can learn from my mistakes and push it through.  I shouldn’t be negative as I’m really optimistic on this one