This latest feature will be released soon. For a given article premium users will be able to see related articles (based on clickstream data) as well as information on total views, views by country and views by profession…
Long-term readers of this blog will know I struggle with many aspects of the systematic review process. At the time of writing, my ‘A critique of the Cochrane Collaboration‘ has been viewed over 18,300 times and ‘Ultra-rapid reviews, first test results‘ nearly 10,000 times.
I believe the main justification given for conducting systematic reviews is to obtain a really accurate assessment of the effectiveness (or ‘worth’) of an intervention. So, the thinking goes that spending 12-24 months is worth the cost (financial, opportunity, etc) due to the accuracy of the prediction it then gives.
My immediate response is that is demonstrably false. In my article ‘Some additional thoughts on systematic reviews‘ (just under 5,000 views) the evidence is clear that if you rely on published journal articles to ‘inform’ your systematic reviews (which is the case in the vast majority of systematic reviews) there is approximately a 50% chance that the effect size is likely to be out by over 10%.
But, even if we suspend being evidence-based and believe that systematic reviews can be relied upon to give us an accurate estimate of an effect size, is everything fine? I don’t think so and the image below illustrates my thinking.
It’s an hourglass! At the top are all the unsynthesised trials, all floating around and the uncertainty is moderate. Someone then spends 12-24 months pulling these together in a systematic review (likely of published trials and therefore ‘a bit dodgy’) and the certainty is reduced at the aperture of the hourglass. But then, when you apply it to the real world of patient care, the uncertainty flares out again. In the above example the intervention has a NNT of 6, so the intervention needs to be given to 6 people to obtain the desired outcome in 1 person. Which is the 1 person? Where’s the certainty?
If we were to spend significantly less time doing a review it might indicate a wider hourglass aperture (perhaps suggesting an NNT of 5-7). In what situations does that matter? I don’t think we’ve even started to explore these issues. In other words, when is it appropriate to spend 12-24 months on an systematic review and when is a significantly less resource intensive approach ‘ok’?
Is it irony that the reality is the type of review (systematic versus ‘rapid’) doesn’t alter the effectiveness of an intervention? After all the compound remains the same, untroubled by the efforts of trialists. Sorry, getting sociological there – must be time to sign off for now.
At the start of the year I posted Ok, I admit it, I’m stuck, which was a cry for help from the Trip community to help me make sense of all our lovely clickstream data. We had a few responses and one was from an Australian research and management consultancy QSPectral, a company specialising in providing strategic insights and predictions through advanced data science and analytics. They have been working with us to help us make sense of our clickstream data.
Article Association
QSPectral used their data science expertise to investigate the connections between the articles based on the user access data contained within the Trip Database.
Figure 1 Snapshot of articles accessed across a session. The colours represent user professions (doctor, nurse, etc.)
In the above image the Y-axis represents individual search sessions and the X-axis is the documentID (each article in Trip has a unique document ID). So, we can see what professions are looking at which articles. We can actually see what articles individuals are looking at, but the above image shows it on a profession basis.
Figure 2 A more focused snapshot of the previous image
As a user do you want to see what other articles are similar to the one you are reading?
Do you want to know what others like you thought were similar?
To provide answers to these questions, QSPectral developed an algorithm based on association rules to explore the relationships between articles on a per session basis. We intended to identify links between articles based on different criteria of interest.
The strength of the links was measured by statistical measures such as confidence and support factors. These led to association rules, which were of the form if {article x is accessed then articles y and z} were also accessed were further enhanced by including additional user characteristics – information such as the profession (nurse, doctor..) as well as country of origin were used to moderate the previously established article relationships.
Figure 3 Snapshot of related article numbers – if the articles on the y axis are accessed it implies those on the x axis would also be of interest.
The data can be further augmented by adding clickstream data that includes the area of speciality (such as cardiology) for a user, where the for example, if you are a doctor from Spain only relationships between articles that doctors from Spain accessed could be isolated and uncovered. It was also possible to group the related articles in clusters based on this multi-dimensional relationship – defined by colour in the figure.
Figure 4 clusters of articles based on relationships
The purpose of this initial investigation was to set the stage for providing users with recommendations based on their initial article of interest and their particular user characteristic. A slightly different approach to PubMed’s ‘related articles’ feature.
As well as finding closely related articles QSPectral have helped us explore recommendations of new articles. So, if we know a user’s activity on Trip we can start to understand them and then – with QSPectral’s help – recommend new articles that should be of interest.
Article Recommendation
How will TRIP recommend articles for you?
Machine learning methods based on clustering and classification are being investigated for providing reliable recommendations.
We believe that initial article clusters should be identified using an algorithm known as k-means clustering. Each user will then be classified as being interested in articles within a cluster based on attributes such as their first choice of article and user attributes (profession, country etc.) using a method where a tree-like graph or model of decisions and their possible consequences, including chance event outcomes, resource costs, and utility is created.
Figure 5 Example of a Decision Tree where the top node could represents you and the other nodes represent related articles based on branch criteria.
QSPectral determined that decision trees are the most appropriate concept for meeting the requirements. Decision tree methods can accommodate more data inputs over time. Various other transformations of inputs are possible and are robust to inclusion of irrelevant fields in the data, and produces transparent models for on-going analysis.
Further, we will use other methods that take a number of simple decision trees and combine them in some way to yield a final overall picture. We propose techniques for iteratively averaging multiple deep decision trees, trained on different parts of the collected data, with the goal of reducing the variance. Each iteration creates a simple decision tree on randomly selected subsets of input variables and input data. The final result where recommendations are provided will be formed through classifying a user through the aggregation of all such trees.
One change we introduced recently is the increased user ‘pressure’ to log in. A few people have contacted me to raise this as an issue and it made me realise we’ve added a barrier to use of Trip but we’ve not communicated why. So, here goes…
Ultimately it’s part of a longer-term strategy to improve Trip and this requires us to better understand our users (which requires the user to be logged in).
Some background; my partners Dad was an eminent Professor of Anaesthetics (now retired) and I showed him Trip, and he said he’d use it for a bit. He came back unimpressed! His interest was in awareness, and a search for awareness on Trip (click here) returns no articles on awareness under anaesthesia, which was his interest/intention (see for yourself).
While this is an extreme example it does highlight that, without knowing the user, how can we optimise the search results? Our system should have realised that the user was an anaesthetist and adjusted the results accordingly. We’re doing lots of work on this area and are making real strides. I blogged about in March with the article The important breakthrough which contained the following image:
As you can see from the results (in this experimental test system) we have detected the example user as a dentist and adjusted the results accordingly. For an information retrieval ‘nerd’ (like myself) this is amazing. I can think of no other innovation Trip has introduced that will come close to improving the search results as this.
And there are loads more things we can do if we know the user. For instance improved email alerts – better linking users with evidence that is likely to be interesting and useful, as opposed to our current crude efforts!
But for it to work we need to know the user, which requires logging in.
We are in the process of switching to a new email system and this is causing problems! If you’re requesting things such as password renewal your email is in a queue of 28,776 and the rate of sending is 1,006 per hour! So, another 24+ hours till we’ve got through those.
Prior to the new system we used an in-house email tool we built from scratch. It worked really well for 7+ years but has recently started to creak at the seams. So, we’ve upgraded to a paid system called Mandrill.
The problem is that when you’re new it doesn’t allow rapid sending of emails as it ‘senses’ your reputation. It looks at things such as number of rejected emails (for instance). The one thing we have are a load of dormant accounts and currently we’ve got a bounce rate of 20% – so 20% of emails are bouncing back as being undelivered. This doesn’t help our reputation, which is ‘poor’ – hence being restricted to 1,006 per hour.
The good news is – and Mandrill is great for this – is that it allows us to easily auto-delete these dormant account so next time our reputation will be much higher and therefore we should have a much higher send rate.
That aside Mandrill does all sorts of things which should allow us to create a much better email experience and also it gives us analytics showing how many emails were opened, how many links were clicked etc. Fascinating reading.
So, apologies if you’re caught in the email queue!
One of the many powerful features of Trip is the ability to refine your results based on the type of evidence you’re looking for. It’s really simple to use and below are some screen shots to walk you through the process.
If you have any questions just ask: jon.brassey@tripdatabase.com.
If you’re interested in upgrading, see the main differences in this infographic and upgrade here.
As we move forward after the introduction of our Premium product we can start to plan future developments and one is a search safety net!
Using our click stream data we can see what articles you’re looking at suggest other documents that you should consider. Take this network map (click to enlarge):
This is based on searches on Trip for urinary tract infections. Each blue square (node) represents an article and the lines linking them are created when a user clicks on two (or more) articles in the same search session.
If we have this information we can build a really useful system. A user comes and does a search of Trip for UTI and finds, from articles in the bottom right of the above image, a number of articles (marked in red in the image below):
It is clear that they may have overlooked four articles (marked as blue nodes) so we alert the user. It gives them a chance to double check the results. They may have deliberately excluded them or they may have simply made a mistake. If it’s the latter then the system will have served its function as a safety net.
I’ve started using the phrase ‘Trip makes finding evidence easy’ but with this technique we could also claim that ‘Trip easily helps you not miss key evidence’. Not quite as succinct, but you get the picture!
This is a value added service so I envisage it only being available to Premium users of Trip. Hopefully another reason to upgrade.
I have the great honour of being part of the KConnect consortia, recipients of an EU Horizon 2020 innovation grant. Trip is involved in a number of great projects within KConnect and I plan to blog about them all over the course of the year. The first to feature is enhancing our PICO interface.
Asking questions may seem straight forward but it can be difficult so by helping users understand the key elements of their questions it typically gives the questions better structure. PICO stands for:
- P = Population (eg what condition the user has)
- I = Intervention (eg a drug, diagnostic test)
- C = Comparison (eg an alternate drug or test)
- O = Outcome (eg mortality, QoL)
Take these two, real question:
- How can you safely treat constipation in pregnancy?
- In diabetes would an AIIRA benefit over an ACE?
In the top Q, the P = pregnancy and the O = constipation. Alternatively the population could be pregnancy and constipation.
The second Q is more complicated but the P = diabetes, I = AIIRA and C = ACE inhibitors
You’ll note that questions don’t need all four elements; it’s a flexible concept! Irrespective of the number of PICO elements it can be really useful in helping users think about the key elements of the question they may have.
From user feedback I hear time and time again that the PICO interface is great and really helps health professionals think through their questions.
KConnect is helping us improve it still further! We will simply allow users to type our their question in full and press search. We will automatically attempt to identify the PICO elements and then pass those elements to our search. By highlighting the suggested PICO elements it will teach users by experience what the PICO elements are as well as speeding up the question answering process.
A further minor step – which might be really interesting – is to record the full question and the articles the user subsequently clicked on. It’s not quite the same as a full answer, but a ‘half way house’.
We’ve a good few months of work on this using, various techniques: machine learning, semantic annotation, hard work.
I’ll keep you posted.
Our previous post Flibanserin and blogs highlighted how adding blogs to the Trip index can be really useful. But the search for flibanserin on Trip highlights lots of issues.
As you can see from the above image the ‘Ongoing clinical trials’ filter shows 14 closed trials (these are trials that are no longer recruiting) and that there are 8 controlled trials. So, we can spot a shortfall of 6. So, is that an issue? It can be as it can suggest hidden trails, which are never a good thing. I had a superficial look and found a couple of things of interest:
- A number of the closed trials were halted for ‘administrative’ reasons. I’m not sure how satisfactory that is.
- Our controlled trials are identified via a filter and this has over-identified controlled trials and there are only actually 6 controlled trials in our index. So, while we over-identified some, we possibly under-identified others.
Irrespective of the above points Trip makes it incredibly easy to spot potential publication bias. If you’re interested in unbiased evidence this feature alone is very useful!
Trip Database Blog 
Recent Comments