jrbtrip

Are you a luddite?

Many will be familiar with my post A critique of the Cochrane Collaboration, it’s been the most viewed article ever published on this blog. Continuing in the theme was Some additional thoughts on systematic reviews and more recently Evidence, hourglasses and uncertainty.

They all point to the current methods employed in systematic reviews (as exemplified by Cochrane) being a mess! As a type of summary, a few problems:

They can’t be relied upon to be accurate
They’re financially costly
They’re typically out of date
Significant opportunity cost

More evidence that my perspective is correct is shown is the presentation Does access to clinical study reports from the European Medicines Agency reduce reporting bias? submitted to the Cochrane Colloquium in Vienna. The conclusions:

“Unpublished clinical study reports held by EMA may be a useful source to reduce outcome reporting bias.“

It’s a testimony to Cochrane’s openness to allow such ‘dissent’ to be published. It’s a dissenting view as the current Cochrane methods rely almost exclusively on published journal articles; unpublished clinical study reports – virtually unheard of. But that’s Cochrane; on one hand a business trying to maximise it’s business model but on the other a collection of individuals doing there best to improve methods. As an outsider, I find this tension fascinating. Why? Because the best one can say for Cochrane’s methods (alongside most other SR producers) is that they are likely to produce ‘ball park’ accuracy – in most cases you simply cannot rely on the methods to produce results you can trust. And this is where the tension comes, if they were being transparent, they would say ‘buy the Cochrane Library, most of the SRs are likely to be out of date and many are likely to be innaccurate’ which even I can see is not great for sales.

But this bring me nicely to diffusion of innovations – an area I studied for the PhD I never completed! The Wikipedia article summarises it nicely by saying its “..is a theory that seeks to explain how, why, and at what rate new ideas and technology spread through cultures.” The spread of an innovation is characterised as an S-shaped curve:

In relation to systematic review methods, where are you on this scale? If, like me, you think there are serious problems by relying on published journal articles you’re probably at the innovator side of things. If, however, you think current methods are great and there is no need to change you may well be a laggard. But, the majority (those that are feeling uneasy) are likely to be in the middle.

August 28, 2015 1

Clinical area tagging of documents

Around 6 weeks ago I wrote the article Logging in to Trip which provoked much negative comment. I have clearly not rushed to reply properly but that’s because I’ve wanted to properly think through the issues. In short, the reason for asking people to log-in meant we knew more about the users and could therefore improve the service we deliver. As I see it there are two connected issues of concern:

Logging in – it’s a pain for users.
Profiling users and altering the results accordingly.

One suggestion (from Paul, see comments in previous post), in relation to the second point, is to focus on our refine by clinical area feature. This is an already available system which tags documents by clinical area. So, an article titled ‘Cholesterol and the elderly‘ would be tagged as cardiology and geriatrics. If a user did a search for, say, cholesterol the above document would be returned in the results, alongside many others. But if they decided to refine by geriatrics, the above document (alongside others tagged with geriatrics) would be moved to the top of the results.

Below are two images showing how it currently works:

The differences are clear and shows the potential for the system. For me, it’s about helping users find the answers they need really quickly. An anaesthetist, interested in ‘awareness’, would find the results of the ‘normal’ Trip disappointing but by selected anaesthesiology all the results are relevant.

As mentioned above this system is available already and can be accessed as shown below:

It’s not particularly developed and we could definitely improve it:

Machine learning to improve the document tagging
Better user interface so it’s more apparent and more intuitive to use.

Ultimately, it might serve a similar role as profiling users in improving the search results.

August 27, 2015 0

A multi-lingual Trip

I have the great pleasure of being part of an EU-funded project called KConnect. It’s a group of academics, health care providers and commercial organisations (such as Trip) working together with the broad aim of innovating to improve search.

One early result has been a collaboration with the Institute of Formal and Applied Linguistics at Charles University in Prague to introduce a very nice multi-lingual tool for Trip. It allows users to search in French, German or Czech, with more languages due in the next 6-12 months.

As you’ll see in the image above we have added a discreet link for ‘language options’ which, when pressed, reveals the three language options. Once pressed the user can enter search terms in the selected language.

In this second image you’ll see that German has been selected and the search term bluthochdruck added. This has been translated to hypertension and the results have also been translated into German.

It’s a very simple yet powerful system which will only improve over time as the translations get even better and more languages are added.

August 21, 2015 0

Subscriptions and geography

Below is a pie chart showing the country of origin of every personal subscriber to Trip.

August 20, 2015 0

Article analytics, again

Earlier today in the post Article analytics I said “This latest feature will be released soon.” Little did I realise it would be live by the end of the day!

In the above image I’ve highlighted four key areas:

Analytics – appears under every link (for Premium users only), this is clicked to generate the data below.
Related by viewer – these are articles that have been clicked on during the same search session as they had clicked on the main article (Canadian clinical practice guidelines for the management of anxiety, posttraumatic stress and obsessive-compulsive disorders).
Viewers by country – this highlights where the users originate from who did the clicking!
Viewers by profession – as above but broken down by profession

NOTE: the above example is very rich as it’s clearly a very popular article. Others will have considerably less data, another reason why we’re keen to get users to login!

July 28, 2015 0

Article analytics

This latest feature will be released soon. For a given article premium users will be able to see related articles (based on clickstream data) as well as information on total views, views by country and views by profession…

July 28, 2015 0

Evidence, hourglasses and uncertainty

Long-term readers of this blog will know I struggle with many aspects of the systematic review process. At the time of writing, my ‘A critique of the Cochrane Collaboration‘ has been viewed over 18,300 times and ‘Ultra-rapid reviews, first test results‘ nearly 10,000 times.

I believe the main justification given for conducting systematic reviews is to obtain a really accurate assessment of the effectiveness (or ‘worth’) of an intervention. So, the thinking goes that spending 12-24 months is worth the cost (financial, opportunity, etc) due to the accuracy of the prediction it then gives.

My immediate response is that is demonstrably false. In my article ‘Some additional thoughts on systematic reviews‘ (just under 5,000 views) the evidence is clear that if you rely on published journal articles to ‘inform’ your systematic reviews (which is the case in the vast majority of systematic reviews) there is approximately a 50% chance that the effect size is likely to be out by over 10%.

But, even if we suspend being evidence-based and believe that systematic reviews can be relied upon to give us an accurate estimate of an effect size, is everything fine? I don’t think so and the image below illustrates my thinking.

It’s an hourglass! At the top are all the unsynthesised trials, all floating around and the uncertainty is moderate. Someone then spends 12-24 months pulling these together in a systematic review (likely of published trials and therefore ‘a bit dodgy’) and the certainty is reduced at the aperture of the hourglass. But then, when you apply it to the real world of patient care, the uncertainty flares out again. In the above example the intervention has a NNT of 6, so the intervention needs to be given to 6 people to obtain the desired outcome in 1 person. Which is the 1 person? Where’s the certainty?

If we were to spend significantly less time doing a review it might indicate a wider hourglass aperture (perhaps suggesting an NNT of 5-7). In what situations does that matter? I don’t think we’ve even started to explore these issues. In other words, when is it appropriate to spend 12-24 months on an systematic review and when is a significantly less resource intensive approach ‘ok’?

Is it irony that the reality is the type of review (systematic versus ‘rapid’) doesn’t alter the effectiveness of an intervention? After all the compound remains the same, untroubled by the efforts of trialists. Sorry, getting sociological there – must be time to sign off for now.

July 21, 2015 0

Clever stuff with the help of QSPectral

At the start of the year I posted Ok, I admit it, I’m stuck, which was a cry for help from the Trip community to help me make sense of all our lovely clickstream data. We had a few responses and one was from an Australian research and management consultancy QSPectral, a company specialising in providing strategic insights and predictions through advanced data science and analytics. They have been working with us to help us make sense of our clickstream data.

Article Association
QSPectral used their data science expertise to investigate the connections between the articles based on the user access data contained within the Trip Database.

Figure 1 Snapshot of articles accessed across a session. The colours represent user professions (doctor, nurse, etc.)

In the above image the Y-axis represents individual search sessions and the X-axis is the documentID (each article in Trip has a unique document ID). So, we can see what professions are looking at which articles. We can actually see what articles individuals are looking at, but the above image shows it on a profession basis.

Figure 2 A more focused snapshot of the previous image

As a user do you want to see what other articles are similar to the one you are reading?
Do you want to know what others like you thought were similar?

To provide answers to these questions, QSPectral developed an algorithm based on association rules to explore the relationships between articles on a per session basis. We intended to identify links between articles based on different criteria of interest.

The strength of the links was measured by statistical measures such as confidence and support factors. These led to association rules, which were of the form if {article x is accessed then articles y and z} were also accessed were further enhanced by including additional user characteristics – information such as the profession (nurse, doctor..) as well as country of origin were used to moderate the previously established article relationships.

Figure 3 Snapshot of related article numbers – if the articles on the y axis are accessed it implies those on the x axis would also be of interest.

The data can be further augmented by adding clickstream data that includes the area of speciality (such as cardiology) for a user, where the for example, if you are a doctor from Spain only relationships between articles that doctors from Spain accessed could be isolated and uncovered. It was also possible to group the related articles in clusters based on this multi-dimensional relationship – defined by colour in the figure.

Figure 4 clusters of articles based on relationships

The purpose of this initial investigation was to set the stage for providing users with recommendations based on their initial article of interest and their particular user characteristic. A slightly different approach to PubMed’s ‘related articles’ feature.

As well as finding closely related articles QSPectral have helped us explore recommendations of new articles. So, if we know a user’s activity on Trip we can start to understand them and then – with QSPectral’s help – recommend new articles that should be of interest.

Article Recommendation

How will TRIP recommend articles for you?

Machine learning methods based on clustering and classification are being investigated for providing reliable recommendations.

We believe that initial article clusters should be identified using an algorithm known as k-means clustering. Each user will then be classified as being interested in articles within a cluster based on attributes such as their first choice of article and user attributes (profession, country etc.) using a method where a tree-like graph or model of decisions and their possible consequences, including chance event outcomes, resource costs, and utility is created.

Figure 5 Example of a Decision Tree where the top node could represents you and the other nodes represent related articles based on branch criteria.

QSPectral determined that decision trees are the most appropriate concept for meeting the requirements. Decision tree methods can accommodate more data inputs over time. Various other transformations of inputs are possible and are robust to inclusion of irrelevant fields in the data, and produces transparent models for on-going analysis.

Further, we will use other methods that take a number of simple decision trees and combine them in some way to yield a final overall picture. We propose techniques for iteratively averaging multiple deep decision trees, trained on different parts of the collected data, with the goal of reducing the variance. Each iteration creates a simple decision tree on randomly selected subsets of input variables and input data. The final result where recommendations are provided will be formed through classifying a user through the aggregation of all such trees.

July 17, 2015 0

	How AskTrip’s new se… on What Is Vector Search?
	A record day for Ask… on A record week for AskTrip
	Help us shape the ne… on Learning from user feedback: h…
	When good evidence g… on HTML Scissors
	A Research Agenda Bu… on Turning Research Into Practice…

	How AskTrip’s new se… on What Is Vector Search?
	A record day for Ask… on A record week for AskTrip
	Help us shape the ne… on Learning from user feedback: h…
	When good evidence g… on HTML Scissors
	A Research Agenda Bu… on Turning Research Into Practice…

Trip Database Blog

Liberating the literature

Author