One thing I learnt while studying the diffusion of innovations and social networks was that the greater the uncertainty the more likely we are to turn to people for advice/reassurance.
Two recent personal experiences highlight this phenomenon. Firstly, I was looking for places to go on holiday. There were a number of companies offering the type of holiday I wanted (sailing/activity) and from multiple locations. All were broadly similar in cost, had similar weather and facilities. So, to help me decide I took to TripAdvisor and located all the potential targets and chose by looking at user ratings and comments.
The second experience relates to me buying a new camera (which broke while on my holiday) I wanted a particular type of camera and to help me decide I went to Amazon and again looked at user ratings and feedback. In the end the newest version of the camera I wanted had pretty poor reviews, so for now, I’ve decided not to buy a replacement and to simply rely on my relatively good mobile phone camera.
I’ve been reflecting on this theme as recently, two separate users of Trip have floated the idea of introducing such a feature in Trip. I posted the idea on our Facebook page over the weekend and idea was received quite positively (based on a small number of responses).
I like the idea as it can help give context to the research, give different perspectives and perhaps help highlight potential problems with the evidence. There are associated problems such as potential bias, inaccurate comments etc. But I’m sure these negatives can be mitigated for, with some thought.
So, what might a rating/comments feature look like? I have my ideas, which I’ll highlight below but I’m really keen to obtain feedback from you. This feature, if it is to be released, will not happen till next year – but it’s useful for me to reflect on ideas.
One thing that is essential is that the system is easy to use and understand. I would like it to be more than a binary ‘good or bad’ or ‘thumbs up or down’. On both TripAdvisor and Amazon I like to look at reviews by score. So, for those who gave a holiday/product a low score, why was that and vice versa for high scores? So, I think it requires a numerical scale and both TripAdvisor and Amazon use a 1-5 scale (although, someone pointed out that people tend to gravitate to the middle).
When people have scored an article we should offer them the ability to comment. We could suggest a structure to comment against (e.g. what did you like about the article, what did you dislike etc) but I think the more formalised you make it the less it’ll be used. So, I favour a free-text response.
The results would need to be displayed somehow but I’ll not give that much thought now, I think our designer would be the best to advice on this. Needless to say it needs to be clear and easy to understand. I also like the idea of being able to sort results by rating (currently, on Trip, you can sort results by relevance and date).
There are still questions to be explored (apart from the big one of do people think it’s a good idea), such as:
- Will it be used? I’ll explore this further with our users over the next few months (after we release the new version of the site – due imminently).
- What will the score represent? I think one can over-analyse. But, I think it indicates a user’s view/opinion on a paper and what this represents is individual to the person. With TripAdvisor and Amazon the comments help explain their rating and you take the points that you feel (as a consumer of the ratings) are important to you.
- Do we ‘seed’ the scores? We could create a starting score for each document to get the ‘ball rolling’. We could create a score based on the quality of the publication and how often it has been viewed. As people submit ‘real’ ratings the seed-score diminishes in worth.
I often get excited by new ideas and this is no exception. If ratings/comments take off I think it will impact how users consume evidence. It’ll also deliver something that is hugely beneficial, yet potentially immeasurable – value!
Finally, now you’ve read this, please do our two question survey – click here.
October 8, 2012 at 3:32 pm
I'm looking forward to seeing the responses to this! In general, as you know, I have not been in favour of ratings and have not been able to see the utility. Just like you I love Tripadvisor and reviews on Amazon. Why? Because these are quite high stakes purchases.. cost a reasonable amount of money, and even more importantly, may have quite an impact on my holiday experience. AND I have no way of really assessing the product until I hand over my money and the product arrives or I turn up on the doorstep of where I an staying. So therefore the subjective opinion of other people who may or may not be similar to me, and who may have different values, is worth exploring before I make a purchase.
Compare this to how I use Trip. First of all I am looking for real-time information when I search. I am going to use that information there and them. I'm not trying to decide whether spending money is justified. If I think that an article in search results meets my needs I click on it instantly and decided there and then whether it meets my needs. If it doesn't then I click on another article. Reading the reviews of others would take longer than me just evaluating the article myself.
So that is why I don't think it would be useful to me.
October 8, 2012 at 4:45 pm
I'm also very keen to see what people make of the idea (on Fb over the weekend, the general view was positive). If we don't get too many responses I'll ask people via the Trip newsletter, which goes to around 37,500 people – we should get a half-decent number of people responding.
One thing I've come to appreciate that Trip is used by lots of people in lots of different ways. You, a GP, use Trip in a completely different way to Chris – another GP. I think, if we do introduce it, we have to do it in such a way as to not detract from the core Trip function. Fortunately rating don't have to be too 'in your face'.
I think, for many people, the decision making process is complicated and fraught with uncertainty. The cost can be high (perhaps not financially). So, I think there may be a function, but that is dependent on the number and quality of the comments.
October 8, 2012 at 4:49 pm
I'm also interested in how a ratings system would help other GPs like Chris- so maybe get him to post? I might be missing out on how I am using Trip. It's easy to assume that everyone does things the same was as you.
Maybe we could start a series of screencasts about how we use Trip?
October 8, 2012 at 5:46 pm
It's not a question of missing out, just using it differently. You were the only one to ask for Patient Decision Aids but they get used.
Some use it for research (we're mentioned in multiple SRs), some for decision support, essays etc.
My view is that we try and support users as best as possible.
I like the idea of screencasts, but not sure if others would be as enthusiastic!