It’s been a long time in the planning, but we have finally released a whole new group of features in the BMJ learning site.
The ratings, reviews and recommendations project has several factors.

Anyone who has ever been in a meeting discussing improving a website will have heard someone say, “can’t we make it like Amazon?”. We’ve added star ratings, which are displayed at the top on each module intro page.

 

We had a lot of debate about how to display the star ratings. We discussed how we could make the star rating useful for our users as well as fair to authors and editors. There’s a lot of information you can put into a rating system and we even debated having stars at all  – YouTube blogged  about why they moved from stars to thumbs up and thumbs down. In the end we chose to display a simple average star rating, with the number of ratings that contribute to this average displayed next to it.  It displays the average rating captured over a rolling one-month window, to allow ratings to take account of updates to modules.

Introducing public ratings on a site for the first time is a daunting prospect, and everyone on the team had some reservations about how it would play out. But 2 weeks down the line it has, subjectively at least, been a success. First of all, the intro pages look better. Engagement and interaction are key factors to the success of any piece of education and a rating is a visible demonstration of this. Second, we’ve been doing well. We had hundreds of ratings in the first few days and it continues. We haven’t found any ratings lower than a “3” and several of our most popular modules are on “5”, which is a real source of pride for the editors and authors who have worked hard on them.

One of the reasons that we were comfortable about introducing star ratings is that we have always collected, read and displayed a huge number of user reviews, which are overwhelmingly positive. Users won’t see it, but we have a lovely new admin tool to moderate the reviews (someone looks at every review, and any review that raises a clinical question relating to the module content is reviewed, and acted upon, by a clinical editor).

We have anonymised the reviews. An unintended consequence of making our module intro pages more easy to find on Google, was that the names associated with reviews were picked up too. Quite reasonably, we had feedback from users that didn’t like their “really useful, thanks” comment as top of a google search on their name. The change has led to a real improvement, each comment is now attributed to the reviewer’s country, profession, and specialty – so if someone says “pitched just right for me” you can tell easily whether it is pitched right for you, as well. We love our user reviews, so we’ve moved them to front and left on the module intro page – we hope users browsing the site will find them as useful as we do.

Recommendations have two facets – the social networking element and the recommendations of other modules the user might like. Social networking tools are pretty straightforward, and we’ll be monitoring their uptake on Twitter, Google+ and Facebook. Thinking about how you might recommend modules for the user is exponentially more complicated than I had initially thought (though that may be a reflection on how much I had thought about it). Amazon has a plethora of different recommendations displays – I’ve counted 5 in two clicks.

In the end we have chosen to display five module titles which users with similar demographics have rated highly. Or, “people like you liked these”. Many features of the new BMJ Learning site are based on matching modules to a user’s professional interests and specialty, which they tell us about in http://myaccount.bmj.com. Whilst our recommendations are good, we have some technology in the pipeline that could make them even better, as well as automatically including recommendations for other resources from across the BMJGroup – Best Practice monographs and possibly journal articles. We’re monitoring how all these new features play out, and do welcome feedback.

– Helen Morant, Editor, Online Learning