Testing at Which?
Which? was once the go-to place for product testing. in fact, it’s no exaggeration to say that gaining a Which? Best Buy award signalled financial success for the makers of such a product. But in an age when the views of users can so easily and so quickly be turned into print, signs are emerging that Which? and its testing are far from what they once were.
Recently, one of our Insight team members needed a new food mixer, their ancient Moulinex having gone the way of all flesh, or its mechanical equivalent. Now, food mixers and their ilk are the very products that Which? used to test, carefully and scrupulously, and they’re also among the very products most commonly needed in kitchens.
What our team member discovered when they searched the reviews in Which? was quite disturbing. As they say
“Looking at the Which? reviews I am shocked how old they are. [Apart from this month they have added an 8 year old Magimix listed multiple times as it comes in four colours].”
Worse was to come however. Which? now includes a comments column, something to which – in this day of social media rampancy – they’ve had to concede. It makes grim reading.
Bearing in mind that it was a recommended Best Buy, user comments on the Bosch MCM4100GB food mixer were in the ratio of 12 – 20 (favourable to unfavourable) with some reviewers describing is as “Disappointing and Hazardous!”.
We wondered how widespread this discrepancy between Which?’s assessments of products and the users’ experiences really was. We also need to point out at this stage that although Which? often refers to ‘our tests’ and ‘our laboratories’ this is playing a little fast and loose with the truth, as Which? no longer do their own tests, nor do they have their own testing laboratories.
User comments on the article, however, were depressing to read – all eight pages of them. Not everyone disliked the machine, but the ratio of good to bad reviews – 12-20 – was a surprise, considering Which had awarded it a Best Buy.
One place we knew there was an issue was in the gardening section. This writer had bought a Bosch ART 35 HD strimmer on the back of a Best Buy award. Within the first few uses it had had to be returned to Bosch so the head could be removed as it proved impossible to do it.
But the reviews beneath the article tell a more worrying story. No fewer than 100% of the reviews are negative, with some being headed “Avoid this machine like the plague” and others “Heavy and cumbersome” and “inferior”.
So why the disparity between the testers and the users, and what use are these reviews if they’re as unreliable as they seem? Another of our team writes
“I still remember ‘researching’ kettles many years
ago (I think Which? had got too grand to do such
stuff); going round the shops, hefting all the kettles
I could find, looking at the ease with which one could
see the water level,how easy they were to pick up
and put down on their stands etc. I finally chose one, which turned out to be as excellent as I’d hoped. Then came an article in Which? by two young male designers.They tore my poor kettle to shreds. It had every fault they could think of in terms of (visual) design. Then finally came a review of kettles by Which?, and my kettle came out as a clear Best Buy.
It’s not simply rather weak reviewing that’s a problem. Members on Which? Conversations discussed the somewhat ropey reviewing pattern in comparison with the Australian consumer group – Choice. One glaring issue was the way in which reviewed items were visually depicted. As can be seen from the image below, Choice shows the food mixers with component parts displayed in full, whereas Which? has a single image without any real detail.
As one team member said “A picture of ‘What’s in the box’ would seem a basic requirement for any review.”
Considering Which? built its reputation on the strength and quality of its reviews this was worrying, but for them to award ‘Best Buy” status to a machine roundly condemned by 100% of its users was starting to suggest a trend. Combined with what seems to be a very poor way of displaying the products and a lack of detail on how the tests were done it’s hard to escape the feeling that Which? are cutting corners and falling down on the job.
A link on the Which? website to their “Who we are” sections makes this interesting claim:
The unique thing about Which? is that we are completely independent. We have no owners, shareholders or government departments to answer to and you’ll never see an advert in our magazines or on our websites.
It sounds grand, but what’s the real truth behind that claim? On our sister site – www.wh1ch.net – in coming articles we’ll be looking at connections their directors have and who the real power behind the Which throne really is.