The “user review” is standard at gaming websites now. This means you don’t just have an “expert opinion” argued by one of their writers or freelancers, you get a rating derived from how an army of gamers evaluated the game and a few brief reviews from some of these same people. 1up, IGN, Gamerankings…all have some way of getting into your gaming brain. Using this rough metric of popularity, by the way, Gamespot is clearly the number one gaming site. It has ten times the user votes for Battle for Middle Earth II that IGN has. Some games only get a few dozen votes, but those popular ones that have thousands of votes, make them as close to “public opinion” as you can get in this hobby.
And you know what? According to Gamespot, reviewers are too hard on games.
Mind you, I only looked at the sixty strategy games that Gamespot reviewed in 2006. So this is probably not a representative sample or anything like that. My SPSS license expired years ago, anyway. But, until I start seeing box scores in the Washington Post again, this will have to do to sate my stats fix.
The average user score was almost a full point higher than the Gamespot score. The biggest difference (UFO: Aftershock) was over three points, and there were over 500 user submitted scores. Rise of Legends had over 2000 user votes and the people think that Gamespot’s reviewer missed it by a point. The combined wisdom of 773 voters agrees with me that Rise and Fall: Civilizations at War is an underappreciated title.
Of these sixty strategy titles, only ten had user scores lower than the official one (usually not by much) with Dominions 3 being the bigger loser (7.6 from the people, 8.2 from the corp) but with only 89 votes. 3500 people think the official ruling on Galactic Civilizations II was a tiny bit too generous.
These votes are a great way to find out who is playing what (or who is claiming to be playing what). The three most voted on strategy games from 2006 were Empire at War, Company of Heroes and BfME2. There were more votes for The Apprentice than for Birth of America, Moscow to Berlin: Red Siege or Brigade E5.
You also learn that some people will like anything. 4 voters consider Moscow to Berlin: Red Siege to be “great”. 19 people think Left Behind: Eternal Forces is “perfect”.
What about the user reviews themselves? They range from fanboy blather to anti-fanboy hate, but barely ten per cent of voters bother to submit reviews. You can see that the much maligned 7-9 scale is still strong with the people, since a 5 or 6 score is often tied with a few words lambasting a game – “overrated” seems to be the most popular word. But by and large, over the long run, fans seem to write more than detractors. The short term seems to be the opposite.
The big business question is who pays attention to these numbers, besides armchair pundits like me? There are hundreds of user reviews for Company of Heroes, and I’m not going to read them. Do developers take solace from the fact that most users think that Gamespot was too hard on Joint Task Force? Do Gamespot editors scan the user reviews looking for the next Jason Ocampo? On all these sites you can vote for reviews or reviewers you find useful, but is this any more than ego stroking? Does this democratization of opinion prevent us from having the net cluttered with fly-by-night gaming websites competing for eyeballs?
They’re great, don’t get me wrong. Anything that lets users feel like a part of your site is something to be applauded. And they can, I suppose, point friends to their reviews. But I wonder if the website managers think that these votes and user opinions are part of the community.
As with all non-scientific polls, user-ratings obviously need to be taken with a grain of salt. For one thing, specific titles are only really likely to receive reviews from people who have actually purchased/played a particular game, and then only those who have bothered to invest enough effort in the game to form an opinion. Professional sites have the moderating factor of having to play/review the crappy titles in addition to the popular ones. Suffice to say, I would expect any user rating to be higher than expected, if only because of participation bias.
An exercise I’ve found to be interesting is to comb the user reviews in order to determine what (a) people who hate your game seem to like about it and (b) what people who like your game seem to hate about it. Of course the usual warnings about non-scientific polls apply, but it’s still an interesting exercise. Another exercise I’ve used has been to ignore the magnitude of love/hate and to simply give each positive mention of an attribute +1 point and each negative mention a -1 point. While I wouldn’t use this sort of information gathering exclusively, it certainly helps to round out the picture.
I don’t know what to make of user reviews myself. If a game has hundreds of “reviews,” does that mean anything more than it sold enough copies to generate responses from a fraction of the gamers who played the game? Some games have a natural customer base that is larger than other games, regardless of their inherent “worthiness.”
I’ll scan user reviews but generally they mean little to me; if they’re too positive or negative, I’m apt to disregard them wholesale. I’m very unlikely to read many of them. I’ve been a member of gaming forums for too long to believe that most games are half as good as they say or half as bad. Frankly, for me, most demos give me a better idea of whether I’ll want to stick around for a game; and ten thousand “this game rocks” comments won’t change that.
As for the discrepancies between professional reviews and gamers’ reviews, I’m not surprised. It’s not as if reviewing were a science and we were all equipped with the same yardstick and tools. Although, I’d be interested in why the discrepancies were relatively huge. You mentioned UFO: Aftershock, a game whose bugginess and lack of originality caused it to receive a 4 out of 10 from Gamespot (one of the lowest reviews it received btw). Well, I never played the original un-patched and unmodded game, but I have played the patched, modded game and I’ve enjoyed, truly enjoyed, the game’s tactical and strategic combat. I’m not claiming it’s particularly original or does much to advance beyond what X-Com did as well (or in some ways better), I am saying that I’ve had a lot more fun than Gamespots’ 4 would lead one to suspect one could have. I’d easily give it 8 out of 10. Dismiss me with “some people will like anything.” No matter. The same thing is said about Dwarf Fortress or Dominions or…