Numbers Lie

Mitch Gitelman is not a happy camper.

The lead Shadowrun developer has been making the rounds on the podcast circuit, discussing in great detail why he feels his game is getting the short end of the stick. Gitelman lays a lot of the blame on the press, claiming that reviewers are being too harsh on his game and holding the title up to unfair standards. On the Official Xbox Magazine podcast this past Monday he points to middle-of-the-road review scores as keeping people from realizing that this title is an instant classic. Game reviews are broken, he claims, and it's screwing over developers who want to take risks. Ballsy stuff, especially coming from a guy who isn't hiding the fact that he wants to sell more copies of his game.

Thing is, Mitch has a point.

Just so we're clear, this is not a review of Shadowrun. I haven't played the game, and I don't know if it's good or not.

On the OXM podcast, Senior Editor Ryan McCaffrey discusses the reasons he gave Shadowrun a 7.0, stating that he enjoyed the multiplayer experience, but he, "didn't think there was enough variety within the maps, and I just felt like it was hard to justify $60 for the experience that it gave." Gitelman asks him if he thought the game was worth playing, and McCaffrey says yes, going so far as to explain just how much the OXM crew has been playing. Gitelman asserts that OXM should have scored the game at an 8.5 and mentioned in the review that maybe there should be more content, but what's presented is top notch. Gitelman is, of course, as objective as possible in this case.

7.0, 8.5. Who cares? Would a point and a half really sell a million more copies? Taken on their own, these numbers tell us nothing about the game.

According to OXM's scale, a 7.0 indicates a game that has a lot of things going for it, but still has a few major issues, or something that limits its appeal. That's fine for OXM, but an arbitrary value like that means different things to different people. I don't think a 7.0 is a game that deserves my attention, and I'm pretty positive that most people feel the same way. The Xbox 360 version of Shadowrun is hovering around 70.3% on review score aggregators like GameRankings.com, and that number means radically different things to developers, reviewers, and consumers.

Part of the problem with these scales is that they're top heavy. Most games are scored no lower than a 6 on a 10-point scale, with the lower scores reserved for reviews of games that are comically bad. If all the values were equal, a 6 would be a pretty above average title. It's higher than 5, and 5 is square in the middle. The way these scores are implemented, however, adds weight to the 8s and 9s, marking anything below an 8.5 as cursed and branding the title as a missed opportunity in the eyes of the reviewer. No one rushes out to buy a game that 1up thinks is a 7, not while we could be spending our scratch on the games that show up in the top end of the graph. For $60, we want blockbusters, not a game that's just barely above average.

Why not abolish review scores entirely? GWJ abandoned writing "reviews" in the traditional sense, and instead focused on Perspectives. A semantic difference? Sure. But one which is at least symbolic in its assertion that each person brings with them their own unique point of view. The problem with not using scores, however, is that no one actually reads reviews. Many of them read like shoddy instruction manuals, detailing gameplay and mechanics in far too much detail. Others are poorly written, with no flash or style. And if people aren't reading your boring reviews, that means they're probably not picking up your magazine. So editors assign a score to give people a chance to determine the value of the game without all that difficult "reading comprehension" business. Even if the value means something different to the reader than the author.

Reviews just aren't important anymore. The Internet and its gazillion ways to communicate have replaced the need for magazines to tell us what deserves our attention. Your online friends probably bought The Darkness already. Do they like it? Chances are there's already a thread with their impressions. Your fellow forum dwellers may not have the "journalistic integrity" of the gaming press, but you know what their tastes are. Many of these communities dove head first into Shadowrun, setting up games night after night, and many of those guys didn't care what number IGN used to judge the game. They played the beta and the demo, decided it was the game for them, and talked their friends into their purchases.

Reviews didn't help Shadowrun, but they certainly didn't kill it either. Unfortunately for Mitch, Shadowrun had perception problems in the community long before the final product shipped, and an 8.5 review at OXM probably wouldn't have eased the pain much. Ryan judged the game on its merits and gave it a score that fit his opinion. It's too bad the number alone is meaningless, and it's worse still that the number is all the industry cares about.

Comments

SommerMatt wrote:

This is sort of the same problem with the ESRB-- you have game companies giving the review board a "highlight video" of their game in question... the ESRB doesn't have anyone actually play the game through, which is why they occasionally are surprised with something that the publisher didn't tell them about.

I had something similar to this in my post originally but dropped it due to not being eloquent enough to finish the thought in a decent matter, and currently I'm too drunk to even consider it.

I find it interesting that so many people say they just "want the facts" and that reviews should dispense with extraneous commentary. That extraneous commentary can tell you a great deal about the reviewer and if he or she is someone you find palatable and ultimately trustworthy. Reviews are commentary, not advertisement, and there is I feel almost a duty of the reviewers to reveal a little bit about themselves in the review. This is especially true of a reviewer who is new to an audience. It establishes a rapport with the reader and that is how a reviewer would build trust. If all they gave us were the "facts" these reviews would mean very little.

Edit Spelling Failure Error 10010

professorvonbeardzine wrote:

I find it interesting that so many people say they just "want the facts" and that reviews should dispense with extraneous commentary. That extraneous commentary can tell you a great deal about the reviewer and if he or she is someone you find palatable and ultimately trustworthy. Reviews are commentary, not advertisement, and there is I feel almost a duty of the reviewers to reveal a little bit about themselves in the review. This is especially true of a reviewer who is new to an audience. It establishes a rapport with the reader and that is how a reviewer would build trust. If all they gave us were the "facts" these reviews would mean very little.

Edit Spelling Failure Error 10010

I would be totally behind that if reviewers actually sounded like they have distinct personalities and managed to let those come forward throughout the review. Writing is an art, however, and most reviewers, in my observation, are piss-poor writers.

WolverineJon wrote:

Why don't publishers furnish reviewers with a copy of the game that allows easy access to the various sections of the game, to facilitate quickly and easily including all of those sections in the game's review?

Because that's not how the player will experience the game. I don't care if the last hour of a game is fantastic if there's a 5 hour section of dull fetch quests before it, because I'd never get to the last hour. The reviewer's experience should match what the reader would experience when they buy the game: allowing the reviewer to skip around the game means they have a different experience.

These days word of mouth (word of forum?) is more useful than reviews. I remember reading reviews of Battlefield 2 on sites like Gamespot saying how great it was, without a single mention of the countless show-stopping bugs.

I agree with Zelos. Allowing the reviewers to by-pass the crappy parts of the game that regular joes can't avoid? Seems like a great way to guarantee a glowing review of what may otherwise be a crappy game.

I just realized what I do when i read reviews. I check the number score to see if it's above a 7 (leeway if I like the pictures or have prior knowledge of the franchise). I then check blurb list of the good, bad, and ultimate take on the game. If I'm still intrigued or confused and if I have free time, I'll skim the first few and last paragraphs.

I think that in the days of the blog and with so many gamers getting their news and reviews online, anything more than a 5 paragraph review is a boondoggle. K.I.S.S.

K.I.S.S.

Ramen! Unless its an antireview like the Vanguard diatribe. I could have probably read that for days should Elysium have turned that into a novella. Is that my guilty pleasure schadenfraude skeleton-in-the-closet?

Quintin_Stone wrote:

I agree with Zelos. Allowing the reviewers to by-pass the crappy parts of the game that regular joes can't avoid? Seems like a great way to guarantee a glowing review of what may otherwise be a crappy game.

I understand where you guys are coming from. It seems like we have a couple of possible outcomes here:

(1) Publishers provide reviewers means to start playing at the middle or end of games. Publishers quickly figure out that the save game files (or equivalent) that they are providing the reviewers with are those that showcase just the very best parts of the game. The reviewer plays just those sections, concludes that the game is great (when there really may be boring stretches that they didn't play), and gives the game a better review than it actually deserves.

(2) The current state of affairs -- publishers don't provide reviewers with save game files, and reviewers are only able to start playing from the beginning of the game. The reviewer hits a long and boring stretch and consequently doesn't end up playing through the entire game (even though the genuinely fun part might be just after the boring part), and gives the game a lower score than it actually deserves, reflecting only the boring part not incorporating that fun part after the boring part they didn't reach -- which they otherwise could have incorporated in the review, with the caveat that the player must play through the boring part to reach the fun part.

Seems neither of the above is really a good outcome. But how about this:

(3) The reviewer does get access to mid-game and late-game savegames. They play at those, and find that those parts of the game in fact are really cool. THEN, given that they know there are cool parts later in the game, they invest the additional time to play through the whole game from the beginning (as they otherwise would have, and as the actual player/customer will end up doing), and review the game based on that -- knowing that if they do hit what seems like an interminable boring stretch, there are still fun parts coming up that still should be reflected in the review.

In other words, the reviewer plays *both* the publisher's "highlight reel", and then the full game all the way through (as time permits).

Would that lead to fairer review scores? Or does that presuppose a perfect word in which the reviewer actually has all the time they need to do a through reviews of games?

Maybe developers should just avoid long boring stretches?

Rea-ding?

The ratings are going to be biased and critical in the same way that movie criticisms are. I've read many a movie review and found that critics often apply a base generic to all movies, which isn't fair. For example, a campy horror flick shouldn't be based on the same criteria as a romance comedy. They're trying to do the same thing, entertain, but their definitely going about it differently. The original Scream was a pretty entertaining movie, but to compare it to, say Shawshank Redemption standards is pretty unfair. I watched and enjoyed both. Movie critics tend to see movies as trying to reach some pinnacle or search for their holy grail (or popcorn) and until that happens (and it won't) they still have a job as long as the studios continue pumping out.

I think the same applies for the computer game reviews, which are built in much the same manner. They are generally using base criteria and broadly applying it to all games, regardless of intended audience or intended effect. For example, a few years back, CGM printed a review about Barbie the Game (must have been a slow month) which was quite witty and funny but totally missed the point because they reviewed it for their audience and not as a, say 5-10 year old girl.

Obviously, there needs to be an assessment on every game they decide to review by understanding what the target audience is for the game and do the producers of the game live up to what they've promised to deliver. This seems pretty simple, but apparently, it doesn't sell magazines or adspace.

BlackSheep wrote:

The ratings are going to be biased and critical in the same way that movie criticisms are. I've read many a movie review and found that critics often apply a base generic to all movies, which isn't fair. For example, a campy horror flick shouldn't be based on the same criteria as a romance comedy. They're trying to do the same thing, entertain, but their definitely going about it differently. The original Scream was a pretty entertaining movie, but to compare it to, say Shawshank Redemption standards is pretty unfair. I watched and enjoyed both. Movie critics tend to see movies as trying to reach some pinnacle or search for their holy grail (or popcorn) and until that happens (and it won't) they still have a job as long as the studios continue pumping out.

Personally, I've rarely seen a critic who doesn't take this into account.

Still, professional criticism is about giving your opinion about something. Some people LIKE horror movies (so their reviews will be more biased that way) and some people don't (which will cause reviews to bend in the opposite direction). The critic is rating something based on how much it "entertained" them (or moved them, made them cry, etc), and so while they can certainly understand what someone was "trying to do" in a particular case, the still shouldn't give it a positive review for that. This is why I sort of get annoyed by critics who say things like "well, I didn't like it... but your kids will!" How do THEY know kids will like it? They're not kids... they can (and should) only speak for themselves.

Your job, as a consumer of reviews, is to find a critic who shares your own tastes.

I tend to read 3 types of reviews: 1.)Games that score ridiculously high 9 or better that I have heard about from some site or anothers. 2.)Games that I am already interested in and will probably buy anyway regardless of the rating. 3.)Games that score so ridiculously bad that it might be entertaining to read the review.

I usually pick up what I am interested in regardless of the review and really only pick up something that got a great review if I am absolutely bored with everything else I am playing. So while I do read tons of reviews they rarely influence my decision on games I am at least moderately interested in.

On the Shadowrun thing... I and pretty much every single person I have talked to in RL that plays computer games... Well we didn't pick it up for the combination of two reasons: 1.) FPS games kinda suck on consoles IMO. 2.) You need Vista to play it on PC.

No one I know is going to upgrade to Vista solely to play some mediocre FPS. Especially not when there are plenty of games coming out in the very near future that will more likely than not run better on XP.

@WolverineJon

(3) isn't a bad idea as I read it. That would be giving reviewers a copy of the full, unrestricted game, as well as suggestions/a highlight list and save files to start from.

That supposes that developers would have time or consider it worth the extra effort to create those reviewer copies, and that reviewers would even want them. Some might be insulted that the developers were trying to direct them or do their job for them.

The other options are adding more hours to the day for a game reviewer. Or improved caffeine.