Numbers Lie

Section: 

Mitch Gitelman is not a happy camper.

The lead Shadowrun developer has been making the rounds on the podcast circuit, discussing in great detail why he feels his game is getting the short end of the stick. Gitelman lays a lot of the blame on the press, claiming that reviewers are being too harsh on his game and holding the title up to unfair standards. On the Official Xbox Magazine podcast this past Monday he points to middle-of-the-road review scores as keeping people from realizing that this title is an instant classic. Game reviews are broken, he claims, and it's screwing over developers who want to take risks. Ballsy stuff, especially coming from a guy who isn't hiding the fact that he wants to sell more copies of his game.

Thing is, Mitch has a point.

Just so we're clear, this is not a review of Shadowrun. I haven't played the game, and I don't know if it's good or not.

On the OXM podcast, Senior Editor Ryan McCaffrey discusses the reasons he gave Shadowrun a 7.0, stating that he enjoyed the multiplayer experience, but he, "didn't think there was enough variety within the maps, and I just felt like it was hard to justify $60 for the experience that it gave." Gitelman asks him if he thought the game was worth playing, and McCaffrey says yes, going so far as to explain just how much the OXM crew has been playing. Gitelman asserts that OXM should have scored the game at an 8.5 and mentioned in the review that maybe there should be more content, but what's presented is top notch. Gitelman is, of course, as objective as possible in this case.

7.0, 8.5. Who cares? Would a point and a half really sell a million more copies? Taken on their own, these numbers tell us nothing about the game.

According to OXM's scale, a 7.0 indicates a game that has a lot of things going for it, but still has a few major issues, or something that limits its appeal. That's fine for OXM, but an arbitrary value like that means different things to different people. I don't think a 7.0 is a game that deserves my attention, and I'm pretty positive that most people feel the same way. The Xbox 360 version of Shadowrun is hovering around 70.3% on review score aggregators like GameRankings.com, and that number means radically different things to developers, reviewers, and consumers.

Part of the problem with these scales is that they're top heavy. Most games are scored no lower than a 6 on a 10-point scale, with the lower scores reserved for reviews of games that are comically bad. If all the values were equal, a 6 would be a pretty above average title. It's higher than 5, and 5 is square in the middle. The way these scores are implemented, however, adds weight to the 8s and 9s, marking anything below an 8.5 as cursed and branding the title as a missed opportunity in the eyes of the reviewer. No one rushes out to buy a game that 1up thinks is a 7, not while we could be spending our scratch on the games that show up in the top end of the graph. For $60, we want blockbusters, not a game that's just barely above average.

Why not abolish review scores entirely? GWJ abandoned writing "reviews" in the traditional sense, and instead focused on Perspectives. A semantic difference? Sure. But one which is at least symbolic in its assertion that each person brings with them their own unique point of view. The problem with not using scores, however, is that no one actually reads reviews. Many of them read like shoddy instruction manuals, detailing gameplay and mechanics in far too much detail. Others are poorly written, with no flash or style. And if people aren't reading your boring reviews, that means they're probably not picking up your magazine. So editors assign a score to give people a chance to determine the value of the game without all that difficult "reading comprehension" business. Even if the value means something different to the reader than the author.

Reviews just aren't important anymore. The Internet and its gazillion ways to communicate have replaced the need for magazines to tell us what deserves our attention. Your online friends probably bought The Darkness already. Do they like it? Chances are there's already a thread with their impressions. Your fellow forum dwellers may not have the "journalistic integrity" of the gaming press, but you know what their tastes are. Many of these communities dove head first into Shadowrun, setting up games night after night, and many of those guys didn't care what number IGN used to judge the game. They played the beta and the demo, decided it was the game for them, and talked their friends into their purchases.

Reviews didn't help Shadowrun, but they certainly didn't kill it either. Unfortunately for Mitch, Shadowrun had perception problems in the community long before the final product shipped, and an 8.5 review at OXM probably wouldn't have eased the pain much. Ryan judged the game on its merits and gave it a score that fit his opinion. It's too bad the number alone is meaningless, and it's worse still that the number is all the industry cares about.

Comments

There are a couple of problems with game reviewing/analysis and I'm sure someone else has outlined them better than I can.

1) Games are inherently a personal experience and any review is by necessity a highly subjective experience. One person's fun is another's tedium. On the other hand purely objective game reviews would be dry reading, with nothing but the unarguable technical aspects being reviewed (e.g. speed measurements on different rigs, bugginess or crappy customer support).

Ideally these would be combined so that you know if you can play the game and if so, if you want to play the game.

2) Review sites are either small or large.
Small sites can be highly opinonated but tend to give honest accounts of what's out there.
Large sites would tend to be more even handed but, run the risk of getting into bed with the publishers just to ensure that they keep on getting exclusives.

3) Reviews for 'long' games (i.e. games with start, middle & end game sections) never go all the way through. Just like game developers I would guess that the vast majority of the work goes into the start game section. That's all most players will play (via demo's) and that's all most reviewers will ever look at (given their writing deadlines)

The NY times weighs in with
this

--edit: my typing~foo is weak.--

I do actually look up game review scores on IGN or Gamespot when a game that's on sale that I've never heard of catches my eye. The number itself is actually a good indicator of how much fun I'll be having with a game, that is after factoring in things like my interest in the genre. 7s and above are usually acceptable scores to me, with anything lower being too flawed to enjoy. Although I like the "5 star" system better, where 3/5 is avg, 4-5/5 is really good, and 1/2-5 is awful.

My non-purchase of Shadowrun would not have been avoided had it scored an 8.5.

I'd rather have something that's heavy on technical details than something heavy with flash and style. Because most games journalists think that flash and style is an excuse to be as florid, introspective and pretentious as possible.

Demiurge wrote:

Reviews just aren't important anymore.

Right, but were they ever important? I still think that Marketing and Word of Mouth have a far, far bigger impact on game sales than reviews do. In that sense, I agree that the difference between 7.0 and 8.5 is marginal.

That being said, I definitely also agree that there is room for "value" in a review score. A review is a recommendation of how you should spend your money. Mitch is arguing from the perspective of a developer and not from the perspective of a consumer, and his argument falls flat to me.

Fedaykin98 wrote:

My non-purchase of Shadowrun would not have been avoided had it scored an 8.5.

Ditto this here. I think the Vista only limitation(I would have been all over it like white on rice, hitting it like the Fist of an Angry God(tm) if not for this) was the bigger stopping point. I get the feeling that a lot of gamers are staying away from Vista, and that's what's really killing sales.

Dysplastic wrote:
Demiurge wrote:

Reviews just aren't important anymore.

Right, but were they ever important? I still think that Marketing and Word of Mouth have a far, far bigger impact on game sales than reviews do. In that sense, I agree that the difference between 7.0 and 8.5 is marginal.

That being said, I definitely also agree that there is room for "value" in a review score. A review is a recommendation of how you should spend your money. Mitch is arguing from the perspective of a developer and not from the perspective of a consumer, and his argument falls flat to me.

Well said. I can't honestly think of a single game I've ever purchased based only on the score. I've bought several based on reviews, only to be disappointed, but never once did the score matter. Supreme Commander, for instance, is probably my favorite to pick on here. It was reviewed glowingly by several magazines and news outlets. Praise was steeped upon it. If it had been rated 10% out of nothing more than spite against GFW, I still would have bought it. I still would have subsequently sold it on the cheap to someone that I hope is enjoying it far more than I did.(You are enjoying it, right?) My reasoning for that is because regardless of what someone else rated it, it was only a 10% on my scale.

So, I suppose my thoughts, when you aggregate them all is that reviews do still matter. Listening to the Podcasts, reading the perspectives of Certis, Elysium and others, without some arbitrary number attached, lends far, far more weight to whether or not I'll buy a game than anything else out there. Establishing a baseline of respectable, reliable opinions(because that's what a review is after all, the opinion of the review presented in a format hopefully relevant to the reader) is of the utmost important in reviewing anything, not just games.

Clear as mud, eh?

Well, reviews have been important to me in the past...in my horrid former life, pre-GWJ. As Demi says in this article, I only want to play the best games. I don't have time to finish every game I buy, so I'm extremely selective, like a condemned man with his last meal: if your time is limited, pick the absolute best.

That's why there are few series I can be counted upon to pick up. Zelda and Metroid are both among them, although Mario got dropped from the list when the reviews and word-of-mouth on Sunshine weren't that good. Halo is a sure thing. Grand Theft Auto 3 was a religious experience, but I never finished Vice City, so I skipped San Andreas altogether.

On the other hand, I've given chances to other games based on novelty, good reviews, and word of mouth: Guitar Hero is now a huge, monolithic presence in the gaming world, but at the time it came out I think it was unclear whether it would be able to succeed at its price point. Reviews and word of mouth both played a role in its success. Katamari Damacy would be another example of a new experience that received both critical praise and strong word of mouth.

I actually think reviews are still very relevant, but as always, depending on the source. More and more the people who review games are like me - after all, they love their Zeldas and Metroids, but they also loved Guitar Hero and Katamari Damacy.

Fed,

If word of mouth is important, then why haven't you bought The Game That SHall Not Be Named?

Here's my new idea for a review score system: the final score is how much you think the game should cost when it first launches.

WoW: $200

Shadowrun: $40

Daikatana: $5

Now any quibbles about pricing are built into the score! It's genius!

1Dgaf wrote:

Fed,

If word of mouth is important, then why haven't you bought The Game That SHall Not Be Named?

If you're talking about...Shadyjog...it's because word of mouth isn't absolute; I haven't had time for the one 360 multiplayer shooter that I already own and know is great, Cogs of Combat.

This is mostly due to me spending a lot of time playing...uh...Cosmos of Conflict-Making.

Quintin_Stone wrote:

Here's my new idea for a review score system: the final score is how much you think the game should cost when it first launches.

WoW: $200

Shadowrun: $40

Daikatana: $5

Now any quibbles about pricing are built into the score! It's genius!

This is truly a brilliant idea. The only problem is that the value of a game may depend on the economic status of the reviewer. For example, Gameguru may place a value of $10,000 on a game that some poor shmuck with no real income can only afford to place a value of $10, even though it is an excellent game.

And I would suggest that $5 is a bit steep for Daikatana.

Quintin_Stone wrote:

Here's my new idea for a review score system: the final score is how much you think the game should cost when it first launches.

WoW: $200

Shadowrun: $40

Daikatana: $5

Now any quibbles about pricing are built into the score! It's genius!

I'm trying to figure out how to riducule this idea, but honestly, it's freaking awesome. I quit. You can have my brain now.

The idea is that when the game drops to that price, it's worth getting? Could be good. Might be tricky over time, as older games (unless classics), may not have the shine and innovation of new ones and thus even the discounted price may be too much.

Fedaykin98 wrote:
1Dgaf wrote:

Fed,

If word of mouth is important, then why haven't you bought The Game That SHall Not Be Named?

If you're talking about...Shadyjog...it's because word of mouth isn't absolute; I haven't had time for the one 360 multiplayer shooter that I already own and know is great, Cogs of Combat.

Ditto here. The Game-that-shall-not-be-named was a fun rental, and I could get more play out of it, but not $60 worth. Especially on a shoestring "fun budget". The Word of Mouth on You-know-what is far from unanimous.

But we all hashed this out in two threads that are multi-page in length now. I'm surprised to see a front page article that seems to oversimplify the issue here by doing just what we all need to stop doing: focusing too much on the score. Though in fairness, it's about whether the score should be so important, which I think most of us agree that it shouldn't.

Really, as far as scoring goes, I still favor the simplest type of all: thumbs up or thumbs down, with a review explaining why.

Really, as far as scoring goes, I still favor the simplest type of all: thumbs up or thumbs down, with a review explaining why.

That makes all kind of sense. You know if the reviewer was positive or negative on the game, and have to read more to find out why. Might be a good system to adopt at GWJ.

Quintin_Stone wrote:

Here's my new idea for a review score system: the final score is how much you think the game should cost when it first launches.

WoW: $200

Shadowrun: $40

Daikatana: $5

Now any quibbles about pricing are built into the score! It's genius!

I'm trying to figure out how to riducule this idea, but honestly, it's freaking awesome. I quit. You can have my brain now.[/quote]

I agree, that's perfect.

Copingsaw wrote:

And I would suggest that $5 is a bit steep for Daikatana.

I paid $2 for Daikatana at a garage sale and was still ripped off.

souldaddy wrote:
Really, as far as scoring goes, I still favor the simplest type of all: thumbs up or thumbs down, with a review explaining why.

That makes all kind of sense. You know if the reviewer was positive or negative on the game, and have to read more to find out why. Might be a good system to adopt at GWJ.

Do you think GWJ needs abstract ratings in its reviews? I'm intrigued.

What Rich Gitelman also talked about a lot during that podcast was the innovation in his game. He was almost pleading with the hosts to assign some extra value to the concept of cross-platform play.

The poor reviews for this game are, unfortunately, tainted by the poor execution of Live play on the PC. These reviews, in this instance, are as much a review of Microsoft as they are of Mitch and his team.

Anyone else who gets paid to teach knows this problem well. Giving a student a "C" is like a death sentence; even at major institutions that care little at all about teaching, administrators will come asking questions if you have a "D" student of two. Back when I had a boss, I was even told, "if they turn in something, they get at least 50%". Same motivators at work; reviewers/teachers that give high scores attract more advertisers(what else is a for-review game?)/students, which in both cases eventually = $$$. AMERICA! F&%K YEAH!

AnimeJ wrote:
Fedaykin98 wrote:

My non-purchase of Shadowrun would not have been avoided had it scored an 8.5.

Ditto this here. I think the Vista only limitation...that's what's really killing sales.

Y yo. Even if I HAD wanted to pay $60 for a Quake-3-with-sprinkles that pisses all over its source material (FASA does need the money, I guess), I've no plans to roll Vista. Maybe Mitch should have thought that over before accepting the filthy Vista-exclusive-lucre.

souldaddy wrote:
Really, as far as scoring goes, I still favor the simplest type of all: thumbs up or thumbs down, with a review explaining why.

Amen; Siskel & Ebert style. Two folks giving independent opinions and thumb-ratings.

I still use Gamespot and IGN before buying games, though primarily for the reader-reviews. While 90% crap, if you see the same word appearing in several reviews (eg. "grind-fest", "crap controls", "A-hole community") you can assume some validity.

Game magazines can still be very relevant imo, the problem might be that many magazines are trying to copy the fast reading style of the internet reviews. And if they do, why not just read the free internet reviews instead. Personally, I'm stilling buying a magazine each month which I like very much, simply because they manage to make more interesting reviews than most, especially compared to internet reviews. For gaming news etc. magazines can't keep up, and shouldn't really try either.

That said, the magazines (and internet sites, but I doubt most of them will improve) surely need to evolve out of the "hey, lets rate games, numbers says everything!" its been in since the beginning, getting closer to a journalistic approach to gaming.
If this happens there should be plenty of room for reviews, also in gaming magazines. Coexisting with "perspectives" from GWJ etc. which is really interesting too. But I wouldn't really like to lose any of them.

And dammit, learn to use the full scale of your grading system, if you are using such a system. Probably never going to happen for most internet reviewers either.

The thing is, I generally end up likeing games that get 6-8 as much as I like games that get 9s.

I just read the GWJ review threads, really. Then I complain about price points.

I guess I was under that impression that it was common knowledge that rating systems never mean anything. Sure, I generally read several reviews before I buy, but I actually read the review, rather than look at the number and conclude that because someone typed a 3 in a form field that it must suck.

The premiere example that comes to mind is Rampage: Total Destruction reviews at 1up. The Gamecube version was given an editor score of 7. The Wii version was given a 1.5, the lowest score I've ever seen on the site. From the review, the only reason for the difference was the fact that it was $10 more than the Gamecube version is, while being pretty much the same game with a new(phoned in) control scheme and some extras. While its a valid point, using 1up's formula one can only surmise that the 7 - 1.5 = $10.

I don't know if its so much that numbers lie, it's that they aren't given any and/or aren't used with any real meaning behind them. I think the only thing that the numbers really do for us is give an idea of the general mood of the reviewer at the time. Maybe its more of a social thing... Perhaps it would be better to review using the Smiley system.

Y yo. Even if I HAD wanted to pay $60 for a Quake-3-with-sprinkles that pisses all over its source material (FASA does need the money, I guess), I've no plans to roll Vista. Maybe Mitch should have thought that over before accepting the filthy Vista-exclusive-lucre.

You do realize they really didnt have any choice or say in the matter?

Are you the Shadout Mapes, or some other Shadout?

Farscry wrote:

Really, as far as scoring goes, I still favor the simplest type of all: thumbs up or thumbs down, with a review explaining why.

When it comes to media, I've always preferred the stoplight over the thumbs because it allows for a degree of exposure instead of buy or don't buy. Something like this:

Green = Buy
Yellow = Try Before Buy (Rent, Borrow, or even Download)
Red = Avoid

I may be misinterpreting your use of the thumbs, though. Do they represent a purchase recommendation or a play recommendation? If it's play, the stoplight doesn't work as well.

Copingsaw wrote:

This is truly a brilliant idea. The only problem is that the value of a game may depend on the economic status of the reviewer. For example, Gameguru may place a value of $10,000 on a game that some poor shmuck with no real income can only afford to place a value of $10, even though it is an excellent game.

And I would suggest that $5 is a bit steep for Daikatana.

I mean, price scores wouldn't exist in a vacuum. It'd all be relative to the existing market, you know?

And I've never played Daikatana, so I'm afraid I can't comment on that.

rabbit wrote:

I'm trying to figure out how to riducule this idea, but honestly, it's freaking awesome. I quit. You can have my brain now.

It's all slimy and gross!

1Dgaf wrote:

The idea is that when the game drops to that price, it's worth getting? Could be good. Might be tricky over time, as older games (unless classics), may not have the shine and innovation of new ones and thus even the discounted price may be too much.

Yeah, more or less. I mean, it's just an idea I was throwing out there and probably suffers from having no shelf life, when you consider changing markets, inflation, killer bees, etc.

I do think numbers in ratings are too slanted to the high end. I think a 5 should be saying its playable but boring. Currently its very rare to see 1s done in a rating, and I can think of a few games that really deserve it.

With that being said, I think Shadowrun was a poorly conceived game.

To be honest I think the demo hurt Shadowrun more than anything.
I downloaded it, played it once, and then deleted it realizing I had already played all of these gameplay elements on other games.

This game wasn't Shadowrun. This was a middle of the road FPS with no single player, which bastardized the cyberpunk feel of the whole storyline and turned it into some generic orwellian opressive society which didn't feel dirty enough. What I want out of Shadowrun is a world like the world in Blade Runner.

So the 7 was fine imho, 60 bucks for that is way too much.

My personal scoring system for games, or any entertainment is simple: I call it p2p, or pennies to pleasure.

I take the cost and divide it by the number of hours I will enjoy it. NOTE: I add in as much of time I can. So, a 60 dollar game played for thirty hours is not just a 2 (60/30=2) because there is also the time I enjoy others playing it on my 360, or the time I spend with friends having a drink and cherishing the moments.

In general, a 60/30 game time is really a 60/45 at least, because I then talk to people about it, or have people check it out on my system. Especially if it has a great ending, this can bolster the attention given to the game. and in general for video games, this works well because many things, like graphics just don't work well for bases for judging the game. It also gives me a good way to project my entertainment budget, and discourages impulse buys until they are usually discounted or available for rent or used purchase.

The reason is simple, I have multiple platforms, and there is no way i can judge DS games on the same scale called "graphics" as anything on the 360. What I can say is Puzzle Quest has a higher base p2p rating than anything I have played on the 360.

For me, Shadowrun has a .8 rating i.e. so far the price of my purchse of the game has cost me 80 cents an hour to play, with 2.5 being the bare-bones "barely acceptable" rating (2.5 is basically the cut-off. That is like a 60 dollar game you only play for 40 hours). I still play Shadowrun, though, so that number can ony go down. For me, as my 40 dollar purchase price, it is well worth it already.

To be honest, I wouldn't want a storyline for this game; Shadowrun360 simply would suck with that kind of ramp up the gear along the conventions of a stroyline. There is not enough in it. I NEED a level designer ala FarCry and character models though. THAT would rock so much more.

My wishlist for shadowrun, or some other game that plays arena style:
a. character customization of stock race types. (too easy to see who's what)
b. level designer (worth the price jump from 40 to sixty, if done right)
c. third person view/ auto-aim (ala thumb through targets) in smartlink.
d. minion like control of summoned creatures.
e. gradated levels of Tree of Life/ Strangle/ and other area effect spells.
f. AI mercs for hire, put into offensive or defensive mode. (all cyberpunk genre is for hire!)
g. clan support with logos
h. more tech upgrades, and some enchant magic.