Developers Reviewing Reviews

Certainly game critics aren't perfect. In fact, some might say that gaming journalism is a horrifically twisted and broken abberation where a 70% is considered awful and anything lower is done out of emotion or for comedic effect. The infamous pulled review (was it on 1Up?) of Neverwinter Nights 2 comes to mind, where the reviewer found fault with the very concept of the game rather than its execution.

Mitch Gitelman recently gave a very passionate interview with Major Nelson, in which he berated game criticism as a whole for the luke-warm reception Shadowrun's been getting. He claims, and not without merit, that it is unfair that Shadowrun loses points because of its price, relative to its limitations as a multiplayer-only game, and a somewhat small one at that. He says his team deserves better.

There is no doubt in my mind that he and his team did the best they could, and should be commended for their work, but should that sentiment skew a review score, even if it doesn't show through in the craftsmanship of the game itself? Is it fair to let price come into it? Monster Madness was panned for its high price, yet Valve started selling Counterstrike, previously a free download, the game's reviews remained sky-high despite the outrage and disgust. And perhaps Counterstrike is the most apt example, one that Mitch refers to himself. It is another multiplayer only first person shooter, and while Shadowrun does seek to improve upon formula, it is clear to see the heritage of Counterstrike shining through. Yet even though Counterstrike is milked for all its worth with new versions in new boxes and with prettier graphics and spin-offs, the core game is still regarded very highly and does not suffer for its price.

Ultimately, it's the job of the game makers to make games and the game reviewers to review. Artistically, an artist defending their work to a critic is a sign that the work is not complete, as it cannot stand on its own.

So is it fair to say that Shadowrun should be reviewed with ignorance to the price? Is it even fair to say that the price is really what has caused Shadowrun to receive less than glowing reviews? Does the fact that "true" FASA fans feel as burned by Shadowrun as Battletech fans felt about MechAssault play into it?

If you're reviewing a car, is it fair to eliminate price (particularly price as it compares to similar cars) from the review? How about a house? A TV? An MP3 player? A cable service provider?

Why should games somehow be so special that price isn't a factor?

Reviewing a product means that you take all the pros and cons, weigh them up against each other and (in some cases) provide a score on a relative scale to which the product can be compared against similar products.

Ignoring a feature or aspect of a product means that the review is flawed (beyond obvious subjectivity issues). Anyone who feels that they got burned in a review of their product for not ignoring features shouldn't allow the press to review it in the first place - it gets done with films.

I think complaining about review is kind of sad. If you have that much faith in your product, release a good demo and market it well. If you accomplish those two things well, the review scores won't matter at all because the public will be able to decide for themselves. Think of movie viewers: how many people decide whether to go see a movie based on
A) The trailer
B) Word of mouth
C) Reviews

I'm sure A and B far, far outweigh C, and I'm sure the same is true for the videogame industry.

Regardless, the notion that reviewers shouldn't consider price in their review is ridiculous. When purchasing a videogame, you are making a decision about how you want to use your time/money to be entertained. A review should weigh these factors. Replayability is an ok measure to review a game on, and it essentially equates with value, so price should too.

The comparison with CS isn't entirely fair either, because CS is only 20 bucks.

Okay, so they slapped a price tag onto Counter-Strike. But did they over-price it?

Dysplastic wrote:

The comparison with CS isn't entirely fair either, because CS is only 20 bucks.

It would seem the answer is "No".

Was Max Payne not worth the MRSP just because it was short?

Price isn't a factor, it's a variable. Yes, the publisher picks the initial price point, but they don't control it long after that. If a game's score is deflated because it's too short or too pricey, but is now in the bargain bin for $15 instead of $50, I'm far more likely to give it a try than if it's a 7/10 because it's got crap controls.

I'd say leave things like price and length out of the actual score. Say how much play and replay I'll get out of a title, but leave the actual value judgements up to the consumer.

Kepheus wrote:

Price isn't a factor, it's a variable.

Well, if you want to get technical, the game's quality is a variable too, because patches and (sometimes free) updates make a difference. And then DLC that adds to the game also adds to the price.

Should they not factor in technical problems to a review, since patches will quite likely address those concerns down the road? How about something like the number of maps. That will quite likely change in the future, via content additions that may or may not be free.

Where do you draw the line?

A review is of a product as of the time of the review. That includes all current factors, because everything is variable. Products go through revisions throughout their lifespans, and you can't control that.

Given that, then yes, I think every factor regarding a product should be counted into a review of said product at that time, including price.

[edited final sentence to be more specific]

I don't think the review scores for Shadowrun are off, but I don't fault the creator for standing up for what he believes. If an uninvolved, 3rd party came in and crapped all over the project you worked on for a long time, wouldn't you hope your boss would stand up for you? This can be as much for the 2nd tier art guy as it is for Gitelman or Jaffe or Dyack. Maintaining morale when the public scrutinizes your labors is very important.

I've had plenty of managers who wouldn't stand up if their chair was on fire, so I can appreciate what Gitelman is doing. It seems petulant to the consumer, but we don't know what's going on in meetings and conference calls to the developers who work for him. They might have had a "Come to Jesus Meeting" or gotten the Sigil treatment on the down low.

Artistically, an artist defending their work to a critic is a sign that the work is not complete, as it cannot stand on its own.

I have to disagree with you there. There's been plenty of critics in book, film and art who don't "get" the piece they are reviewing. I'm not saying it's the case here, but just because the artist defends their work is not an admission of incompleteness. The creation of art is a passionate exercise and I would expect any artist to defend their work passionately. Those that don't, appear to be the ones who have written their work off.

All I know is that 70 bucks for 9 maps and no single player is too much.

For that reason alone I won't consider picking up Shadowrun, as fine a game as apparently it is.

Hemidal wrote:

I don't think the review scores for Shadowrun are off, but I don't fault the creator for standing up for what he believes.

I don't have a problem with that. But Gitelman crossed the line into essentially telling customers "hey, if you don't like the way things are going, then $#@& off", which I do have a problem with.

For a productive example, look at how Bill Roper's been defending Hellgate: London's two-tiered multiplayer system. While their choice of "plebe" as the term to describe non-subscribers is questionable, he's otherwise been constructive in his attempts to sell people on it, rather than basically saying "hey, the market is going this way, and you need to just accept it and deal with it, bucko".

"hey, if you don't like the way things are going, then $#@& off"

While it's a tad terse, I enjoy that kind of honesty in developers. Fans, publishers or reviewers, I don't particularly think any of them are hands-off when it comes to giving back as good as you get it. Granted, the above mentioned are the lifeblood of your company but I'd rather some personality shone through when speaking, as opposed to this Rove inspired "don't stray from the talking points" kick that seems to be the norm these days.

As a gamer, I'll say this. Pretty much every idea I have for what would make a game good is either so horrendously awful it should be avoided at all costs or so terribly difficult to implement that even attempting such would cause the universe to collapse. If I chose to open my mouth and run it anywhere near a dev he has every right to tell me to "go f*ck a duck".

I appreciate developer honesty, but there's also this thing called "tact" that I'm rather fond of as well, even though I'm not always the best at using it either. I'd wager I usually use it more effectively than guys like Gitelman or Terry Goodkind, though.

Tact is key, and I do think Gitelman went a bit too far as he came of as a bit of an emo. I'd prefer the "Levine Don't Fear Vivendi" brand myself. J.E. Sawyer is another decent example of what I'm talking about. At this point, it just seems like all the comments about every game coming out are totally homogeneous, meaning they could probably get away with hiring just a single person to do all the pr for the entire industry.

I miss the humour and wit. Plus, who doesn't like seeing some rabid fan get put in their place by a dev?

Farscry wrote:

I appreciate developer honesty, but there's also this thing called "tact" that I'm rather fond of as well, even though I'm not always the best at using it either. I'd wager I usually use it more effectively than guys like Gitelman or Terry Goodkind, though. ;)

I honestly think that Gitelman has absolutely nothing to lose and he's letting it fly. Unlike Jaffe, who gets a blank check thanks to God of War or Dyack, who's "eccentric," Gitelman feels he's getting screwed by Microsoft for the price point and the lack of marketing and the reviewers who only look at something that's out of his control. So, he unloads. It's human nature. Is it professional? No, it isn't. Will it hurt sales? Probably not, as Shadowrun has probably seen all of the sales it's going to get until it hits $17 at a Circuit City clearance. Is his career at FASA done? Maybe, if the rumors of the studio closing are true.

I'd shoot my mouth off too if I had nothing to lose. You get to a point where you don't have goodwill towards your fellow man anymore. You become the boiling kettle, and your shrill whistle screams. It's just sad that his tirades after Shadowrun are the legacy he's going to leave behind.

I still don't see what's to stop a gamer from asking, "Am I willing to pay $60 for an 8/10?"

My answer: Buy used!

Farscry wrote:

I appreciate developer honesty, but there's also this thing called "tact" that I'm rather fond of as well, even though I'm not always the best at using it either. I'd wager I usually use it more effectively than guys like Gitelman or Terry Goodkind, though. ;)

First you want designers to be writers, and now you want them to be diplomats. Just because they think they're gods, and just because we worship them as gods, why should we expect them to be anything other than former code monkeys with dreams and big mouths?

Oh wait, maybe because of that thing I said? Yeah, time for better management and delegation of authority/power within these projects.

Well, we haven't seen a good flail out of the industry in a blaze of... well, not glory really, but it was damn shiny whatever the hell it was... since Doc Smart. Here's to hoping it just keeps getting better and better.

Farscry wrote:

If you're reviewing a car, is it fair to eliminate price (particularly price as it compares to similar cars) from the review? How about a house? A TV? An MP3 player? A cable service provider?

Wow, that's the first thing that I thought of.

How childish is this? Is he in kindergarten, where you get a star for "effort"?

I've spent months working on projects that got "Decent" qualifications, and I did not whine "But I worked SO HARD on this!"

Farscry wrote:

If you're reviewing a car, is it fair to eliminate price (particularly price as it compares to similar cars) from the review? How about a house? A TV? An MP3 player? A cable service provider?

Why should games somehow be so special that price isn't a factor?

My thoughts exactly. Why would a consumer eliminate price from the equation? That makes no sense. In the end, it's completely up to the customers to decide what is worth their money.

So uh, people don't want to buy Shadowrun because it's a $60 game with no single player mode and only 9 maps? Well, sh*t... I can't imagine why.

LobsterMobster wrote:

There is no doubt in my mind that he and his team did the best they could, and should be commended for their work, but should that sentiment skew a review score, even if it doesn't show through in the craftsmanship of the game itself?

Not a chance in hell. Look, I respect the effort software takes, but if the end result is shoddy, I honestly don't give a rat's ass if the devs worked 19 hours a day down at t'mill and were thrashed to death nightly with a broken bottle. Their effort should be commended by their employers; the consumer only need care (and, I think, rightly so) about what can be picked up at the mall.

Mystic Violet wrote:
Farscry wrote:

If you're reviewing a car, is it fair to eliminate price (particularly price as it compares to similar cars) from the review? How about a house? A TV? An MP3 player? A cable service provider?

Why should games somehow be so special that price isn't a factor?

My thoughts exactly. Why would a consumer eliminate price from the equation? That makes no sense. In the end, it's completely up to the customers to decide what is worth their money.

So uh, people don't want to buy Shadowrun because it's a $60 game with no single player mode and only 9 maps? Well, sh*t... I can't imagine why.

A good car review of a Corolla that was priced at $35k would probably be similar to the ones you see. It would talk about how great the Corolla is for practical reasons, but also mention that the price doesn't seem to jive with the design.

Then again, car reviews that use number summaries tend to also rank cars against each other by category. Perhaps the spokesfolk should be arguing that the media is misclassifying the title?

wordsmythe wrote:
Farscry wrote:

I appreciate developer honesty, but there's also this thing called "tact" that I'm rather fond of as well, even though I'm not always the best at using it either. I'd wager I usually use it more effectively than guys like Gitelman or Terry Goodkind, though. ;)

First you want designers to be writers, and now you want them to be diplomats. Just because they think they're gods, and just because we worship them as gods, why should we expect them to be anything other than former code monkeys with dreams and big mouths?

Oh wait, maybe because of that thing I said? Yeah, time for better management and delegation of authority/power within these projects.

So you're saying that some people don't have to be held to the same standards of tact in public speaking that the average person would be held to? Saybuhwha?

I never said I want designers to be writers or diplomats. I said I expect basic standards of respect and decency. Granted, in this nation, that would beget Paris Hilton-esque behavior if we're going by the law of averages in generating "basic standards", but that's not what I mean.

I was thinking about this the other day. With development budgets and cycles the way they are for the current gen systems (360, PS3) it seems like it would be very difficult to start to start a new IP with a new engine. What do you guys think about the strategy of doing a cut down version of a game, like Shadowrun appears to be, charge a lower price (say $40) and then in 12 months release an "sequel" that has the original vison for the game for full price.

In this case (Shadowrun), what if in 12 months they release the same game with minor graphical updates but included more content (weapons, levels, game types, etc.)? Would that be a good thing or a bad thing? Is that a workable model for game development of new IPs? What would the reviews for my "Shadowrun 2" be like?

JimmDogg, that sounds a lot like what I proposed in the Spore thread.
Gimme the early stages of the game, like the first 1/3 or 1/2, for $25, prove the viability, and continue building the rest of the game with a re-release/episodic continuation/GoldPlus version when the rest is done.

That's about the only way I'm ever going to pay more than $50 for any game - $25 for the early stage release and $40+ for the full/rest of the game, or $25+$25 if it's done as an episodic release.

How did Valve do with their Episodic stuff? And What about SiN episodes? Anyone here got anything to say about the Bone Adventures? Is the world ready for episodic? No, not the hardcore, the guys that are browsing XBLA and Steam on the weekend looking for a new, inexpensive timekiller. $10 for a gamelet that takes 5-10 hours to complete sounds like a perfect weekender to me. And what about Valve's model of breaking out EVERYTHING from a game, like HL2, where you can buy the "normal" old-style package - SP+MP Modes, or you can buy just the MP Mode CounterStrike or all the MP Modes or Just the SP? Do they release numbers?

Sin episodes didn't fare too well... though i don't know whether there was a large drain on income due to pirating (ironically)...

Part of the problem with the Shadowrun is that reviewers are comparing it to the game they wanted it to be.

It is expensive for the amount of maps and gametypes it offers. No disputing that. Certainly, I didn't buy it at full price and couldn't recommend it at full price. I bought it for 40% off. For me, it was worth it.

In the words of a foul mouthed games journalist, reviewers should be the condom, not the pimp. A writer shouldn't let feelings about the dev team interfere with the assessment of a game in any way. He's there to look after the consumer's interests, and price definitely factors into that. If a game doesn't offer much content for your money, it's entirely fair to give it a lower score.

Of course, what constitutes enough content is subjective, but that's where all those pesky words should come in.

Excellent topic, Lobster.

Obviously, the whole game review/score system is flawed in a myriad of ways, but it's fairly well institutionalized at this point, especially with the advent of aggregate sites like Metacritic/Gamerankings. I think Gitelman and the rest of the industry need to come to terms with the fact that the game review machine serves cost-conscious consumers. Games that appear overpriced (comparatively) are going to lose points even if they're otherwise great games. And for good reason--the industry's insistence on a single price point for all games is absolutely ridiculous from a consumer point of view.

As I see it, developers/publishers have two options: create the games they want and price them appropriately, or create the types of games that fit the $50/$60 tag.

The Fly wrote:

Excellent topic, Lobster.

Thanks.

duckilama wrote:

How did Valve do with their Episodic stuff?

Half-Life 2: 11/16/04
Half-Life 2: Episode 1: 6/01/06
Half-Life 2: Episode 2: ?

Guild Wars was doing pretty well with the episodic stuff for a while, but then gave up on it.

JimmDogg wrote:

I was thinking about this the other day. With development budgets and cycles the way they are for the current gen systems (360, PS3) it seems like it would be very difficult to start to start a new IP with a new engine. What do you guys think about the strategy of doing a cut down version of a game, like Shadowrun appears to be, charge a lower price (say $40) and then in 12 months release an "sequel" that has the original vison for the game for full price.

Mount and Blade is doing something similar. I think that is a great way to keep developing while getting money to keep doing it.

But I wonder, if a game get a lower grade because of the price, shouldn't you also give a game a higher price if it has a low price? Or should price be a final conclusion item where the author gives his take on the price/value of the game.

Koning_Floris wrote:

should price be a final conclusion item where the author gives his take on the price/value of the game.

This is, I think, the best way to handle it.

As for scores, I still think that numerical scores are unnecessary and a copout, but that particular topic's been debated ad nauseum. I think that a review's "score" should be a section that lists the pros and cons and leaves the consumer with a solid idea of whether they think the game would suit them or not. If the pros are solid for you, and the cons aren't anything that would seriously detract from the game for you, then hey: the game is a thumbs-up for you, isn't it? Price is one of those factors, as for most people, money is a finite resource, and depending on the price/content ratio, a value is derived for that product (which is what we've been ddebating in this thread).

So yeah; I like Koning's perspective here.