Part Of The Conversation

Something was bothering me yesterday as I trolled Metacritic for PSP game scores, trying to refresh my memory on what's actually worth buying for the system. I have the infinite wisdom of the GWJ community to draw from, but even our collective memory may fail under the strain of nearly two years worth of games. With this is mind, the oft derided aggregate sites we disdain for reducing the quality of a game to a mere number serves a valuable function. If I want to know if Metal Gear Solid Portable Ops is good, I can punch it in and see that most reviewers seem to like it. No spoilers, no outright bias or flaming, just a raw number denoting quality so I can feel safe making the order.

What bothered me was that these scores were missing something. Our input. Not one aggregate site includes the Gamers With Jobs perspective on games because we don't score them. I'm starting to wonder if we should.

We dislike the common review structure so much we call our articles "Perspectives" so that we can do our own thing. While it suits our own purposes, I wonder if we're doing a disservice to the wider audience who may benefit from our perspective but will never be exposed to it because we're unwilling to play ball.

Putting aside the old debate of assigning a number to an experience, there are some broader issues at play that concern me as someone running an independent gaming site. For starters, a number would mean we appear on aggregate sites next to Gamespot, IGN, 1Up and the dozens of others sites that score their games. People who look up a title might see "Gamers With Jobs" and read something they would never have been exposed to otherwise. While we primarily write for out own amusement and our community, I like to think that the internet at large could use more of what we're cooking here.

The second thing that springs to mind is access to games before release. Like it or not, calling your articles "Perspectives" and refusing to assign a score is a total turn-off for a lot of PR people when they're deciding who should get early review code. If you're not going to contribute to the Metacritic pool -- which can determine whether or not they get paid a bonus -- they may be less inclined to offer you coverage opportunities. There are hundreds of gaming sites out there and early code (or even final boxed software) is often limited. It's a tough gig wading through the countless fan sites vying for your attention. On the surface, we're a pretty enigmatic site to understand already.

We also have our writers to think of. As a creative cat herder, I'm more likely to get a review from writers if I can spare them the expense of buying the game themselves. We're blessed with all the great content we get from such talented people, but as paid opportunities come up and personal lives get busy or financially shaky, every little bit can help.

Is getting a Gamers With Jobs review on the podcast or the front page before the release date of a game something readers want, even if it means making our review content a little more in line with what most sites are doing? I don't think assigning a point value to a game suddenly means the content of the articles has to change, but the fear of people skipping to the end of a review is a real one. No one wants their hard work reduced to a final paragraph, a number and a few screen shots.

With all this in mind, I open the floor to you, dear reader. Would we be selling our souls by scoring games? Is there some rating system that both satisfies the intellectual and the more mainstream reader? Maybe we need our own internal aggregate with a final score that reflects the views of many on this site. Help me out!

Comments

I've been reading GWJ for years, though I've only just started to participate more in the community recently, I trust the guys who run and write for the site. I don't think that adding a number to their reviews is going to change the way they approach them. I don't think better relationships with publishers and PR companies is going to lessen the standards they set for them. If those things open doors to provide better coverage and attract a larger audience, I say more power to them.

The one thing I do worry about is what might happen to the community with additional traffic. But as someone who has come to that community late, it is difficult to deny others the same opportunity.

Mystic Violet wrote:

I guess a 1-5 scale is pretty simple and straightforward.

I like the five point scale as well but it does kind of break in Metacritic and Gamerankings since they just do the division and convert a three point score (which most people tend to think of as 70% since that's the way the A through F grading system works here in the states) into 60%. I hate to say it but if GWJ wants to be relevant in the aggregate score then it'll need to adopt the dreaded "7 to 9" scale.

Certis wrote:

If we went with scoring, community involvement would be a must.

Gamerankings won't go for this at all. They want a score that isn't a moving target. The best you can do is a reviewer's score and a community score similar to what Gamespot does. I'm not sure about Metacritic but I'd bet they'd have the same problem.

I love the idea of community ranking across a slew of categories. The more info the better, imho. Even better if you can show a separate graph for 'I generally like this genre' vs. 'I don't generally like this genre.'

No scores please - I love the place as it is, don't change a thing.

long version:
Personally, I prefer reading your personal perspective on a game and deciding for myself whether I would enjoy it or not. I also prefer reading a number of reviews on a number of sites and forming my own opinion... I could care less about scores.

Thanks!

Rat Boy wrote:

Screw other people. They want to find out what Goodjers think about games, then we ain't gonna make it easy for them.

I propose a Dadaist or surrealist system for ratings. With ratings like: A spanish toilet, 3 mangos, or a hundred cockroaches.

YOU figure it out, plebe.

I don't like the idea of scores. The only site that I read the reviews on other than GWJ is Ars Technica, because real effort is put into explaining why they like or dislike the game. Too many other outlets simply rehash old arguments, dip into frothing at the mouth fanboism, or think that the score is all they have to say about a game.

Scores are a crutch for people that cannot write or cannot take the time to really express themselves in writing.

That's what is great about GWJ: real thought is put into each opinion, and there is actualy reasoning employed in the course of the review.

I worry that adding scores to gain favor with Gametab is precisely the wrong way to go about "joining the conversation". I understand that it prevents us from getting early releases, that the PR folks dislike having to think....frankly, once you play that game, you can never get out. You have sold your soul, and your editorial process, to the PR hacks.

We aren't the average gamers. We don't have all that much in common with the bulk of gaming community, we don't really care about bunny-hopping for the next 5 years in Battlefield 24000, or playing Madden 2010. We expect more from a game, both individually, and as a community. We are a niche brand.

Frankly, I doubt very much that PR people really give us a second thought, and there is no reason why they should. We don't really give them snazzy soundbites, and we don't push the latest BS title on our readership and community.

A niche strategy, one where we virally get our message out, is needed, not a compromise of our principles.

mateo wrote:

Scores are a crutch for people that cannot write or cannot take the time to really express themselves in writing.

That's what is great about GWJ: real thought is put into each opinion, and there is actualy reasoning employed in the course of the review.

Are you implying that the addition of a score at the end of a review will somehow destroy logical reasoning on the part of the reviewer at this site?

mateo wrote:

A niche strategy, one where we virally get our message out, is needed, not a compromise of our principles.

Isn't this jumping to conclusions a bit? Adding a review score wouldn't compromise their principles, pandering to the PR people and wearing hats of money would. Why does it seem that by adding scores this site would magically transform into a web commercial? I don't get it, could somebody explain it to me?

Certis wrote:

I'm curious, for those of you concerned about stupid people finding the site, how did YOU discover the site? If you didn't come here when we first opened, there's a very good chance it was a link that brought you to the page. Are you afraid there are no more worthwhile folks left? Were we supposed to lock the gates once you were through? ;)

They might be concerned about the sheer numbers overwhelming your ability to moderate, not that the new arrivals would be lesser gamers. Or that could be me putting words in their mouths.

Trainwreck wrote:
mateo wrote:

Scores are a crutch for people that cannot write or cannot take the time to really express themselves in writing.

That's what is great about GWJ: real thought is put into each opinion, and there is actualy reasoning employed in the course of the review.

Are you implying that the addition of a score at the end of a review will somehow destroy logical reasoning on the part of the reviewer at this site?

mateo wrote:

A niche strategy, one where we virally get our message out, is needed, not a compromise of our principles.

Isn't this jumping to conclusions a bit? Adding a review score wouldn't compromise their principles, pandering to the PR people and wearing hats of money would. Why does it seem that by adding scores this site would magically transform into a web commercial? I don't get it, could somebody explain it to me?

I agree... I don't get it either. To be honest, the reviews on GWJ, while well written, usually come too late to help me make buying decisions... in fact, I rarely ever look to GWJ Editorial for review information (rather, I go to the forum). If you guys want to make a bigger impact in the reviewing world, then do what you have to do. If there is some huge "influx" of "morons" here, I'm sure it will be dealt with (and surely there are enough people here who could step in and help moderate if needed).

Sounds like there is mostly benefits from adding scores to the "reviews". I dont really see how it should be problem anyway. People who care enough will read the entire review whether there is a score or not, people who dont, would simply not have read the review in the first place without a score, but used UP1, Gamespot etc. instead.

Layout wise it should be possible to signify the review itself over the scores too, opposed to many other sites where you get the impression that the scores are the most important, with the review being second.

Beside, when using scores, a few lines explaining the score(s) also helps a lot. And use the whole damn scale whatever that scale might be

While I do read reviews elsewhere (hell, I even buy a gaming magazine every month, old-school!) I really love the personal perspective style at GWJ, which is especially shown in the podcast.

As long as you keep of the good work with perspectives on games, gaming and everything related it hardly matters if there is some numbers at the end of the text. If it will make it possible to make the site even better due to publicity and access to more games/publishers, go for it.

Maybe you could have each of your writers, who ended up playing a game which had been reviewd at the site, assigning scores to it and a few lines of comments, and not only the one writing the review in the first place (even though the original reviewers score would be the one for Metacritic and such). Could signify the personal perspective. Beside those scores, I guess there could be a community score, which many seems to like (I never saw the idea in Community scores at other sites, it always end up in people posting either Max scores or Lowest scores and a flamewar between lovers and haters of the particular game, but maybe it would work better here, I dont know).
Gamespy has used the concept of having different writers comment on the same game (mostly in their previews afaik), and while I guess you dont want to be compared to them too much, it works quite well with those inputs from different writers.

I certainly don't count myself among those who are worried about an influx of IGN forum members. GWJ has braved the dangers of popularity quite well so far, and I don't expect that to change any time soon.

My concern is that reviews that are centered around a rating scale tend to have a different character than those that are not. They both read and write differently, since the rating scale is a very important feature--it makes a difference. Readers pay a great deal of attention to the rating; that, after all, is why we're considering adopting a rating system in the first place. And since readers pay ratings a great deal of attention, so too (contra Duoae) must the reviewers. Good reviewers adapt their writing to their audience, as well as to the format of the publication. So, in asking whether or not GWJ should adopt a rating system, we are in fact asking what style of review we prefer. Whatever your own preference, I hope at least to have made the case that the decision ought not to be utterly carefree.

Trainwreck wrote:
mateo wrote:

Scores are a crutch for people that cannot write or cannot take the time to really express themselves in writing.

That's what is great about GWJ: real thought is put into each opinion, and there is actualy reasoning employed in the course of the review.

Are you implying that the addition of a score at the end of a review will somehow destroy logical reasoning on the part of the reviewer at this site?

mateo wrote:

A niche strategy, one where we virally get our message out, is needed, not a compromise of our principles.

Isn't this jumping to conclusions a bit? Adding a review score wouldn't compromise their principles, pandering to the PR people and wearing hats of money would. Why does it seem that by adding scores this site would magically transform into a web commercial? I don't get it, could somebody explain it to me?

I'm saying that all anyone cares about, seeing a score, is the score, not the reasoning behind it. After a while, writers being human beings, start to think less of explaining the experience, and more about what the score should be. It won't happen immediately, but it does tend to happen over time.

And no, I don't think my second point is that much of a leap. The stated reason for adding a score is to make the reviews here more palatable/understandable to PR flacks, ostensibly to get access to more games, get early views of code, get some nice GWJ soundbites out there, and generally spread the GWJ brand.

It's not such a leap that once you start down that road, the PR types will want to have editorial oversite for reviews of their game, want advanced copies of the review, or even set parameters for the review....getting access to code often comes with strings attached.

Once you start getting in bed with the same people that you are ostensibly reviewing, you tend to lose your autonomy.

Will Certis and Elysium be able to maintain the same autonomy and control of the editorial direction of the site when Id, as an example, is there saying "here is Quake 10 for preview, say something nice or you won't get access from us again?"

I hope so, but the very question puts a bad taste in my mouth. Scores or not, GWJ is here for the community, not to make Game Publishers, Developers, and Marketers life easier. They have IGN and Gamespy for that.

mateo wrote:

And no, I don't think my second point is that much of a leap. The stated reason for adding a score is to make the reviews here more palatable/understandable to PR flacks, ostensibly to get access to more games, get early views of code, get some nice GWJ soundbites out there, and generally spread the GWJ brand.

It's not such a leap that once you start down that road, the PR types will want to have editorial oversite for reviews of their game, want advanced copies of the review, or even set parameters for the review....getting access to code often comes with strings attached.

Once you start getting in bed with the same people that you are ostensibly reviewing, you tend to lose your autonomy.

Ok, now I see where you're coming from and you do have some valid arguments. In a corporate site where reviews are the major draw for it the allure of increased access over ethical standards is very tempting. However I think this site relies much more on the editorials and community rather than "Halo 3 gets 1000000! Come look!" If anything they're looking towards making the review section more of a gateway for new members, not as a core for the site itself.

I'm not going to speculate about the direction Certis and the others want to take this site but up to this point it hasn't been a very commercial site. I don't think they have dreams of retiring from GWJ income, getting tons of ad-driven revenue, or making up a whole new business model. In other words there isn't as much of an incentive to kowtow to the developers. This site hasn't been very pro-establishment and I don't see any indications of it changing.

I'm not saying that your fears aren't warranted but the founders and contributers of this site are for the most part level-headed enough to make grounded decisions that enhance and moderate the community. Would this really mark the beginning of the end? It just seems much ado about nothing for me.

Let's face it, the site has flourished for years without publishers granting early access to games. Certis and Elysium have done a good job moderating the periodic rushes whenever a firebrand issue causes a sharp uptick in this community. Basically adding a scored review to the output of the site: creates more potential work for the moderators, which, to be blunt, is not my problem; adds an opening for new people to join the community; possibly gives the writers a direct benefit; and maybe adds an opening for some conflict of interest reviews, which has never been a problem in the past.

I would love to see review scores being community based, but as AI pointed out, this might pose a problem for the meta- sites. Perhaps the community review score could be done on a dollar basis, with a 1-5 poll from the writer. The aggregate sites would be able to pull a simple 1-5, but people following the link would see that the community feels like 60 dollars is fair, and read why the reviewer went with the number they did.

Pie charts, it is all about pie charts! Screw the numbers...

Add a score for people who want it. I myself would rather read about a person's experience with a game. What they liked and disliked over a number. So I'll do what I do with other sites, and mags and ignore it. No biggie for me. If you go with a community score require people to back up their score with a mini review of sorts.

Certis wrote:

I'm curious, for those of you concerned about stupid people finding the site, how did YOU discover the site? If you didn't come here when we first opened, there's a very good chance it was a link that brought you to the page. Are you afraid there are no more worthwhile folks left? Were we supposed to lock the gates once you were through? ;)

I literally have no God-damned idea how I found this site!

Certis wrote:

If we went with scoring, community involvement would be a must.

8 out of 10 Goodjers agree..

What do you want Shawn? If you want to be main steam and have a wider audience then do it. The site will get more hits but do you really want more guys like me hanging around?
If you want to be hardcore with different idea's some how, Then grow slowly and think of a better way to score a game.

Buy, Rent, Run,

Lobster Mobster's idea of percentage scoring if a game is worth 60 bucks or 40?

How much out of a dollar is a game really worth? A full buck or just 50 cents.

How many hours out of the day is this game worth playing. 12 hours or 2?

You guys are always coming up with great idea's so I don't think you should change that philosophy now. I don't think scoring is bad so I'm cool with you guy's doing it. I just think you guys should put a twist on it if you do.

Would you be willing to go back to games reviewed 6 months later and then give a late re-review? I find that scores for games change after playing a game after 6 months.

Lobo wrote:

I certainly don't count myself among those who are worried about an influx of IGN forum members. GWJ has braved the dangers of popularity quite well so far, and I don't expect that to change any time soon.

I do. I mean. Dang. I guess I'm a hypocrite because I couldn't pull together money in time for the donation drive, but I'll say it anyway. I would literally pay for this site to stay the PBS style way it currently runs. A few ads, a sponsored podcast, donations and lots of love. I can totally understand wanting to make a living doing this. I can completely understand wanting the fruits of your hard work to enable you to live a better life.

However, I don't think pushing the GWJ "brand" out there is the end goal. It's a means to an end, that end being more money, more access. I don't think many of us care if the GWJ brand is out there. It gets out there via word of mouth, links from Slashdot, etc. This site is findable by those who really want to be here and be cool and make friends and have fun being semi-grown ups who love games.

Back to the IGN argument. How did I find this site? I was sick of IGN. I was sick of Gamefaqs. I think I stumbled onto the site because of an article Slashdot linked to. So we get traffic here. We get traffic in a way that's much more organic. Someone Diggs an article written for the escapist or posts it to Slashdot, it gets picked up by older working geeks like myself and then we find our way here and discover friends, foes, fun, the best gaming podcast I'm aware of and some of the best people who love gaming.

So yeah, given that getting here took some effort I would personally be worried about the site turning into another IGN or 1UP, where the whole vibe and purpose was rendered pointless.

And let's be frank. Part of what makes this site so great, Certis.... part of what makes this community so great, is that regardless of your intent, your thought process, people's disagreement with you, etc. you took the time to actually have *this* discussion instead of just waking up one day and changing GWJ without any feedback. You can still do whatever you want with the site, but I hope you appreciate that as cool as you and Elysium and others are, what makes this site for me is that people like Danjo and Souldaddy and Donan and Fedaykin and LobsterMobster and Luna and Zero and many other unmentioned friends aren't drowned out by a howling horde of miscreants.

So, yeah, I have mixed emotions about this site getting too popular. I don't want the door closed. But I guess I want people to find their way here because the site does interesting reviews and writes interesting articles that make their way to Digg, Slashdot and other sites that care about more than just a number.

All I know is that I have a gamer tag with GWJ in it. As do many people here. When people ask me what it means I tell them happily. And I'm always happy to tell them what a great site it is. I paid $10 to have my tag switched to that and would gladly do it again. I'm happy to let people know where I call "home" as a gamer. I hope home stays home.

Oh, and if I get to be your portable gaming reviewer I take back everything I just said.

Dysplastic wrote:

The thing is, GWJ is quite honestly a lonely island of greatness in a sea of game-community filth. Even respectable sites like 1up with quality podcasts and legitimate, interesting reviews and content have forums that are absolutely un-readable.
I agree that the internet at large could use a little bit more of what we're cooking here.

I would agree completely. The question, though, is whether GWJ would change the tone and conversation of the gaming community or whether it would be absorbed and become just another site.

mateo wrote:

Scores are a crutch for people that cannot write or cannot take the time to really express themselves in writing.

Absolute nonsense. Even Consumer Reports gives products a "scored" rating, and it's not because they're hacks that can't write. It's because a score is a handy at-a-glance tool.

Are there writers, particularly in gaming, that use scores as a "crutch"? Certainly. But such a blanket statement as the one quoted above is nonsense. EDGE Magazine offers up some of the best and most insightful reviews of games I've read anywhere, and their inclusion of a score at the end is certainly no "crutch".

Mixolyde wrote:

I love the idea of community ranking across a slew of categories. The more info the better, imho. Even better if you can show a separate graph for 'I generally like this genre' vs. 'I don't generally like this genre.'

This is exactly what I think. I have worked in a spanish online game mag and I asked them not to give me any typical point and click adventure game. Because I don't like then and I would never give a good mark to them. For example I think Silent Storm is great a 10 for me, but this is because is the type of game for me. The liking of the type of game is esential for the score. Adding something like, "if you like this genre add +1 to the score", or if you like XCom add +2 to the score of Silent Storm

Honestly, folks, this is no big deal. These waters of change are Certis and Elysium's to navigate. If you've watched the character of the front page, GWJ has gone through far more dramatic transformations simply from the changing of the guard. We're still here.

The reviewers will get a form of compensation, something they deserve. Their work is quoted on other sites with heavy and non-biased traffic. Plus, those attention whores get to plant their opinions a little earlier and become involved with the debate instead of waiting a week after release when we've burned ourselves out on the topic.

If anything, GWJ can dip it's toe in the water. I suspect that the best solution is to lower the fidelity of the score. A reviewer is more likely to defend a score of 93.5 than he or she is likely to defend "2 out of 3 stars."

Lobo wrote:

Good reviewers adapt their writing to their audience, as well as to the format of the publication. So, in asking whether or not GWJ should adopt a rating system, we are in fact asking what style of review we prefer. Whatever your own preference, I hope at least to have made the case that the decision ought not to be utterly carefree.

I agree that a strong editoral vision is necessary.

I like scores.

Edit: But if you guys also want to go with the dollar value score that I invented, I won't stop you.

Braehole wrote:

Buy, Rent, Run.

I've been of the opinion that this is pretty much the perfect scoring system, at least for console games where renting is an option. It neatly sidesteps the age old question of "what's the difference between a 7 and an 8 anyway?" It tells consumers, at a glance, what they're looking for without providing so much detail that it makes the review text feel redundant. It still has the "two divided by three is sixty six" problem though.

I also like the dollar value score. After all, we are Gamers With Jobs.

I'm going to leave Duoae's comments off the map for the moment, because I have a tremendous amount of arrogance and self-confidence in what I actually put on a page now and then. Not always, but enough to know I'm at least "OK" at this writing thing.

I have thought about this a tremendous amount. You'll note (I believe) that I've never tossed a number at the end of a review here. If you dig around, I haven't done it in print either. This isn't an accident. I don't want to give them.

I am not against scores based on some kind of ridiculous "where's Lester Bangs" crapola that thinks somehow "games journalism" should rise up to some pulpit from which only English majors can preach. I, like many consumers of games media, appreciate the shorthand that a review score provides. I've browsed games by score more than once, and will do so many times in the future.

This has absolutely nothing to do with whether I have any interest, talent, or inclination to WRITE a review which culminates in a score. There have been a dozen great suggestions here on ways to improve what scoring is and what it means: dollars, time, rent/buy. These are all good ideas. At various times all sorts of methods have been tried in the press, from group scores, paired scores, breakdown scores, pros/cons to stars, numbers, thumbs and rampant chickens. (OK, I made the chicken part up). But here's the thing, I just don't want to write them.

There two hardest things to write (not just in games) are previews and reviews. They are hard because they imply requirement. If I give up on a game I'm supposed to review after half an hour, then what kind of review can I possibly write? Yet if I'm "supposed to" write a review of it, I have to force myself to either phone it in, or continue playing a game I'm not enjoying for many hours more. Forget it. I don't have the time.

Therefor, I'm going to self-select to only write about games that I either enjoy (Guild Wars), or that have some deeper reason for playing (Super Columbine Massacre RPG). I chose those two examples for a reason. In both cases, I personally enjoyed the process of writing them, and I am reasonably happy with the outcomes. In neither case can I even imagine putting a number at the end of them. I'm curious what numbers you all might have thought appropriate.

So, to be blunt: I'm all for folks tossing a number at the end of their reviews if they think it fits in with what they're writing. I'm more than happy for our benevolent overlords to start asking writers to do so. I just won't be doing it. Not because I'm a prissy prima dona, just because I don't have any interest in writing "those kinds of reviews." I simply Reading them? Sure.

Is that selfish? I suppose it is. Sorry. I've just got way too much writing to do already without cramming out stuff I simply won't enjoy doing.

The thing I like about putting a dollar value on things as a final score (not that I think this is the way to go if ratings are a forgone conclusion) is that some games get knocked down in part of the prices these days. There are games out there (All Pro Football 2k8, Shadowrun and Crackdown come to mind) that got sub-par reviews almost based purely on their perceived value. All three games probably deserved better than they got. They were being judged rightly on their value as $60 games. If you were to go to metacritic now, however, knowing that you can get each for much less, you might still wonder if they were worth the $25-35 you can now get them for.

while these variable rating "score" systems might be interesting (although I really don't see how a $ value or a buy/rent/run thing is any less subjective that a 1-5, or 1-10 scale), the whole POINT of Certis asking this question is to be included in the larger reviewing community. If that's his goal, then he needs to use a system that allows the aggregator sites to include the GWJ scores in their "meta" reviews.

I join those without a burning iron in this fire. My only complaint against the current setup is that we don't see enough of the great reviews that the front-pagers have proven their ability to write. If adding scores would increase their frequency, then more power to it.

Like Mixolyde and anyone (the poster, not the pronoun), the biggest beef I have with "big site" reviews is the lack of reviewer context. Is the reviewer predisposed to hate the genre, or enjoy even its most bitter fruit? Can she play even the most difficult platformers blindfolded, or is every stage of Super Mario Bros. a controller-hurling nightmare? Does he have a maxed out rig, or one that can barely handle the weighty demands of Windows 3.1?

Those are the things that readers need to know in order to reach a good decision about how applicable the review is going to be to their own experience, and very rarely are they mentioned in a professional review.