Part Of The Conversation

Section: 

Something was bothering me yesterday as I trolled Metacritic for PSP game scores, trying to refresh my memory on what's actually worth buying for the system. I have the infinite wisdom of the GWJ community to draw from, but even our collective memory may fail under the strain of nearly two years worth of games. With this is mind, the oft derided aggregate sites we disdain for reducing the quality of a game to a mere number serves a valuable function. If I want to know if Metal Gear Solid Portable Ops is good, I can punch it in and see that most reviewers seem to like it. No spoilers, no outright bias or flaming, just a raw number denoting quality so I can feel safe making the order.

What bothered me was that these scores were missing something. Our input. Not one aggregate site includes the Gamers With Jobs perspective on games because we don't score them. I'm starting to wonder if we should.

We dislike the common review structure so much we call our articles "Perspectives" so that we can do our own thing. While it suits our own purposes, I wonder if we're doing a disservice to the wider audience who may benefit from our perspective but will never be exposed to it because we're unwilling to play ball.

Putting aside the old debate of assigning a number to an experience, there are some broader issues at play that concern me as someone running an independent gaming site. For starters, a number would mean we appear on aggregate sites next to Gamespot, IGN, 1Up and the dozens of others sites that score their games. People who look up a title might see "Gamers With Jobs" and read something they would never have been exposed to otherwise. While we primarily write for out own amusement and our community, I like to think that the internet at large could use more of what we're cooking here.

The second thing that springs to mind is access to games before release. Like it or not, calling your articles "Perspectives" and refusing to assign a score is a total turn-off for a lot of PR people when they're deciding who should get early review code. If you're not going to contribute to the Metacritic pool -- which can determine whether or not they get paid a bonus -- they may be less inclined to offer you coverage opportunities. There are hundreds of gaming sites out there and early code (or even final boxed software) is often limited. It's a tough gig wading through the countless fan sites vying for your attention. On the surface, we're a pretty enigmatic site to understand already.

We also have our writers to think of. As a creative cat herder, I'm more likely to get a review from writers if I can spare them the expense of buying the game themselves. We're blessed with all the great content we get from such talented people, but as paid opportunities come up and personal lives get busy or financially shaky, every little bit can help.

Is getting a Gamers With Jobs review on the podcast or the front page before the release date of a game something readers want, even if it means making our review content a little more in line with what most sites are doing? I don't think assigning a point value to a game suddenly means the content of the articles has to change, but the fear of people skipping to the end of a review is a real one. No one wants their hard work reduced to a final paragraph, a number and a few screen shots.

With all this in mind, I open the floor to you, dear reader. Would we be selling our souls by scoring games? Is there some rating system that both satisfies the intellectual and the more mainstream reader? Maybe we need our own internal aggregate with a final score that reflects the views of many on this site. Help me out!

Comments

Mayfield wrote:

Will you set the number rating system out of 20, and then just toss darts at a dartboard please?

What if they have skill at darts?

Bioshock ranks a double bull? Halo's a trip 20?

Personally I think game scores are stupid. The fact that they are the only thing that most consumers of game reviews are interested in is and indication of the complete and utter vacuousness of most criticism in the industry. It also leads to weird effects like people comparing Halo to Bioshock or Persona 3 on the basis of the score, when there really isn't any reason to compare the games at all.

If, on the other hand, the score is a fact of life and will allow better content at the site, then I think this is a good tradeoff. I would suggest a tri-valued scale: 1, 2 and 3, with 1 being bad, 2 being OK and 3 being really good. Given that the 1-5 scales usually really mean 2-4 and the 1-10 scales usually really mean 7-9, I think this is reasonable.

I think that the "perspectives" point of view should be preserved even in articles with scores in the sense that the real purpose of a good game review should be to describe an experience in an evocative way, not to provide some kind of pseudo-objective reductionist evaluation of the "quality" of the game. If that makes any sense.

edit:

P.S. Jeanne D'arc is really good, as are the MLB: The Show baseball games.

I think you have some good points, and I think extra exposure for GWJ can only be good - you already deal with the negatives of higher levels of exposure..

It's not like any of us are going to stop reading the articles in favor of just looking at the scores, so what do you really lose? I don't care if the scores are arbitrary points in a vacuum, because I don't care about them at all. Smack a number on, call it a day.

Use something other than 1-5, 1-10 or 1-100 though. Fight the power!

ChrisGwinn wrote:

Use something other than 1-5, 1-10 or 1-100 though. Fight the power!

For reviewing RPG games you should use 3-18, like D&D attributes.

To pick and choose from a couple of the ideas here, I am a fan of the suggestion that the rating should help me make a purchasing decision regarding the game. Someone mentioned buy/rent/skip/wait for a price drop/etc. The simpler, the better, IMO. If you want to tie that to a numeric in order get more exposure, I'm all for it. It's all about the rubric, really:

5/10/100/A doesn't have to mean "this is the best game evar!" It could just be an indication that the game is worthy of an immediate purchase. "Buy it"

0/F likewise doesn't have to mean it's the worst ever - just that it's not worth spending your money on... or even bothering to try your friend's copy. "Skip it"

50: "Rent it"

In short, define the rubric first and then assign points to it but keep it simple enough (i.e. 3-5 gradations and that's all) that people aren't trying to figure out how much better a 95-point game is than a 90-point one.

Just my 2/5/10 cents. (Stupid falling dollar.)

Screw other people. They want to find out what Goodjers think about games, then we ain't gonna make it easy for them.

I think we could have a little cake and eat a little too if you included a score, even a standard 10 point system, but did it in a creative way. If the writing staff can manage to put out their usual high quality "perspectives," (i.e. not sacrifice the normal quality of the site) they could then include with that a sort of "TPS report" (keeping with the corporate theme of the site) that contains a score but doesn't take itself so seriously. If I remember correctly, GWJ used to include a score like that. Maybe make it a fake supervisor evaluation and add in witty comments in the comment lines like "Too much FP and not enough S in your game, Johnson. See me in my office." That's a lame example, but something along those lines.

That's probably too contrived for people here to buy into. I'm not suggesting we dumb the site down at all. I just think there's a way to pull it off and I have faith that would be the case if a scoring system were adopted.

Personally I don't need numbers. All I really need is an informed, mature outlet to talk about games and network with other gamers and hear their impressions. I don't think numbers and quality are mutually exclusive, though.

Maybe we could do something like DAoC's con system.

gray, green, blue, yellow, orange, red, purple, p+,p++,p+++

=P

I agree with zeroKFE-- a really coarse system (1-4 stars at the most granular, but a three-star system would be even better) if it lets you reap some kind of benefits doesn't seem like it would hurt.

You could do something perverse like assign random numbers, but I think metacritic accounts for that and shuts you out.

You could do it like Monster in my Pockets!

1 would be crap

5 would be average

10 and 15 are strong

20 is powerful, like Dracula.

and 25 is legendary. One of my 25's was The Phoenix, I think.

Even if you assigned numbers, there will still be plenty of debate about the merits of a given game for the people that want it.

I see no problem with this at all.

IMHO, being snooty (not that you're being snooty) about giving review scores seems to force readers to use your information the way YOU want them to... not that way THEY want to. If I want to read every paragraph of your review, savoring each verb clause and prepositional phrase, then so be it. If I, on the other hand, just check to see that you gave it an "8" or higher before I buy it, then why can't I?

You're right-- by not giving a summative score, you're cutting yourselves off from the people who want those scores... you're forcing them to go look elsewhere for reviews that come on or slightly before release, and as you pointed out, we ALL look at Rotten Tomatoes or Metacritic for a resource anyway.

If Pulitzer prize winning critics can assign stars to films, I don't see why GWJ can't do likewise for games. I don't think Roger Ebert sits up at night worrying about this kind of thing Even the Onion AV Club, which used to disdain scores, started adding them a few years back.

BTW, Certis, you KNOW where to go for PSP game recommendations. Why are you going to Metacritic?

Oso wrote:

It may not be possible for the site to remain static. It probably isn't desirable for the site to remain static, change is a constant and GWJ must evolve or die. If this is an evolutionary step that will make things better for the site, or even if it just makes things better for those who bring us the site, it may be a good idea.

I fully agree with this. Having said that, why are we defining the site based on outsiders preferences? I don't mean it in a "them vs us" way, but if no one else gave numerical ratings would we be having this discussion? No.

To me, what has made GWJ so great has been the community that did not find what it wanted from the high profile sites, and settled on mature gaming discussions and articles that made sense. GWJ hasn't really ever done things the "normal" way of the high traffic sites, and despite some high profile links, hasn't changed the way.

At what cost is giving up our semi-uniqueness of not having a numerical score?
Prerelease games? I've never complained that there wasn't a review the day of the games released. In fact, I like that in the podcast, the "Games we're playing now" is by in large a game to which I have access.
Larger audience? A valid point, but if that is a goal, I'd rather that it be addressed in a different way.
A larger payoff or stepping stone for Certis and staff, via the larger audience and connections? Not specifically addressed, but if that is the case, I'm seriously all for it. You guys have built up the site with your toil, and I have no problems you reaping the benefit.

I do think it's worth pondering that most of the front page writers here have gone on to the Escapist, but not gamespot.

Certis wrote:

Putting aside the old debate of assigning a number to an experience,

I realize this is necessary in order to consider the points you wanted to make, but I don't think it should be casually brushed aside.

To be blunt, I'm uniformly against numerical scores. If the staff here decides they want it, it won't keep me from coming. I just think it will have lost a little bit so something in my (and maybe only my) mind.

I am 100% all for scores in reviews.

I not only have a life outside of games, I certainly have a life outside of reading game reviews. I don't have the time to read everyone's 5 page writeup on every single game. Scores are a perfect way of doing some basic weeding out, and bringing games to my attention that I wouldn't have otherwise slogged through a multi-page review to find out about.

When I see a high score, it gets my attention, even if it's something I would have otherwise completely ignored (example: Lego Star Wars, which seemed like a recipe for a double helping of licensed badness, especially since I am no SW dork and look at the SW license as a liability rather than an asset for something like KOTOR. But the scores got me to look more closely into the game, and what do you know, it's not just licensed-game shovelware)

I like this new "Can of Worms Fridays" feature.

There are very good reasons not to generate numerical scores, which have already been discussed in depth here and in previous articles. But in the current enthusiast press climate, I think that in order to participate in the overall critical dialogue (yeah, I use that term loosely) and maintain a presence within the industry you have to have review scores. Given the visibility of the aggregate sites, this is increasingly the case.

There are a handful of reviewers and publications out there that have earned credibility in my book, and I do find their review scores meaningful. I don't believe that scores are anathema to good critical writing. I've got no doubt that Gamers With Jobs would retain its credibility even if scores were attached, and that the quality of the site's output would remain just as high. I'm also not worried about an influx of asshats. As Certis noted, GWJ does a good job of weeding out the undesirables.

I think that the people who come here for the in-depth review content won't run away because scores are attached. I also think scores would draw new readers to the site. And although I'm certain that many would show up simply to debate a game's numerical score, I'm sure that plenty of others would be just as interested in the reasoning behind the number.

For me, it boils down to a few basic questions: Is Gamers With Jobs content to remain just outside the margins of the current "system" of game evaluation? How badly does GWJ want to be heard? Does it want a "vote" in the debate, even given the flawed nature of that debate? I don't know the answer. I'm perfectly content to see GWJ soldier on, independent of the rest of the enthusiast press, doing what it does best: providing an intelligent, alternative voice. But I don't think scores would be the end of the world. And it'd make me smile to see GWJ linked at Gamerankings and quoted at Metacritic.

Then there's the issue of how games should be scored. From my perspective, scores aggregated on a 100-point scale end up functioning like schoolwork letter/percentage grading. 90-100=A=excellent. 80-89=B=great. 70-79=C=average. 60-69=D=below average. Anything below 60=F=fail. Some sites maintain a system that places average or middle-of-the-road games at 50% (or 5 out of 10), but in reality you'd be hard-pressed to very many decent games with aggregate scores in the 50s. But I suppose that's another conversation altogether.

I have to confess I like scores... Not because I'm lazy and need a number. I just like hearing editors and readers bicker over it.
Also, I hate rent-it as a review score.. It's tacky like a velvet elvis.

What about community based numbers with the average being the ranking? Keeps with the feel of GWJ and gives a score to games.

Mixolyde wrote:
ChrisGwinn wrote:

Use something other than 1-5, 1-10 or 1-100 though. Fight the power!

For reviewing RPG games you should use 3-18, like D&D attributes. ;-)

Awesome. I approve of this idea! You could also do hours in the work day (Call in Sick/ 10 AM / Overtime!).

In general, I like tough three or four star rating systems, but the big problem with using one here is that the reason for having ratings at all is to get into gameranking/metacritic/etc. And gamerankings, at least, doesn't bother normalizing, so you end up with, say, a four star system mapping to 25%/50%/75%/100%, which doesn't do a good job of expressing your review in the aggregate, which makes it harder to do what you want to do. Gamespy has a five star system and has this problem - it's worse with fewer.

The simplest solution is probably to just give letter grades: A, B, C, D, F, Incomplete. Gamerankings would probably map those to meaningful percentages, and you wouldn't have to deal with the people who like to whine about why one game got a 9.3 and another game got a 9.5.

Now, this article has escalated in the few hours i've been in the pub so i can't really be bothered to read through several million similar posts (read 44 and counting) to then start my diatribe.

1. I agree: I dislike scoring systems and like the perspectives articles that are currently shown on GWJ.

2. I like the fact that GWJ supervisors interact and elicit the thoughts of the users.

3. I understand that GWJ has reached a current limit for expansion in its generally current form and needs either a)offer something more or b)get more exposure.

4. My opinion is that all of the above should be allowed.

The solution as i see it? Allow perspectives as they are currently formulated but also allow users to score the game (perhaps including or excluding the reviewer - difficult for me to discern which is best). Make our score a meta-GWJ score which is then put into the meta-scored sites. Have our score be an agregate rather than a definite.

For games that aren't reviewed by the GWJ crew you can have your Wiki-style (as currently employed but perhaps enhanced via the upcoming site upgrade) games pages where screenshots and also user reviews and aggregate scores are kept - minus a GWJ proper perspective.

Duoae wrote:

Make our score a meta-GWJ score which is then put into the meta-scored sites. Have our score be an agregate rather than a definite.

I don't think the meta-scored sites do this. They're probably not setup to have changing score values per site.

Desram wrote:

velvet elvis.

Good book.

The Fly wrote:

critical dialogue... debate... conversation...

Cacophony seems like a better word for what we'd be joining. Certis seems to think that we would bring order to the chaos -- that we would speak into the darkness of the internet milieu and create light. He has a vision that GWJ would saunter boldly into the tumultuous bedlam of "enthusiast news blogs" and valiantly feed its denizens their much needed Ritalin.

OK, maybe he doesn't think in such terms, but I'm not afraid to (so long as I can hide behind Ely)!

ChrisGwinn wrote:
Duoae wrote:

Make our score a meta-GWJ score which is then put into the meta-scored sites. Have our score be an agregate rather than a definite.

I don't think the meta-scored sites do this. They're probably not setup to have changing score values per site.

You'd have the score that's submitted to the meta-score sites as a week's worth of scores. After that it would be locked.

[edit] To clarify: This way if you allow community participation you get:

1) Scores that reflect the release or near release feelings and attitudes of the game.

2) A fairly non-biased score as it is an aggregate and not just one person's opinion.

The way it would work is that you wouldn't have a review score for a preview. ie. if you receive preview code you wouldn't allocate a score and instead the score would be retroactively added to the game in it's first week of release. You could have a section on the GWJ main page that has the current games that are open for rating - on the sidebar.

Also, as i already pointed out, GWJ staff wouldn't be assigning the scores. but the community. Thus the integrity of the staff/site is intact and the current system of being "outside" the general current system would continue - just that the community's consideration (one that Certis says that he values greatly) is also taken into account and given a general score.... which is then reflected on the meta sites.

Nay.

Reviews are inherently subjective, and a review score is just dressing it up to look objective. That's the very definition of inane, or if it isn't then it should be.

Would adding review scores be selling out? Yep, selling out for exposure. Do the benefits of exposure outweigh the negatives of conformity? A difficult question.

A subtle positive of our current, nonconformist way of doing things that might be overlooked is that we occasionally do get treated like one of the pack even without review scores. When Ken Levine or Todd Howard recognize that sites like ours are worthy of their time, that tells me something about them, their motives, the respective corporate idealogy, etc. There is just as much information to be gathered by their very speaking with us as there is in their words; information that is lost when you link the potential of bumping up their game's metacritic ranking.

That loss would be even greater because I do not know of an alternative site that would fill that void. Alternatively, I can't wade through gaming media without tripping over dozens of reviews condensed to a number.

Putting a (usually) single digit number at the end of a review isn't going to end the world. My lord.

GWJ still isn't going to turn into a review mill-style site like a 1Up or IGN.

Danjo Olivaw wrote:

Would adding review scores be selling out? Yep, selling out for exposure. Do the benefits of exposure outweigh the negatives of conformity? A difficult question.

This is really the question (succinctly put, thank-you-very-much-Danjo!) Free stuff and maybe meeting new people, or avoiding creating something that many of us feel is silly, and is already in abudance.

I think those not currently at GWJ have and will continue to be given many opportunities to stumble upon the site, as Certis noted. Access to games before release is a non-issue to me - I view reviews here as invitations to discussion, so why should perspectives/reviews go up before anyone else has even played the game?

So the two main potential benefits are increased exposure and increasing GWJ appeal for early release copies for review, right?

But if traffic to our beloved GWJ is double compared to last year, and references by slashdot/Penny Arcade/etc bringing even more folks to our neck of the internet, it doesn't seem like exposure is really a problem. Cool people are gonna stick around.

So how important is it to attract the coveted early review copies of games? Personally, I don't think GWJ is lacking anything, so my opinion is that it's not necessary to change anything with the format.

Oh, and to add my bio to this:
I come to GWJ for two main reasons. It is just about my only source for all my gaming needs, but more importantly I'm incredibly addicted to this web site. Please God, help me to stop.

Can we handle much more traffic, Bagga? (Can your mom?)

I think there's a bit of fixation on early review content here, which is something I mentioned, so fair enough. To me though, this part is more important than getting things early.

We also have our writers to think of. As a creative cat herder, I'm more likely to get a review from writers if I can spare them the expense of buying the game themselves. We're blessed with all the great content we get from such talented people, but as paid opportunities come up and personal lives get busy or financially shaky, every little bit can help.

You may have noticed we don't actually do that many reviews right now, this is part of the reason why.

Gamers With Jobs has changed quite a bit in the two years that I was away. The podcast is now a feature of the site that gets regular mainstream exposure, many of our writers/contributers are now members of that mainstream, and Certis actually has a regular job.

What hasn't changed is the fact that we are still the best site on the internet for those that love games and are interested in having something resembling normal, humane, and decent discourse about games and gaming.

I don't believe that adding numerical scores to our reviews will significantly increase the amount of riffraff hanging around. We've always had them and they've always been dealt with. It will get the name out there, though. It may bring in the sorts of attention that Certis is hoping for.

Do we want that attention? It depends. It will all matter on what Certis and Ely decide to do with it.

Leverage that attention and use it to produce something that our community will be proud of. Use it to create something that shows a place like GWJ can work. Use it to show that what the gaming community needs isn't another corporate sphincter leaking the usual fecal matter but rather that it needs a real voice for mature gamers. Gamers that created a world where a video game can earn THREE HUNDRED MILLION DOLLARS in one week don't need or want another IGN.

They want GWJ. Give it to them.

GIVE THEM STAN!

Offering a review 'score' isn't going to change the face or feeling of GWJ. I understand the desire to want to be included in the catalog when it comes to game review scores and to possibly save that one person from laying down the full amount for Halo3 when TF2 (and the rest of the Orange box) is worth SO much more.

I'm fine with your choice either way but some food for thought for all parties.
1) What other site would even bring this up to the user base? The oligarchy that controls the reigns knows they dance with the devil but they're still willing to have open communication before rendering final judgment even though they don't have to.
2) An aggregated score of registered member's opinions intrigues me but the system may be a pain in the ass to design and since it most likely is, here are some features I'd like to see if such a thing were implemented from my imagination.
A> A limited time window for the ability to 'score' the game and the ability to only vote once (but the ability to change your vote if within the time slot).
B> Forcing the user to put how much time they've played the game in hours (ranges would suffice).
C> Your system specs
D> The ability to share your vote history if it's not mandatory to have them shared.
3) I say keep the perspectives the way it is and create a new category for the 'reviews' (keeping with the office worker format I recommend, "Evaluations")

I for one welcome the unclean masses, odds are they're already here they just haven't registered yet.