Part Of The Conversation

Something was bothering me yesterday as I trolled Metacritic for PSP game scores, trying to refresh my memory on what's actually worth buying for the system. I have the infinite wisdom of the GWJ community to draw from, but even our collective memory may fail under the strain of nearly two years worth of games. With this is mind, the oft derided aggregate sites we disdain for reducing the quality of a game to a mere number serves a valuable function. If I want to know if Metal Gear Solid Portable Ops is good, I can punch it in and see that most reviewers seem to like it. No spoilers, no outright bias or flaming, just a raw number denoting quality so I can feel safe making the order.

What bothered me was that these scores were missing something. Our input. Not one aggregate site includes the Gamers With Jobs perspective on games because we don't score them. I'm starting to wonder if we should.

We dislike the common review structure so much we call our articles "Perspectives" so that we can do our own thing. While it suits our own purposes, I wonder if we're doing a disservice to the wider audience who may benefit from our perspective but will never be exposed to it because we're unwilling to play ball.

Putting aside the old debate of assigning a number to an experience, there are some broader issues at play that concern me as someone running an independent gaming site. For starters, a number would mean we appear on aggregate sites next to Gamespot, IGN, 1Up and the dozens of others sites that score their games. People who look up a title might see "Gamers With Jobs" and read something they would never have been exposed to otherwise. While we primarily write for out own amusement and our community, I like to think that the internet at large could use more of what we're cooking here.

The second thing that springs to mind is access to games before release. Like it or not, calling your articles "Perspectives" and refusing to assign a score is a total turn-off for a lot of PR people when they're deciding who should get early review code. If you're not going to contribute to the Metacritic pool -- which can determine whether or not they get paid a bonus -- they may be less inclined to offer you coverage opportunities. There are hundreds of gaming sites out there and early code (or even final boxed software) is often limited. It's a tough gig wading through the countless fan sites vying for your attention. On the surface, we're a pretty enigmatic site to understand already.

We also have our writers to think of. As a creative cat herder, I'm more likely to get a review from writers if I can spare them the expense of buying the game themselves. We're blessed with all the great content we get from such talented people, but as paid opportunities come up and personal lives get busy or financially shaky, every little bit can help.

Is getting a Gamers With Jobs review on the podcast or the front page before the release date of a game something readers want, even if it means making our review content a little more in line with what most sites are doing? I don't think assigning a point value to a game suddenly means the content of the articles has to change, but the fear of people skipping to the end of a review is a real one. No one wants their hard work reduced to a final paragraph, a number and a few screen shots.

With all this in mind, I open the floor to you, dear reader. Would we be selling our souls by scoring games? Is there some rating system that both satisfies the intellectual and the more mainstream reader? Maybe we need our own internal aggregate with a final score that reflects the views of many on this site. Help me out!

Comments

Pages

I personally breathe a sigh of relief whenever I read a "review" here that does not have a review score. It takes the article out of the context of the "game" of game reviews, and reminds me of a thoughtful conversation with a friend instead of detached analysis of a critic.

As for the comments on games prior to release, I can wait. I've got a job, wife, kids, and much more to stave off any compelling urge to buy a game on release day that I'm not certain I'd like. I wish I could say more on the subject, but I'm off to a job meeting...

Interesting questions. Gamers With Jobs is the only internet website I post on and the only group of people (outside of my friends) I choose to play games with willingly online because of the mature tone and atmosphere that has been carefully crafted here. While I do not think that GWJ adopting a points based rating system will instantly result in everything that has been built crashing to the ground as if some magical lynch pin has been pulled, my concern would be for us actually showing up on sites like Metacritic.

GWJ reviewers have always been completely honest and forthcoming about games that are reviewed here. What worries me is that someday, someone reviews a game that has a rabid fan base (everyone remember Pred's impressions of Oblivion and Elysium's critique on Vanguard?) and assigns a low score which gets posted on Metacritic. The rapid fan base, who carefully monitor what scores their game is getting, spends their days refreshing Metacritic's page to continuously check what scores it receives. They see the low GWJ score and instantly the Fan Boy signal (TM) is triggered in each of their heads, resulting in a massive influx of irate zealots bombarding us with reasons why the game is so great and why we are the suck for not giving it a higher score.

I may be over generalizing and predicting doomsday scenarios here, but the absolute righteous zeal some people have to defend a game (or on the flip side, trash it) demands that they seek out any and all heretics who do not share their view. By joining pages like Metacritic, I think it will just provide the zealots with a way into our little piece of paradise on the internet.

I admit, I enjoy seeing hard numbers next to reviews and secretly enjoy Gamespot's new icon rating system as well. I still however, read the review from start to finish to get an idea of why those numbers were assigned. Assigning numbers to a game I have no problem with; it's getting linked by the other sites is what scares me.

(Also, if you ever have a need for more people writing reviews for the front pages, I am more than happy to apply )

Can you get your score on the metacritic sites if it is just a simple "thumbs up, thumbs down" or a "skip, rent, buy" sort of scale? Because some sort of scale with very limited gradation that boils down to either recommending a game or not might might have less effect on how the reviews are actually written, and would bring up less questions of what the hell it actually means to label an experience with a number. And at the same time if a scale like that is good enough for the aggregators, then it would still serve your desired purposes of getting the GWJ word out and getting the early review code in.

I want to put my vote in that I think number scoring is a horrible idea.

Looking up review numbers is a habit. One, if broken, will never be missed.

The only people I listen to about new games now are my friends and some comprehensive video reviews (humorous and serious alike). After I stopped caring about the score I found myself latching on to sources with better content.

I still get Game Informer and this helps me disregard numbers even more because on many occasions their own numbers don't match their opinions.

You just can't quantify affection and disdain. At least, not with a 1 to 10 score pulled out of your ass you can't.

I think the best you can do if you really need something is to have a thumbs up or thumbs down. Even then you're setting an opening mindset for your own reviews that is going to instill bias into your readers before they even read the review. More likely than that you are going to cause people to stop reading your reviews altogether.

I still really like rating games by dollar value. I guess you could convert that to a percentage score by calculating what percentage the dollar value is of the actual retail price?

Anyway Certis, Portable Ops is great but I think there's an enhanced version coming out. I've also been playing me the hell out of some Monster Hunter Freedom 2. Got about 160 hours on it.

I don't know what you should do, but I can tell you how I "interact" with this site and you can judge if thats how you want it to be or not:

When a game comes out that I'm interested in, I usually try to hit up a couple review sites to find out a bit about the game, to make sure its a valid purchase. GWJ is never one of those sites. Maybe its because I tend to buy games the day they come out and GWJ doesn't have a review of them by then or maybe it is because I am looking for a review number. However, that isn't to say that GWJ hasn't influenced my purchasing choice in the past. Sometimes I will be browsing the GWJ site and find a "perspective" of a game that influences me to run out an buy a game. This hasn't happened often, but it is nice when it does, because that game is usually something I would never have even thought of considering to purchase before reading your reviews.

I also tend not to use GWJ as a portal to information about games prior to launch. When I'm anticipating a game, again I usually head over to one of the bigger sites to browse for info. However I must say, that the recent interviews you had with Ken Levine was some of the best stuff I've ever heard, and I'd love it if you were able to get more inside scoops like that.

So I've listed all the things I don't use GWJ for. Why do I still end up on this site multiple times a day? I think its because I love the "perspective" you put on a culture that has become so mainstream that coporate culture has started to rape it, taking all the fun out of gaming. Coming to this site reminds me that there are still people out there that love gaming for the sake of gaming. GWJ doesn't seem to buy into the latest game hype, but instead creates its own opinions of things and I love it. And above all, GWJ is a very mature site!! I'm a hardcore multiplayer gamer, but its gotten to the point that it sickens me to play with most of the people that are on the internet. But with my interactions I've had with the GWJ crew, every one is amazing. Jumping onto the TF2 server, I've never once been assaulted with verbal abuse. And I'll keep coming back for this reason.

So do I want you to start giving scores to reviews? I don't know.

Certis wrote:

Like it or not, calling your articles "Perspectives" and refusing to assign a score is a total turn-off for a lot of PR people when they're deciding who should get early review code.

Isn't honesty and the access to early code mostly mutually exclusive though? What happens if you get access to some game early, publish an honest opinion that paints it in a negative light and then receive less access in future as a result? Doesn't that put you back in square one? I realise that not all publishers will be that fickle, but I don't want GWJ to risk sullying itself just for some sneak peeks.

Lord_Xan wrote:
Certis wrote:

Like it or not, calling your articles "Perspectives" and refusing to assign a score is a total turn-off for a lot of PR people when they're deciding who should get early review code.

Isn't honesty and the access to early code mostly mutually exclusive though? What happens if you get access to some game early, publish an honest opinion that paints it in a negative light and then receive less access in future as a result? Doesn't that put you back in square one? I realise that not all publishers will be that fickle, but I don't want GWJ to risk sullying itself just for some sneak peeks.

We do it all the time, that's not even a concern for us. I always tell company reps that honesty works both ways, we may not like your game today, but the next one could be great and our readers will respond to that if we maintain that level of trust. Some won't talk to us after, but that's ok.

"That's a terrible idea, a terrible idea."
"That's the worst idea I've ever heard. Forget this idea it is terrible, terrible."

Scoring kinda barfs in the hair of the concept of the Perspectives, don't it?

I may be over generalizing and predicting doomsday scenarios here, but the absolute righteous zeal some people have to defend a game (or on the flip side, trash it) demands that they seek out any and all heretics who do not share their view. By joining pages like Metacritic, I think it will just provide the zealots with a way into our little piece of paradise on the internet.

A valid concern, but it's worth noting that both Pred's Oblivion piece and Elysium's Vanguard piece did not have scores and we still got hammered. We get linked from game-specific sites and places like Bluesnews, Shack, etc. anyways, it's part and parcel with running a site. Remember, it's all about netting a handful of cool people out of the hundreds of smacktards that may show up initially. Most of them move on once they realize they're not welcome, and the cool people stick around. I can't tell you how many people have told me they came by the site through some review link or another. Some of your favorite posters here, to be sure.

Scoring kinda barfs in the hair of the concept of the Perspectives, don't it?

I think Perspectives could stick around as a seperate article type without a score. It's a good format for going off on tangents and not actually feeling like you need to finish a game before writing about it.

Certis wrote:

A valid concern, but it's worth noting that both Pred's Oblivion piece and Elysium's Vanguard piece did not have scores and we still got hammered. We get linked from game-specific sites and places like Bluesnews, Shack, etc. anyways, it's part and parcel with running a site. Remember, it's all about netting a handful of cool people out of the hundreds of smacktards that may show up initially. Most of them move on once they realize they're not welcome, and the cool people stick around. I can't tell you how many people have told me they came by the site through some review link or another. Some of your favorite posters here, to be sure.

Quite true, I myself was relayed here via Voodoo Extreme way back in the day.

I don't hit GWJ for reviews of games that are just coming out. Scoring the games and getting on the early review train might change that, I don't know. However, that's not what I currently try to get out of GWJ. I get more enjoyment about picking up an old game that we're talking about on the forums than trying to pick something up on day one. Usually if I pick up a game on the first day, I already know I want to buy it sight unseen.

Would I welcome GWJ reviews if they were on release day and scored? If they're done with the same consistent style and quality of the rest of the site, I think I might. However, I don't think that's a guarantee that I'll check it out any more than any other site on metacritic. It might bring in new people, there's a chance.

The other thing is do we really want people drawn in by a metacrtiic link? Possibly, really depends. I think there's a chance it could bring some new blood to the site, and there's an equal chance it'll flood us with pubtards.

I wouldn't have a problem with Gamers with Jobs scoring games. As much as we love to bemoan numerical and symbolic rating systems, it's what most of us rely on for a quick fix. Considering the high levels of integrity the GWJ writing staff, I personally would feel that if GWJ says it's a 9.5, the game is definitely a 9.5 and not what the PR companies want it to be.

As for being listed on an aggregation site, I see both a boon and a bust. The GWJ community is a very special one indeed, one I'm very proud to be a part of and actually talk about it a lot with my real-time friends. While I would love to see it grow and mature I think driving more traffic to GWJ is going to make moderating the site that much more difficult. I love the forums here because people are kind and respectful and the moderators do their best to weed out the trolls.

I'm not sure how hard that is right now, but is it going to be that much harder when you've got more people coming to the site and joining the forums after following the link from Metacritic and Gamerankings to us?

Is there a way we can give ratings in a variety of categories without giving an overall rating? Say reviewers give ratings in a ton of categories. Start with the usual stuff that gamespot rates like graphics, sound, replay value, tilt, and add as many more as you can think of. Add more in depth things like difficulty curve, depth of story, pacing, mood effectors, control scheme, value for the dollar, twitchiness, brain usage, tendency to induce seizures, tendency to induce aneurysms, etc. And then instead of numbers just go with 'below average', 'average', and 'above average.' The reviewer should note whether you're comparing it to all games, or all of certain genre, or all in the last year or whatever you can.

So now you're giving ratings, but on such a multi-dimensional scale that they might actually be helpful to players looking to decide. Your reviewers could decide which categories are actually applicable to the game in question and only rate those. This is probably a lot more trouble than it's worth, but maybe it's a starting point.

I'm not warm to the idea, but I can be convinced. Basically, if Goodjer Illuminatus weigh the benefits that rating will give the site compared to the risks expounded on above and come out feeling that the site will be better if we post ratings, well, you have certainly earned my trust so I'll support your better judgment.

That said, I'd ask you to weigh what ranking will entail with the sentiments expressed in "How to Enjoy Games More" article. Will ranking games give us anything that the "Enjoy Games" philosophy doesn't council against? If you all think it does, you have my trust and support.

It may not be possible for the site to remain static. It probably isn't desirable for the site to remain static, change is a constant and GWJ must evolve or die. If this is an evolutionary step that will make things better for the site, or even if it just makes things better for those who bring us the site, it may be a good idea.

Over all, given the information at hand, since you are asking my opinion I'd suggest not adding rankings. I can change my mind, but the gut says no.

I like being able to read about a game and come to my own conclusions. It's nice not being subjected to a 5/10 because a game doesn't look as good as Bioshock. I think the perspectives are great change of pace.

I can see which way you're leaning, Certis, and I understand it fully. After all, denying our opinion to the greater population of the internet is a crime on par with denying the ladies the glory of basking in my presence and/or wang. I don't personally have a problem with numbers, and we've had some good discussion about what kind of scale would be most useful for readers (worth in dollars, buy/rent/pass, etc.) I do have some concerns, however.

Mainly, if we increase traffic, I fear that we may become too big for our sexy, sexy britches. That is, we have a wonderful community here, and I think that increased traffic could result in a dilution of our potent communal and virtual mojo. It's certainly possible that the waves of internet madness will wash upon our server and break upon the front page, leaving the rest of the forums largely pure. If that's true, then I suppose I could learn to survive on that higher ground more exclusively (Front Page fanboy though I may be). If it's worse than that, then I hope your ban hammer is ready, and may God speed it toward its undeserving victims.

Future GWJ mottos:
Believe us, it's for your own good.
Rapture is leaking.
Welcome to Xanadu.
Hammer of Troll Bane (+2)
butt pat nostalgia

Score or no score, I find reviews by any single person do not match my opinion. I always read several reviews before letting it influence my decision. I say sure, put a score on there if it means you will hit a wider audience, the score is worthless to me by itself anyway. HOWEVER, I am a huge opponent of getting pre-release games. I think reviewers should spend their own money to buy games. Anything that would make a reviewer have a different experience then a regular joe buying a game at a store or on steam with his own hard earned cash, or by having a hyped game 2 weeks early could jade the reviewer.

I see a lot of concerns about having our ecosystem ruined by public attention. Understand that we've been slashdotted, Penny Arcade Wanged and much more in the past six months. The site's traffic has nearly doubled since this time last year and everything is still going fine. Like the question of integrity, the question of idiots finding the site is not a concern for me because we're all pretty good at weeding them out. It's easy to over-estimate the impact of a Game Rankings listing versus say, a link from No Mutants Allowed. I wouldn't anticipate a sudden flood of new readers because we start ranking games.

PoderOmega wrote:

HOWEVER, I am a huge opponent of getting pre-release games. I think reviewers should spend their own money to buy games. Anything that would make a reviewer have a different experience then a regular joe buying a game at a store or on steam with his own hard earned cash, or by having a hyped game 2 weeks early could jade the reviewer.

I agree. Furthermore, I don't think reviewers should be reviewing games that they don't "play". I would prefer that reviewing the game is not the primary focus, but playing the game is.

As far as scoring systems go, I dislike a number/points based scoring system. I'd prefer to see either a letter grade system (A for excellent, B for above average, and so on) or a rent/buy/skip rating system.

Mixolyde wrote:

Say reviewers give ratings in a ton of categories. Start with the usual stuff that gamespot rates like graphics, sound, replay value, tilt, and add as many more as you can think of. Add more in depth things like difficulty curve, depth of story, pacing, mood effectors, control scheme, value for the dollar, twitchiness, brain usage, tendency to induce seizures, tendency to induce aneurysms, etc. And then instead of numbers just go with 'below average', 'average', and 'above average.'

Interesting idea.

I vote nay to numeric rankings.

This might require a lot of extra work, but I find Boardgamegeek.com's rating structure to be incredibly helpful. Essentially, their users rate games individually, and each game's database page displays a Bayesian average of those rankings. What really makes it work well is the rubric attached to the rating system, which basically comes down to rating how much you'd like to play it.

# 10 - Outstanding. Always want to play and expect this will never change.
# 9 - Excellent game. Always want to play it.
# 8 - Very good game. I like to play. Probably I'll suggest it and will never turn down a game.
# 7 - Good game, usually willing to play.
# 6 - Ok game, some fun or challenge at least, will play sporadically if in the right mood.
# 5 - Average game, slightly boring, take it or leave it.
# 4 - Not so good, it doesn't get me but could be talked into it on occasion.
# 3 - Likely won't play this again although could be convinced. Bad.
# 2 - Extremely annoying game, won't play this ever again.
# 1 - Defies description of a game. You won't catch me dead playing this. Clearly broken.

Obviously the replayability of board games is on the whole greater than that of video games, but I think a pared down version of this, something similar to a buy/rent/stop system geared towards video games, would be great. On BGG, users can also attach comments to their ratings, so you can really get a sense of where the rankings come from. And then we could internally look at the collective GWJ results as our own little metacritic.

I'm not sure how this could be done without implementing an unwieldy games page database ... but food for thought.

Certis wrote:

I wouldn't anticipate a sudden flood of new readers because we start ranking games.

Guilty Spark wrote:

Why naturally the Flood is simply too dangerous to release, and mass sterilization protocols may again need to be enacted. Of course, samples were kept here after the last catastrophic outbreak -- for study. It seems that decision may have been in error.

Eh, I couldn't care less about numerical rankings.

I just need you to help me decide:

Buy, Rent or Skip?

Or even "Wait for price drop".

---------------------
edit: If you need an explanation:

Because anything else is just for neurotic, anal people who will obsess over a game's rating and fight the reviewer for a decimal of a point.

It will also aggravate people who liked the game and feel bad for playing a game that got a "7.5 which is MEDIOCRE", and so on, and it ultimately does not do anything for the gamer.

It also does nothing for editors who fight with reviewers because their review is implying the game is crap, yet they gave it a 8.5

And it also aggravates reviewers, because what if they had a fun time with a game, but ultimately they know the game is way too short and not worth 60 bucks?

Quantifying a subjective review is a convenient time saver, but I don't think it adds anything to the article. I like scores to get an quick idea of the opinion of a game without having to read the review. I actually look forward to and read the reviews here, so the score wouldn't really appeal to me.

Edit - oops, stupid work, thought "subjective" and typed "objective."

If you have to add a ranking, I'd prefer to something like skip/rent/buy long before a numeric system. Anything less than 80% is like kryptonite to a publisher anyway, so the gradient is same, regardless.

>= 9 is Buy, 8 is Rent, <= 7 is Skip

EDIT: ooo, Mex's "Wait for the price drop" is a good idea.

Will you set the number rating system out of 20, and then just toss darts at a dartboard please?

I thought I was against scores, but I realized upon some honest self-review ("7.5, looks like crap, the control mechanism is weird, but you can get hours of entertainment out of him.") that sites like metacritic did play a part in my consumption process.

What I'd like to suggest, though, is that any numbers given aren't allowed to exist in a vacuum. For every glowing 9.5, I want to see a list of other games that scored in that area and why this one is above or below them. Ditto for a list of games in that genre. Context is everything and so many reviewers think that just vomiting up the feature list from the marketing text is all they need to do. Not that I think GWJ reviews would have this problem, mind you; an island of rationality we've got here.

7inchsplit brought up a good point: Board Game Geek's rating system is quite helpful! I just wonder how much more database work that would entail.

Pages