Part Of The Conversation

Something was bothering me yesterday as I trolled Metacritic for PSP game scores, trying to refresh my memory on what's actually worth buying for the system. I have the infinite wisdom of the GWJ community to draw from, but even our collective memory may fail under the strain of nearly two years worth of games. With this is mind, the oft derided aggregate sites we disdain for reducing the quality of a game to a mere number serves a valuable function. If I want to know if Metal Gear Solid Portable Ops is good, I can punch it in and see that most reviewers seem to like it. No spoilers, no outright bias or flaming, just a raw number denoting quality so I can feel safe making the order.

What bothered me was that these scores were missing something. Our input. Not one aggregate site includes the Gamers With Jobs perspective on games because we don't score them. I'm starting to wonder if we should.

We dislike the common review structure so much we call our articles "Perspectives" so that we can do our own thing. While it suits our own purposes, I wonder if we're doing a disservice to the wider audience who may benefit from our perspective but will never be exposed to it because we're unwilling to play ball.

Putting aside the old debate of assigning a number to an experience, there are some broader issues at play that concern me as someone running an independent gaming site. For starters, a number would mean we appear on aggregate sites next to Gamespot, IGN, 1Up and the dozens of others sites that score their games. People who look up a title might see "Gamers With Jobs" and read something they would never have been exposed to otherwise. While we primarily write for out own amusement and our community, I like to think that the internet at large could use more of what we're cooking here.

The second thing that springs to mind is access to games before release. Like it or not, calling your articles "Perspectives" and refusing to assign a score is a total turn-off for a lot of PR people when they're deciding who should get early review code. If you're not going to contribute to the Metacritic pool -- which can determine whether or not they get paid a bonus -- they may be less inclined to offer you coverage opportunities. There are hundreds of gaming sites out there and early code (or even final boxed software) is often limited. It's a tough gig wading through the countless fan sites vying for your attention. On the surface, we're a pretty enigmatic site to understand already.

We also have our writers to think of. As a creative cat herder, I'm more likely to get a review from writers if I can spare them the expense of buying the game themselves. We're blessed with all the great content we get from such talented people, but as paid opportunities come up and personal lives get busy or financially shaky, every little bit can help.

Is getting a Gamers With Jobs review on the podcast or the front page before the release date of a game something readers want, even if it means making our review content a little more in line with what most sites are doing? I don't think assigning a point value to a game suddenly means the content of the articles has to change, but the fear of people skipping to the end of a review is a real one. No one wants their hard work reduced to a final paragraph, a number and a few screen shots.

With all this in mind, I open the floor to you, dear reader. Would we be selling our souls by scoring games? Is there some rating system that both satisfies the intellectual and the more mainstream reader? Maybe we need our own internal aggregate with a final score that reflects the views of many on this site. Help me out!

Comments

Reaper81 wrote:

what the gaming community needs isn't another corporate sphincter leaking the usual fecal matter but rather that it needs a real voice for mature gamers.

Is that a Halo 3 spoiler?

Part of me doesn't like scoring systems at all, for many of the reasons previously stated. And yet I turn to them to help me at least understand what the zeitgeist is about the current hot titles. Yep, i'm a hypocrite. I think the GWJ thoughts/opinion should be included in such aggregate sites, because there's a point of view here that should be heard outside the site itself.

Buy
Buy at discount price
Rent
Don't even bother

That's how it should be.

How about this for a rubric, with the rubric popping up as a comment when you mouse over the number (again like Boardgamegeek):

5 - Every gamer should buy this game.
4 - Every fan of this genre should buy this game.
3 - A solid and enjoyable rental.
2 - Pass.
1 - Avoid at all costs. Unplayable.

No decimals, no sub-category ratings, etc.

Edit: Edwin's is good, too.

I think any GWJ rubric should be based on current GWJ memes, or possibly prime numbers. That way, the ever elusive "perfect" scores is either an indecipherable in-joke or impossible, because 100 is a power of 2. Either way, the riff-raff can't stumble in and start pissing everywhere.

This site is absolutely freaking amazing.

Adding a numeric or symbolic score to some of the reviews will not take away from that (Especially if you do not FORCE reviewers to add a score should they not want to).

The typical 12 year old forum troll will not find a home at GWJ. The community here (not to mention the moderators) does not encourage or play into it.

As long as GWJ continues to have some of the best, most insightful, well-written articles you can find anywhere on the internet, it will be a haven for those of us who don't want to read the same rehashed story on 8 different "news" sites.

I was fortunate enough to find GWJ through a PA link to the Kevin Levine/Bioshock article. I think that we should take every opportunity to try and draw more people here.

The vocabulary alone will drive off 90% of the undesirables.

Certis wrote:

People who look up a title might see "Gamers With Jobs" and read something they would never have been exposed to otherwise. While we primarily write for out own amusement and our community, I like to think that the internet at large could use more of what we're cooking here.

To me, that is the primary point of interest. I'm not a fan of score-based review, but the possibility of GWJ attracting a wider audience can do very little harm to the gaming community. With that in mind, along with the acknowledgment that score-based review is just a necessary evil to get us to that end, I support the notion of putting scores on our reviews, but suggest that we keep the scale simple. Something like a 1-5 scale without decimals will do just fine.

5. I highly recommend this game.
4. I like this game.
3. I don't feel anything in particular for this game. A perfect rental.
2. I am dissapointed by this game.
1. Playing this game was a waste of my time and energy.

Furthermore, emphasize that the reviews are highly opinionated. I wish that the reviewers will retain their personal perspectives, with varying importance placed on specific aspects of games that may not occur to the greater part of the player base. That will explain the "I" and "my" next to all the scores above. The review score is strictly a representation of personal opinion.

Danjo Olivaw wrote:

Reviews are inherently subjective, and a review score is just dressing it up to look objective.

Really? I've always thought of numerical scores as just as opinionated and subjective as any written review. Even more so, because they don't mince around with intangibles--a score is more definitive (albeit more contrived) than a nuanced discussion of a game's attributes. I always felt I could be much more objective in a written review than I could be if I had to simply assign a score.

I think scores do lend a slightly more authoritative air to reviews, in that they diminish the voice of the author, in favor of the voice of the publication. People are far more likely to say "Eurogamer gave Bioshock a 10/10" than "Kristan Reed gave Bioshock a 10/10." Something else to consider.

I've never been big on penning reviews anyway, but for what it's worth, I vote no. This is mainly because I do not know how to give games numbers, or letters, or thumbs. I know how to devote words to them.

I have no problem with it as long as a rubric is set up and followed. Edwin's seems the most in line with how we talk about games anyways, and the topic could also serve as our "Catch-all" that pops up for virtually every high-profile release.

For me personally, I think the interests of the site and what makes it unique are best served by not having scores. Aside from lumping our reviews in with a lot of the other dribble on the Internet (that is subjective of course), my biggest problem with scoring is the skewed scales that are used and how the public at large interprets those scales. On most sites, anything less than a 7 or an 8 is usually assumed to mean the game is no good and anything less than a 6 or a 5 usually means the game is so bad, the review was written almost for comedic value. Even the few publications that try to use a truly level scale like EGM (and I still question how level their scale us based on their implementations of it) have a hard time convincing their readers that a 5.0 means decent and not laughable. If Gamers With Jobs were to implement a scoring system, I've no doubt it would be completely fair and balanced but our problem is convincing the new readers from Metacritic of that. The reviews the writers here create are very intelligent and well-written, a cut above most other web sites. If the visitors who come in from the outside can't be bothered to look at more than the score, honestly I say then let them go back to IGN. I know that sounds elitist but I think this site operates at a higher standard. Obviously, that's not necessarily the best way to drive new traffic here but I think it's the best way to maintain the strongest level of integrity, both actual and perceived.

Thirteenth wrote:
Certis wrote:

People who look up a title might see "Gamers With Jobs" and read something they would never have been exposed to otherwise. While we primarily write for out own amusement and our community, I like to think that the internet at large could use more of what we're cooking here.

To me, that is the primary point of interest. I'm not a fan of score-based review, but the possibility of GWJ attracting a wider audience can do very little harm to the gaming community. With that in mind, along with the acknowledgment that score-based review is just a necessary evil to get us to that end, I support the notion of putting scores on our reviews, but suggest that we keep the scale simple.

I agree 100% that this is the sticking point of interest here.

The thing is, GWJ is quite honestly a lonely island of greatness in a sea of game-community filth. Even respectable sites like 1up with quality podcasts and legitimate, interesting reviews and content have forums that are absolutely un-readable.
I agree that the internet at large could use a little bit more of what we're cooking here. As someone who managed to swim from the filth to the GWJ island, I almost have the urge to go out and rescue more. GWJ seems to have grown into a community that's stable and well moderated enough to survive any kind of trouble caused by an influx of people (and strong, unapologetic, uncompromising moderation really is the key) and the inherent requirement that you must put thought into your posts before posting here should be enough to keep out most of the undesireables.

I think GWJ can and should aspire to be an important voice in the gaming community. We can do so much good by providing qualities that are sorely lacking in our hobby of choice and are often a sticking point of criticism: those being maturity, reasonable discourse, intelligence, acceptance, and values. How much better would the gaming community be if more gamers posted, talked, and acted with these values in mind? Can GWJ alone bring about some kind of rampant change to the xbox live community like technicolor spreading across Alice in Wonderland? Of course not. But perhaps we can bring to the public eye a voice that says "You know what? We can do better as a community, and we shouldn't have to tolerate mediocraty" which may just be an extra stone in the path to legitimizing gaming culture as a mature, intelligent hobby. Wouldn't you want to be part of that? I know I would.

Alright, rose-coloured goggles off. I know something as simple as adding review scores won't emancipate throngs of frustrated gamers from the shackles of mediocre gaming sites and cause some kind of gaming zeitgeist, but I don't see how it can hurt to try and be part of the dialogue, and the solution.

As for "review scores are garbage and mean nothing" argument, well, I'm inclined to agree. For a game I actually care about I'll be sure to delve through the text to figure out the reasoning behind the score. But I don't see us losing any actual integrity by adding scores so long as the reviewers aren't trying to
-pander to a certain audience
-be controversial for the sake of being controversial
-etc, Ie, Trying to gear their review towards attracting traffic.

Other cool features that could go along with the review:
-"Tagged" goodjers could also vote in a sort of "community review" to go alongside the reviewer: I don't think it's far fetched to think that those who have been in the community that long would have the good sense to only "vote" on games that they've actually played enough to have been able to form an opinion on. It might be a fun way to have a "reader review" that might actually mean something to the community, being that we will know it comes from respected community members.
-The reviewer could choose 3 community members (Say, a few who were very active and posted intelligently in the pre-release "catch-all" thread with varying attitudes and perspectives) to contribute mini paragraph long side reviews to go along with the actual review, and maybe form some sort of aggregate score with that of the reviewer.

Just ideas. To me, since the greatness of GWJ is due in large part to the unparalleled quality of the community, it only makes sense to me to try and include it/us in the review process somehow.

How do those that would be writing and scoring said reviews feel about this?

If you go with scores, do grade ratings. You don't have to have a key that no one will read if you do grades, because everyone knows what grades mean.

Interesting points. I don't really have anything to add that hasn't been said. There are some very real positives for "playing ball" as you say and I wouldn't personally be turned off by putting scores on the reviews.

Is getting a Gamers With Jobs review on the podcast or the front page before the release date of a game something readers want, even if it means making our review content a little more in line with what most sites are doing? I don't think assigning a point value to a game suddenly means the content of the articles has to change, but the fear of people skipping to the end of a review is a real one. No one wants their hard work reduced to a final paragraph, a number and a few screen shots.

If I wanted the same thing other sites spew I would go to those sites. I thought that's why this site was created in the first place, to be different and better.

I find the running assumption that we would revert to some standard schlock review format by adding a score quite disheartening.

Certis wrote:

I find the running assumption that we would revert to some standard schlock review format by adding a score quite disheartening.

I think ever game should have a 3-Dimensional graph plotting the relationship between graphics, stability, and fun. You could even add a 4th dimension for intangible qualities (defined by the reviewer for each game).

Either that or have a GWJ metareview value. Open up the review system to votes by all of GwJ, and then weigh the vote based on a member's time active on the board, number of posts, visits per month, and past voting quality (to prevent certain rabid crowds from really having that much of an effect). After a week or so, GwJ will reveal the true ranking of a game to all the internet. WE will become THE voice of gaming.

Aside from these two ideas, I'm fine with whatever it takes to keep GwJ relevant in the gaming community. I think the more weight the writers here carry in the industry, the better it is for all of us. I have faith that GwJ as a community will come up with a good answer for handling the bad that may crop up as a result of this.

EDIT: Well, that'll teach me to respond without reading. Dysplastic pretty much said everything I thought of, but much more eloquently. Of course, he left out the cool 4-D graphs....

Certis wrote:

I find the running assumption that we would revert to some standard schlock review format by adding a score quite disheartening.

Not the whole review, but what you said of users treating it like a schlock review. Missing the entire point of the site.

Their loss, in that case

That's my point.

Edwin wrote:
Is getting a Gamers With Jobs review on the podcast or the front page before the release date of a game something readers want, even if it means making our review content a little more in line with what most sites are doing? I don't think assigning a point value to a game suddenly means the content of the articles has to change, but the fear of people skipping to the end of a review is a real one. No one wants their hard work reduced to a final paragraph, a number and a few screen shots.

If I wanted the same thing other sites spew I would go to those sites. I thought that's why this site was created in the first place, to be different and better.

I simply don't understand this reasoning that putting a number at the end of a review suddenly causes writers to "fear" that people won't read their article. You can't FORCE people to use things they way you want them to... if people just want to look at the score, then that's what they'll do. If they want to read the article, they WILL.

Maybe you can make two versions of the site... one for the elites with no scores, and one for the Neanderthal-like plebes who are (obviously) the only ones who'd dare to read a review with a score attached.

Certis wrote:

Their loss, in that case ;)

No, not their loss. Your loss for pandering to them. Your reviews are not improved by adding a meaningless number (or set of numbers) to the end of them. Their impact is lessened, because you simply can't mathematically sum up a block of prose which weighs strength against weakness. Every reader is interested in different aspects of different games to different degrees, and a one-size-fits-all score does a disservice to all of them. The only absolutes that are meaningful are "skip this game at all costs" and "this is game of the year material." Those sentiments are best communicated in the title of the review anyway.

If you're going to be mediocre, go all out. Keep your current reviews, and at the same time add number-based reviews that gloss over games like every other mediocre site out there. Accept bribes and exclusive advertising deals from publishers. Wallow in the free market. (And hehe... just as I hit "Post" I glance up and see a similar idea in the previous post :D)

Actually I do kind of like the idea of having a community score be the thing published to review aggregators, but that sort of thing is awfully prone to manipulation. Maybe if it worked something like slashdot's moderation system, where a random subset of goodjers were given rating points to apply to a current set of games.

Better still, maybe you could work with the editors of the sites in question, like metacritic/gamerankings, gametab, rottentomatoes, and ask them to please read your reviews and weigh them in whatever scale seems reasonable. RT in particular seems to be able to do that with unscored movie reviews.

A blunt post, I can appreciate that!

No, not their loss. Your loss for pandering to them. Your reviews are not improved by adding a meaningless number (or set of numbers) to the end of them. Their impact is lessened, because you simply can't mathematically sum up a block of prose which weighs strength against weakness.

If the number is meaningless, how could it impact the content of the review one way or the other? What exactly is it about a score that would lessen the impact for someone who would just as soon gloss over the number anyways? You can't argue that a score is meaningless AND claim that the article is impacted by it.

If you're going to be mediocre, go all out. Keep your current reviews, and at the same time add number-based reviews that gloss over games like every other mediocre site out there. Accept bribes and exclusive advertising deals from publishers. Wallow in the free market. (And hehe... just as I hit "Post" I glance up and see a similar idea in the previous post :))

This kind of approach isn't even on the table and for all the work we do to run a site with integrity, the suggestion that we'd just throw it all out the window is insulting. Like somehow adding a score makes us a "mediocre" site all of a sudden. Give me a break.

Actually I do kind of like the idea of having a community score be the thing published to review aggregators, but that sort of thing is awfully prone to manipulation. Maybe if it worked something like slashdot's moderation system, where a random subset of goodjers were given rating points to apply to a current set of games.

Woah there, so it's a completely meaningless number if one person does it, but it's pretty cool if everyone can throw their own number into the pot? It's either meaningless or it's not. It either makes us a pandering, mediocre site or it doesn't. You can't have it both ways.

I say go for it... as long as you're prepared to take all of the bullsh*t that comes with it. Having masses of fanboys sending you hate mail will give you a special feeling on the inside.

Reaper81 wrote:

What hasn't changed is the fact that we are still the best site on the internet for those that love games and are interested in having something resembling normal, humane, and decent discourse about games and gaming.

f u n00b

I think you should do reviews that go all the way to eleven. Not only because it would be awesome, but also because i'd like to see how metacritic and gamerankings normalized the scores.

I really like the idea of a Bayesian-weighted average score of the readers -- but to limit gaming of the system itself, maybe limit the voting to people who've been here awhile?

I find BGG's system incredibly useful, and I'd love to see the average scores of this particular community.

Edwin wrote:

Buy
Buy at discount price
Rent
Don't even bother

That's how it should be.

Except for the fact that it's even more meaningless than a 10 point scale. Everyone has their own thresholds for what a "buy" versus a "rent" is, some people buy more games and others buy fewer and rent more. Taking the step from saying "this is how good this game is" into the realm of "this is what you should do with your money" is not an improvement, it's an obfuscation.

"Buy" and "Don't buy" works for the very best and very worst games, and then completely falls apart in the middle. You end up attaching "rent" to a game (like a run-of-the-mill Atlus PS2 RPG) that may easily be a "buy" or "buy at discount" for fans of the genre and an "avoid" for people who aren't the hardcore genre fans - and so you've attached a "score" to a game that doesn't apply to anyone!

I know that a 70% in a genre I love may still be a game I want to buy, and the same score in a genre that I may enjoy at times but don't pursue deeply is a game that I probably won't want. You could argue that we could interpret the "rent" score in the same manner, but that just proves my point: it's not an improvement, it's an obfuscation.

I've been following the site and the podcast for some time, had to register to post on this.

I really don't see how you would lose if you assigned, say, a five-tier star rating at the end of perspectives. It's abstract enough to not devolve the conversation into meaningless bickering about the "score", yet it gives you that numerical value the PR reps are looking for, and I do think it makes things easier for the reader.

I don't assign scores to my assessments of games in my blog, simply because I tend to cover them from some very specific angle and describe my personal journey with the game and not think about other gamers much at all. But when I review stuff for a magazine, I actually appreciate giving the 1-5 stars - it helps to tie together the text, which must often be cut down to a very short length.