Part Of The Conversation

Section: 

Something was bothering me yesterday as I trolled Metacritic for PSP game scores, trying to refresh my memory on what's actually worth buying for the system. I have the infinite wisdom of the GWJ community to draw from, but even our collective memory may fail under the strain of nearly two years worth of games. With this is mind, the oft derided aggregate sites we disdain for reducing the quality of a game to a mere number serves a valuable function. If I want to know if Metal Gear Solid Portable Ops is good, I can punch it in and see that most reviewers seem to like it. No spoilers, no outright bias or flaming, just a raw number denoting quality so I can feel safe making the order.

What bothered me was that these scores were missing something. Our input. Not one aggregate site includes the Gamers With Jobs perspective on games because we don't score them. I'm starting to wonder if we should.

We dislike the common review structure so much we call our articles "Perspectives" so that we can do our own thing. While it suits our own purposes, I wonder if we're doing a disservice to the wider audience who may benefit from our perspective but will never be exposed to it because we're unwilling to play ball.

Putting aside the old debate of assigning a number to an experience, there are some broader issues at play that concern me as someone running an independent gaming site. For starters, a number would mean we appear on aggregate sites next to Gamespot, IGN, 1Up and the dozens of others sites that score their games. People who look up a title might see "Gamers With Jobs" and read something they would never have been exposed to otherwise. While we primarily write for out own amusement and our community, I like to think that the internet at large could use more of what we're cooking here.

The second thing that springs to mind is access to games before release. Like it or not, calling your articles "Perspectives" and refusing to assign a score is a total turn-off for a lot of PR people when they're deciding who should get early review code. If you're not going to contribute to the Metacritic pool -- which can determine whether or not they get paid a bonus -- they may be less inclined to offer you coverage opportunities. There are hundreds of gaming sites out there and early code (or even final boxed software) is often limited. It's a tough gig wading through the countless fan sites vying for your attention. On the surface, we're a pretty enigmatic site to understand already.

We also have our writers to think of. As a creative cat herder, I'm more likely to get a review from writers if I can spare them the expense of buying the game themselves. We're blessed with all the great content we get from such talented people, but as paid opportunities come up and personal lives get busy or financially shaky, every little bit can help.

Is getting a Gamers With Jobs review on the podcast or the front page before the release date of a game something readers want, even if it means making our review content a little more in line with what most sites are doing? I don't think assigning a point value to a game suddenly means the content of the articles has to change, but the fear of people skipping to the end of a review is a real one. No one wants their hard work reduced to a final paragraph, a number and a few screen shots.

With all this in mind, I open the floor to you, dear reader. Would we be selling our souls by scoring games? Is there some rating system that both satisfies the intellectual and the more mainstream reader? Maybe we need our own internal aggregate with a final score that reflects the views of many on this site. Help me out!

Comments

Certis wrote:

Woah there, so it's a completely meaningless number if one person does it, but it's pretty cool if everyone can throw their own number into the pot? It's either meaningless or it's not. It either makes us a pandering, mediocre site or it doesn't. You can't have it both ways.

Ok, first off, numeric reviews are not meaningful. They're not meaningful for movies, they're not meaningful for books, they're not meaningful for video games. They're like movie ratings, PG, PG-13, R... in themselves, they're useless, because there's a huge array of invisible guidelines that goes into assigning a movie one of those magic values. They're further rendered meaningless because of variation in the audience. Some people who watch movies are fit to handle a racy PG-13, but might at the same time be disturbed by a violent PG-13.

I guess what I was trying to get across is that numeric ratings are kind of the same way. A number is only meaningful in the context of the reviewer's brain. Conversely, you can read a good reviewer's prose and understand where they're coming from. You learn which parts of the game they think are well executed, which parts are sloppy, and what they expect from the game. The number by itself lacks both the context of the reviewer's attitude and the context of the reader's preferences, so it's not only meaningless, it's meaningless in multiple dimensions.

For incontrovertible proof of this, I submit most major game review site's ratings for any game released in the past 3 years. The amount of inflation in review numbers just doesn't make sense. 2006 was not the best year of all time for gamers. And 2007 hasn't been that much better (yet).

On the other hand... a group opinion is something different. If you posit that the average GWJ reader/forum contributor is a level headed all-around great person who you'd buy a beer without thinking about it, then the attitude of those people in aggregate does have meaning. Sure, even in this community, not everyone likes the same thing, but I believe the group could reach a "recommend" or "avoid" consensus. It's like those rating sites that aggregate peoples' opinions about music (Pandora) or movies (Netflix), or everything (Amazon), and "rate" unseen things based on aggregate behavior. They're not perfect by any means, but in many cases, they are quite good. Amazon makes a lot of money because of its recommendations.

If you believe you've built a like-minded community of people who generally get along, random sampling of that community should be able to produce some sort of rating generally meaningful to people in that community. (How's that for a sentence full of weasel words!) See where I'm going with this? Incidentally, I think this is also why people tend to use game review aggregators--maybe you can't always trust one game reviewer, but in aggregate, you can at least get a thumbs up or thumbs down. Think how much more powerful an aggregator would be if you could weigh the opinions of specific reviewers you knew you generally agreed with, or certain sites whose community mirrored your likes & dislikes.

As for wallowing in the evils of capitalism, I didn't mean that to be insulting, just ridiculous. Obviously you're not going to send your reviewers out on EA press junkets to shake hands with pro hockey players the next time an NHL game comes out. Right? RIGHT? Remember that when you become good buddies with a few publishers, though, because they will throw buckets of money at you to fill your landing page with links, game logos, and obnoxious flash "experiences".

Don't change.

I propose the following mathematical model for calculating the average review score, given that we have

1) each user's proposed review score, ranging from -2 (terrible) to 0 (neutral) to 2 (great)
2) each user's ratio of correct spelling over the past 3 months
3) we already found the user with the highest spelling ratio from 2) and assigned his spelling ratio to a variable called, say, HSP.

It's almost like pseudocode really, just to get the principle across.

[quote]

// GIVEN VARIABLES

#define num_votes X

float scores[num_votes]; //array of member's scores
float spelling[num_votes]; //array of each member's correct spelling percentage over past 3 months

// END OF GIVEN VARIABLES

float sum(0); //sum of all votes
float psum(0); //sum of all persons
float result_score;

for (int i=0; i

I'm a dirty skimmer.

I do not support a rating system here. If I were forced to choose one it would be the 6 label system but not so much numbers in a rank style but more of an idea or how much someone needs to play the game.

- This game is awesome! Pre-Order or buy it this very instant! It's Ambrosia on a disc!
- It was pretty good. If I had to do it over I'd still buy it.
- Eh it was alright. Nothing special. Buy it if you are a collector otherwise rent it.
- Blah. I wouldn't buy this game but it's a possible rental if you are extremely bored. Better to borrow it for free.
- This horrible game forced me to wash my hands and flush my eyes out with saline it was so bad. My console would be seeking counseling right now if it were conscious from being violated with that filth.
- Burn every copy of this game immediately. Do not attempt to make contact on skin. This game is so toxic that even reading about it may induce violent vomiting, explosive diarrhea and flu like symptoms.

Essentially from Zero to Five stars is the scale. You can even change up the commentary whenever you want. That's part of the comical side of having a rating but really not having a rating. It's an impression of a rating for the naive or stupid.

Certis wrote:

Understand that we've been slashdotted, Penny Arcade Wanged and much more in the past six months. The site's traffic has nearly doubled since this time last year and everything is still going fine. Like the question of integrity, the question of idiots finding the site is not a concern for me because we're all pretty good at weeding them out.

The community was wanged? I didn't see that one. Is there a link, or, um, is it degrading?

Anyhow, when a title reaches the Front Page, I think that's a way to bring a game some sort of "rating".

Thirdly, when it comes to selling out? See the "selling out" thread- where do I sign up?

I usually don't contribute to much of the conversation around here being too much of a lurker and all. I don't think a scoring system is a bad thing for games. My problem is finding out if the reviewer has similar tastes as mine before taking the scores to heart. That to me is an impossible task and one I've found out the hard way too many times.

I personally could give a rat's behind if Halo3 scores a 10/10 because I know I will not enjoy the experience because I'm not a console FPS guy. I bought Halo based on reviews and tried to like it, I rented Halo2 and didn't even keep it through the rental period. I'm just not the guy that you're after with this style of game. Yet I loved Bioshock and I think that was because of the slower pace/atmosphere.

If there was some kind of bio of individual reviewers that could be included with each review at the least we would know the reviewer and be able to come to our own conclusions based on an aggregate of the score and the reviewer's slant. I think knowing the reviewer helps to align our own likes and dislikes of certain genres and gives a clearer picture into what the score means to that reviewer and ultimately the readers.

Certis wrote:

I find the running assumption that we would revert to some standard schlock review format by adding a score quite disheartening.

Your point is well taken, but at the same time I think it's important to observe that we're not just talking about tacking a number on to the end of our normal reviews. For when reviews sport bright numbers at their ends, and given the provocative, eye-catching effect those numbers invariably have on readers, even very talented writers find it hard to avoid writing in order to justify that number. The review becomes more like a defense of a number and less like what we here dub a Perspective. I think the latter is much more interesting.

Will Elysium get to review the JRPGs? If so, then I'm suddenly all for it. Also, you're not doing away with Perspectives but adding "Evaluations", right?

Just as an aside, I do not think any less of the reviews I read (on a variety of things, games included) at Ars Technica, and I think they take especially good care (most of the time) in choosing their reviewers, and ensuring their reviews bring some intangibles to the table that many other sites do not have. I see GwJ in the very same way. Reviews here bring that something extra to the table. The reviewers write what they feel, and they do not pander to the lowest common denominator. The implementation of a scoring system would make me think no less of GwJ, and I can imagine myself using it in about the same way as the Ars summaries and scores.

I love how there are many more elitist people on here that i originally expected. Onward they march, mantras ejaculating from their throats: "Our way is better! Why change what isn't broken?"

Anyone else find it funny that one of the purposes of GWJ was to found a place where people were enlightened and accepting. To do something different that wasn't found elsewhere. Adding a score to the end of a review won't make what is already done any less worthy of that sentiment. In fact it makes it better because it opens the reviews to a larger audience. Anyone who is scared of a number either fears change in general or is scared of their own talents in being able to write a review.
I don't understand why you'd write the review to fit the number. Surely the number comes from the review? (If the number is presented by the reviewer) It smacks of insecurities rising to the surface - people often like doing things as long as they're not graded and as soon as they are instantly become stressed or worse at their job.
I realise that writers are a fragile lot but it doesn't excuse the types of panic that i'm seeing in this thread...

Duoae wrote:

Onward they march, mantras ejaculating from their throats

Blech, the imagery is.. well..

Will Elysium get to review the JRPGs?

Not if I have to play them!

Though judging by some reviews I've read, that doesn't seem to be a requirement.

If you're going to do it, you might want to go back in time and retrospectively assign numbers to games you've already reviewed in order to provide a reference point and thereby ensure greater consistency to the numbers you assign. If you look at the early days of Gamespot, they were always giving out very high scores to games that often sucked, which hurt their credibility. I think it's only since they've become one of the bigger deals out there that they've added any honesty to their reviews, and it's still rare for them to completely trash a game from a big publisher, even if it sucks.

You will also need to develop some method for ensuring that reviews by different reviewers are all generally on the same scale, e.g. take some measure to ensure that Certis doesn't want to give Stroker 64 a 10 when Elysium only thinks it merits a 2. One possibility would be to maintain an easily accessible database of other scores that reviewer has given.

Moderating is for moderators. - Certis

Duoae wrote:

I love how there are many more elitist people on here that i originally expected. Onward they march, mantras ejaculating from their throats: "Our way is better! Why change what isn't broken?"

Anyone else find it funny that one of the purposes of GWJ was to found a place where people were enlightened and accepting. To do something different that wasn't found elsewhere. Adding a score to the end of a review won't make what is already done any less worthy of that sentiment. In fact it makes it better because it opens the reviews to a larger audience. Anyone who is scared of a number either fears change in general or is scared of their own talents in being able to write a review.
I don't understand why you'd write the review to fit the number. Surely the number comes from the review? (If the number is presented by the reviewer) It smacks of insecurities rising to the surface - people often like doing things as long as they're not graded and as soon as they are instantly become stressed or worse at their job.
I realise that writers are a fragile lot but it doesn't excuse the types of panic that i'm seeing in this thread... :)

You've been making posts like this more often lately and I'd like you to stop and consider your approach. I've bolded the parts that are especially useless, insulting, ad hominem and flame inducing. Adding an emoticon at the end does not forgive that sort of post, and following up any angry replies with "chill out bro" isn't ok either. Just so you know.

I didn't think it was that controversial tbh. I just think that some people are making more of a meal of this purely from the fact that they're resistant to change than real reason to worry about the addition.

Just read some of the responses here and then go to a website that's had itself a redesign and you'll see the same things. I really pity Certis and the rest when they release the new-look GWJ sometime in the future.

You've been making posts like this more often lately and I'd like you to stop and consider your approach. I've bolded the parts that are especially useless, insulting, ad hominem and flame inducing. Adding an emoticon at the end does not forgive that sort of post, and following up any angry replies with "chill out bro" isn't ok either. Just so you know.

I'll try and change my posting habits.

Edwin wrote:
Certis wrote:

I find the running assumption that we would revert to some standard schlock review format by adding a score quite disheartening.

Not the whole review, but what you said of users treating it like a schlock review. Missing the entire point of the site.

Certis wrote:

Their loss, in that case

I have every confidence that the reviews will be just as intelligent, unbiased and well-informed as the rest of the editorial content on the site. If new viewers come to the site and treat it like a schlock review than it is their loss. They wouldn't have come to the site anyway unless the review was there, and if the review didn't have a score, they wouldn't have it fully anyway.

I'm in a agreement with many of the other posters and think that this really couldn't do anything to hurt the site. The staff here do a wonderful job of moderating users and threads, so my concerns about the seamy underbelly of the internet making their way here and invading the site are pretty well covered. I'd personally love to see what the writers here can do with reviews.

Word of mouth is the best form of advertising.

Getting free games to GWJ won't help anyone. The decision to buy / not to buy will be made primarily by the community and probably before the review is posted.

You should only have review scores if you think you need 'proper' reviews.

Duoae wrote:

I really pity Certis and the rest when they release the new-look GWJ sometime in the future.

that was unnecessary. its just a number and i personally see nothing wrong with that. i think adding a GWJ score to metacritic is just a way for a small voice to do its part, which is all anyone can ask from anyone or anything. i mean, people DO look at metacritic scores, as certis testified in his original post... so why not try and make the score more representative of the truth by having another (much more accurate) voice be heard?

I'm with Bill on this one. For all their evils, review score aggregation works and drives traffic. Personally, I don't think that having the writers attach a score to their reviews would adversely effect the writing. People who are just looking for review scores aren't going to read them and people looking for quality writing and more nuance will.

If you are going to have scores, only review games which attract the right kind of people.

If adding a score to your reviews allows you more access and pre-release copies, I say go for it. Reading all these posts it seems the general fear is that the addition of a score to a game review will somehow destroy the sites integrity and user base. I don't think all the naysayers are giving the entire community enough credit.

I firmly agree that adding a score to a review is basically arbitrary, but if it is what difference does it really make if it has a score or not? GWJ isn't going to review the game any differently and we won't be influenced by that score either. Would it really change anything that's fundamental with GWJ?

If it were to happen I'm sure that at launch there will be a large influx of trolls and flamers but I don't see it continuing over the long term. If there's no one to feed them they'll eventually move on or go back to Gamefaqs.

GWJ wants to progress, so why shouldn't we let them try? Doing this won't ruin the site, it's not a major change of direction, just a minor addition really. We all know how expensive it can be to maintain this hobby and I'd dread having to shell out $60 out of my pocket just to buy a game for review.

I'll echo the suggestion to use a grade base rather than a numerical base score. If there's one thing I can't stand it's people arguing why one game gets a .2 higher score
than another.

I'm curious, for those of you concerned about stupid people finding the site, how did YOU discover the site? If you didn't come here when we first opened, there's a very good chance it was a link that brought you to the page. Are you afraid there are no more worthwhile folks left? Were we supposed to lock the gates once you were through?

I've just started working with a market research company that deals in 'predictive marketing'.

Basically, the company polls a large group of people - from all walks of life - to get their opinion on something. (Normally an advertising campaign or an idea for a new product.)

If the group says 'It's good', it's worth investigating. This is because the overall opinion of a large group of people is better at predicting success that the opinion of a small group of experts.

Just put a poll against every review for people to say whether they do / don't like the game. Only GWJ members with a certain post count will be eligible (not necessarily tagged).

EDIT:

Or that have been members for certain amount of time.

You'll have to go on trust that the people voting do have the game, but I reckon the kind of people that lie won't have a high enough post count to take part.

And I signed up because of Sway.

Evil Avatar, when you first launched the site. If I'm any indication then clearly you know what you need to block first.

GWJ is a community web site. I'm for a community score. Maybe only Office Linebackers or higher can enter a review score?

magnus wrote:

GWJ is a community web site. I'm for a community score. Maybe only Office Linebackers or higher can enter a review score?

I like this idea too. The only question would be what to use for the scale. I guess a 1-5 scale is pretty simple and straightforward. Personally, I always liked reviews that rated graphics, presentation, gameplay, story and replay value separately.

Having thought about it a bit more, 'a member for X amount of time' flag is probably better.

1. Discourages people posting in order to be able to add to poll (including marketing plants*).

2. Doesn't penalise long-time lurkers.

3. Gets lurkers to register.

* Unless they register six months ahead of the title being released, in which case they're in deep cover and probably undetectable anyway.

If we went with scoring, community involvement would be a must.