Part Of The Conversation

Section: 

Something was bothering me yesterday as I trolled Metacritic for PSP game scores, trying to refresh my memory on what's actually worth buying for the system. I have the infinite wisdom of the GWJ community to draw from, but even our collective memory may fail under the strain of nearly two years worth of games. With this is mind, the oft derided aggregate sites we disdain for reducing the quality of a game to a mere number serves a valuable function. If I want to know if Metal Gear Solid Portable Ops is good, I can punch it in and see that most reviewers seem to like it. No spoilers, no outright bias or flaming, just a raw number denoting quality so I can feel safe making the order.

What bothered me was that these scores were missing something. Our input. Not one aggregate site includes the Gamers With Jobs perspective on games because we don't score them. I'm starting to wonder if we should.

We dislike the common review structure so much we call our articles "Perspectives" so that we can do our own thing. While it suits our own purposes, I wonder if we're doing a disservice to the wider audience who may benefit from our perspective but will never be exposed to it because we're unwilling to play ball.

Putting aside the old debate of assigning a number to an experience, there are some broader issues at play that concern me as someone running an independent gaming site. For starters, a number would mean we appear on aggregate sites next to Gamespot, IGN, 1Up and the dozens of others sites that score their games. People who look up a title might see "Gamers With Jobs" and read something they would never have been exposed to otherwise. While we primarily write for out own amusement and our community, I like to think that the internet at large could use more of what we're cooking here.

The second thing that springs to mind is access to games before release. Like it or not, calling your articles "Perspectives" and refusing to assign a score is a total turn-off for a lot of PR people when they're deciding who should get early review code. If you're not going to contribute to the Metacritic pool -- which can determine whether or not they get paid a bonus -- they may be less inclined to offer you coverage opportunities. There are hundreds of gaming sites out there and early code (or even final boxed software) is often limited. It's a tough gig wading through the countless fan sites vying for your attention. On the surface, we're a pretty enigmatic site to understand already.

We also have our writers to think of. As a creative cat herder, I'm more likely to get a review from writers if I can spare them the expense of buying the game themselves. We're blessed with all the great content we get from such talented people, but as paid opportunities come up and personal lives get busy or financially shaky, every little bit can help.

Is getting a Gamers With Jobs review on the podcast or the front page before the release date of a game something readers want, even if it means making our review content a little more in line with what most sites are doing? I don't think assigning a point value to a game suddenly means the content of the articles has to change, but the fear of people skipping to the end of a review is a real one. No one wants their hard work reduced to a final paragraph, a number and a few screen shots.

With all this in mind, I open the floor to you, dear reader. Would we be selling our souls by scoring games? Is there some rating system that both satisfies the intellectual and the more mainstream reader? Maybe we need our own internal aggregate with a final score that reflects the views of many on this site. Help me out!

Comments

KillerTomato wrote:

Those are the things that readers need to know in order to reach a good decision about how applicable the review is going to be to their own experience, and very rarely are they mentioned in a professional review.

This is why I've always felt the community here provides me a better sense of the value of a game than any reviewer. The Goodjer Hivemind can more often than not nail the value of a game. Even to the extent that you can find *someone* here that looks almost any game, including Bratz Chernobyl. Like I know that if I want some opinions on Strat RPGs there are people here who like them and can give me honest opinions on which one to choose if I could only choose one. Etc.

I think I understand this thus far:

Metacritic and their ilk are, by design, basic systems. A perspective is, by eloquence, far more granular. In order to be recognised, you need to score. To be GWJ, you need to write perspectives.

The holy grail of a GWJ review score would be to translate the sentence:

It is like playing lair on valium whilst being beaten by a japanese kendo master wielding some two by four. After that, I hit the S&M club for some light relief.

Into:

The gameplay is poor.

To contribute to the metapool, your choices are limited:
- Ask your reviewer to submit a score, or;
- Ask that the reviewer provide sufficient information in a form to be able to divine a score.

Conventional media opts for the former (for reasons and results already well stated in this thread).
The latter leads to much more interesting future. GWJ is able make the latter a possibility, and with support, stop it being a short-lived one.

KillerTomato wrote:

I join those without a burning iron in this fire. My only complaint against the current setup is that we don't see enough of the great reviews that the front-pagers have proven their ability to write. If adding scores would increase their frequency, then more power to it.

This is a point that needs to be made more. If the addition of scores and the increased attention it brings results in more reviews, that's more intelligent, thoughtful content that we wouldn't have gotten otherwise - resulting in the exact opposite of the "dumbing down" effect that some of the panic reactions in this thread are suggesting.

The current problem with that is that there is no mandate for writers to write perspectives/reviews right now. While I have regularly beaten the bushes to make sure our writers stay on-point for a gaming site (with occasional success), I have never said "we need X perspectives out of you per month". The reason for this, as Rabbit's thrown down gauntlet suggests above, is because virtually no one likes writing reviews. In an environment where we open the door to limitless opinion making, reviews are kind of stale meat, and so we really wait until a game compels us toward a thought. I think that's where part of the quality comes from, is that we're waiting until we have something meaningful to say about a game, and I don't think that decision would be altered by adding scores.

How about this?

Have a scoreless anti-review and a community poll grade for each of the key factors of a game. (graphics, gameplay, story, design, etc.)

And use grades instead of numbers. The poll grades must be whole so that the average cant be revealed in +/-

For instance, graphics gets 5 A's, 4 B's and 1 C from the community poll. The final score for graphics could be an A- or B+

If you are going to have scores, have one score along some scale of value. If there is one thing I like less than scores, it's scores that are a vector of numbers for various arbitrary "components" of a game's performance.

Also, I think this is wise:

Elysium wrote:

I think that's where part of the quality comes from, is that we're waiting until we have something meaningful to say about a game, and I don't think that decision would be altered by adding scores.

Here's a tongue-in-cheek scoring system that I proposed on my humble blog: the number of days you can play the game before you hate it and sell it on Ebay. From one to five. If a game makes five it means it's generally worth playing through and maybe writing a longer article about.

I also agree with Rabbit that writing reviews of games you hate is a tough situation unless there is money on the table for the time and effort needed to do so. Why use your hobby time to finish a game you hate and then use your writing hobby time to write about it?

Elysium wrote:

While I have regularly beaten the bushes to make sure our writers stay on-point for a gaming site (with occasional success), I have never said "we need X perspectives out of you per month".

To remain listed in Gamerankings that would pretty much have to happen. I forget the exact number but it was something along the lines of three or four reviews need to be posted each month or the site isn't listed.

I love writing reviews. Just saying.

I have a question with the idea of a "review" anyway. Rabbit mentions not being willing to write a review because it requires him to finish a game before he can call it a review. I'm not I agree with this definition of a review. Movie critics walk out of movies that are bad enough to not finish. Food critics neither have to to clean their plates, nor order the entire menu to review a restaurant. Why do game reviewers have to finish a game to review it? I understand that some time needs to be spent on a title, but if the game stops being fun, that is a valid concern, the review can reflect that fact easily enough.

Atras wrote:

Movie critics walk out of movies that are bad enough to not finish.

No movie critic I would read does that.

I don't agree with the presumption that a review proper would need to be written just to support a score. A review, by nature, describes and evaluates a game, primarily for potential players/purchasers. If it doesn't it isn't a review. If it does, than the inclusion of a numerical score shouldn't be problematic. The written and numerical elements should compliment each other. I don't see it as some sort of tug-of-war between the two.

I like writing reviews and I think most of the reviews I've written here and elsewhere would support a numerical score, were I to tack one on (like rabbit, I've never scored a review). I also enjoy writing articles about games that are too narrowly (or broadly) focused to be called reviews--articles that could never support or explain a score. I think there's room for both kinds of writing about games at GWJ and elsewhere.

Like rabbit, the games I write about are the games I've selected to play. But I don't mind slogging through a mediocre or even lousy gaming experience once in a while in order to write about it later. I can be passionate about bad games, too.

Does putting a score on a preview or review enhance skimming as a practice instead of forcing the reader to develop their own rank in their head from reading the full article?

I think it might.

What if...

- Writer does his fancy word thing (that we all love so)
- Community gives a score. By voting (simple number or letter multiple choice form)

This way the integrity of the writer is not in jepordy. The "score" is a overall viewpoint and may or may not be complemntary to what the writer is saying.

Basically a GWJ community report card. It would have to be clear that the score is not given by the writer.

The short version: It's your ball. Take it and go home if you want to.

Longer version: Most potential disadvantages fall squarely on the shoulders of the administrators (moderating, forcing reviews, etc.) If those are challenges you are willing to face to increase publicity for the site, more power to you.

I for one, couldn't possibly care less if you put a number at the bottom of a review. I am as free to ignore it as you are to place it. I look to the prose and the ensuing discussion for guidance. That is why I have lurked here for a while. If that number brings more worthy folks to the discussion, even better.

Shall we name it: GWJ 3.0 - The search for more money?

If you really think we run this site for the money, you are sadly misguided and we are incredibly inept. I still work 50+ hour weeks aside from running the site.

I've only been coming to GWJ for a month or so now, but I wanted to chime in and agree with those saying that someone shouldn't have to finish a game to write something about it.

I know one thing I dislike about other review sites is that I have a hard time getting a sense of who the reviewer is; if someone doing a review of a game starts off with " I couldn't bring myself to finish this " a couple of times, and then does a review starting with something like " I played this one and finished it in a week ", that gives me an idea of who this person is, and it means more to me if/when they do finish a game.

I don't mind numbers as part of a review, but I do want to know about the person doing the review so I can decide if I generally agree with them or not. I'm not so sure about community reviews, as I'm always wondering what the bias of the masses is.
Is that Lego game getting a good community score because of people who really love the formula?
Is that new OutRun getting panned by people who Gran Turismo and think everything else is crud?
If there's going to be a community score, I would like to have one from the individual reviewer as well; again, getting to know that reviewer = knowing if I agree with them much or not at all.

For anyone who doesn't think that ratings will effect the quality of discourse too much should check out our review of Halo 3. Ratings are a bit of curse. Yes, they generate more views (depending on the review/rating) and interest, but they will also attract the worst kinds of fanboys and immature posters. Also, while a majority of the posts were flames, supporters were also very vocal and offered nice words of encouragement. Its up to the GWJ staff if they want to deal with that kind of heat.

Certis wrote:

If you really think we run this site for the money, you are sadly misguided and we are incredibly inept. I still work 50+ hour weeks aside from running the site.

Not at all. Simply an apparently inept attempt at humor. My wife tells me at least weekly, "You're not funny." She is a wise woman.

If you were in it for the money, there would be a lot more ads I imagine. If you can find a way to make it enormously profitable, good for you. If that is even on your radar, I imagine increased page traffic to be step one. I have never gotten the impression that this site is about money. Well, except for the donation drive, and I completely understand bandwidth costs. . .

My real point was that the onus would be completely on you guys. If a publisher hooks you up with a free pre-release copy of a game, I should think they would expect a review. Make sure you, or Rabbit, or E, or whomever has time and inclination that day/week. I know my schedule would never allow it.

Good point. I'm trying to imagine the nuanced opinions we share on Halo 3 getting translated into a number. If you feel like the campaign is too short, the Flood sequence too long and think the MP is brought down by the pubbies (an opinion many of us here share) then Halo 3 would be like a 7 for you. A 7 would become total flamebait on the interwebs, subjecting the site to all kinds of ridicule from fanboys. It would be a page 1 article on xbox360fanboy.com and would generate a hundred threads on gamefaqs.

On the other hand, if you feel like MP Halo 3 with Goodjers is a grand time, then maybe the game is an 11 out of 10. It has a fun campaign, especially coop with friends. It has great MP, especially with some of the people we have coming up with creative ways to Zombie Skate.

I think it would be difficult to put an accurate number on the worth of Halo 3, given that a lot of the value depends on how much you enjoy what playing with friends (especially friends here) brings to the game.

rabbit wrote:

I'm going to self-select to only write about games that I either enjoy (Guild Wars), or that have some deeper reason for playing (Super Columbine Massacre RPG). I chose those two examples for a reason. In both cases, I personally enjoyed the process of writing them, and I am reasonably happy with the outcomes. In neither case can I even imagine putting a number at the end of them. I'm curious what numbers you all might have thought appropriate.

Maybe for GW, but I can't begin to imagine putting a score on Super Columbine. I'd feel wrong putting a dollar amount on it, but I might be able to manage.

For the record:

  • I like lists
  • I vote for dollar amounts if we add scores.
  • I have Billy Joel's "Just the Way You Are" stuck in my head.
  • I don't know how we'd set the dollar scores to fit MetaCritic. Would we cap value at standard AAA price for the platform? What about Guitar Hero and the like?
  • Community voting sounds great, but wouldn't get meaningful scores from enough goodjers at release time. Would this really mean anything if our scores came out a couple weeks after launch instead of a couple weeks beforehand?

I'm skimming filthily because this thread is long and work is busy. But I felt I should say something.

The value of GWJ comes not from any one opinion, but from the way posters with a wide variety of tastes dissect games in an articulate and coherent way. There are some scores that would rate 95% on a review site that I won't like, and some games that would rate 80% that somehow grab hold of me. Trying to reduce the banter to a score loses that value.

That said, the Supreme Overlord of Goodgezan (TM) are the final say here. I agree that it's a shame many gamers will never know of GWJ.

Perhaps instead of trying to come up with a score, the goal should be to develop a soundbite. A short phrase that summarizes the experience. Kind of like having a tagging thread. Not a number, where all the detail is lost, but at least something that a meta-site can pick up. Then the reader can come here to read more.

If that doesn't work, you could always start coming up with scores. But maybe this step would be enough.

I feel like this really is a decision to be made by the front page writers. I think that GWJ's writers have enough talent and integrity to not allow the introduction of scored reviews to compromise the quality of the writing, if they decide they're up for it.

DevilStick wrote:

Perhaps instead of trying to come up with a score, the goal should be to develop a soundbite

I see your sound byte and raise you a Haiku. I'd be all about Haiku reviews. They do them for movies, why not games.

And my point about Guild Wars wasn't that it was un-scorable, but that the review I ended up writing didn't leave room for a score. It was totally self indulgent to write a script, and turn what was ostensibly a review into a diatribe about cutscenes, but it was what turned me on at the moment I sat down to write it. I can't even conceive of then saying "I give it an 8" at the end.

This week's EGM Live podcast has the crew talking about their strangest and/or worst encounters with the PR of game companies. Maybe we should be careful what we wish for. Here's the link.

Except we've been working with all the major publishers since we launched the site, that's nothing new

rabbit wrote:
DevilStick wrote:

Perhaps instead of trying to come up with a score, the goal should be to develop a soundbite

I see your sound byte and raise you a Haiku. I'd be all about Haiku reviews. They do them for movies, why not games.

As long as the Haiku makes some sort of sense. Most of the Haiku poems on that website you linked are kind of ... dribble ambiguous crap. Though a few were spot on. Eh *shrug*. I'm not an English major nor a professional poet. Maybe they're good... and maybe Lindsey Lohan will see the error of her ways, travel to India and become Hindu. I rather have a dollar amount system or simple commentary implying good or bad. I like snarky.

Actually and now that I think about it, adding "I give it a 9" to the end of your Columbine review would be hilarious.

wordsmythe wrote:

Actually and now that I think about it, adding "I give it a 9" to the end of your Columbine review would be hilarious.

QFT

Also forgot to mention that not all reviews on Metacritic have ratings (like the NY Times for example). If the review doesn't have ratings, the editors at Metacritic will guesstimate a rating for you. So the whole "perspectives" vs "review" labeling may have more to do with the exclusion than not having ratings.

Is it possible to do reviews and perspectives as entirely separate things? With a different set of writers for each as necessary? I just ask because it seems like the whole Perspectives thing isn't really about reviewing a game, most of the time they're about other interesting topics that are just called forth by a particular game. Usually these tangents are more entertaining than the game discussion anyway. Seems logical to just break them apart.