So, after an unexpected vacation, work continues. Sort of. A common request was for the ability to compare an outlet internally against itself to see how it reviews games on different platforms. I’m still working on a dynamic means to do this, and one which will fit a smooth curve to the data points. But in the meantime I took the Gamespot data and did the necessary database query for PS2, ‘Cube, Xbox, and PC and slapped them together manually into this animated GIF.

Is Gamespot “BIAS”, as the kids who haven’t learned the adjectivial form of the root word like to say? That you’ll have to decide for yourself. I can tell you the 23 scores in there that have my name attached to them are individually the one true score for each game they were given to.

But check out the weird distribution on Gamecube scores. The most common score is 6.5, with 19 games scoring 6.5. But, the next most common review score is a tie between 8.0 and 7.2, with 18 games scoring thusly. There’s no clear tip of the curve, but one thing to keep in mind is that there are a lot fewer datapoints for Gamecube, with only 442 reviews compared. The average score per platform comes out to be almost the same— in between 6.9 and 7.0— except for PC, where it’s a lot more profitable to publish crappy budgetware.

Another weird thing you might notice is how certain scores tend to not be represented in the expected curve. In particular notice how no matter which console platform you look at, it seems particularly difficult to score a 7.7 or a 7.8. There’s a noticeable dip at 7.4 too. PC seems more immune to that phenomenon. Like I said, weird.

Anyway, have fun comparing and contrasting each and every point on the scale, to prove whatever point you think needs making.

Hi there. The stuff you’ve been linked to probably involves the next two articles after this one. Please also enjoy these fine articles linked below, even though they have less interesting pictures and involve far fewer, if any, numbers.

—Matthew J. Gallant

• Your Guide To The Cherokee People (Look for “Your Guide To Ancient History” in the next issue of Computer Games Magazine)
• Videogames Are Art
• Alyx Is A Nerd Succubus
• Rockstar Cancels A Game
• Documentaries On Making Games: We Care
• Half-Life: 2, Episode: 1, Boats & Buggies: 0
• $600 PS3 Is Not A Toy; Therefore, $650 PSP Watch Is Not A Timepiece.

Well, it was significantly harder to grab IGN’s scores because of their tendency to throw interstitial ads at you. But it’s done. The histogram reveals IGN’s weirdo 20-point/100-point scale waffling. I’m tempted to round their scores to the nearest .5 and force them onto a 20-point scale.

Still, their most common score being an 8.0 is revealing. Not to mention there being almost as many 9.0’s as there are 6.0’s.

Now that I have two review outlets in the database, I’m going to spend more time working on the presentation of the site and the tools that will let users get more detailed information for one single site (like comparing score breakdowns by platform), as well as comparing between sites.

Related - Kotaku Doesn’t Read The Articles They Link

I spent the last day or so constructing the database and writing a script that would grab all of Gamespot’s reviews from their review list and massage the info into the relevant database tables.

Then I wrote the script that generates the image you see above. It’s a histogram of all Gamespot’s review scores for any game that isn’t for mobile phones, the Gameboy/Gameboy Color, the N-Gage, or the Neo-Geo Pocket. I’ll be expanding its functionality to make comparisons by platform as well as between two or more different sites. And I suppose I’ll have to make it look nicer, too.

I did this really quickly because I wanted to see if this was really worth doing. Out of the 7,244 reviews, 239 scored a 7.0. There’s some weird gaps, but you can definitely see the bell shape and how scores crowd towards the right of the scale. Now to figure out a way to determine which ranges of scores get translated to 1, 2, 3, 4, and 5 stars. It looks like that a 3-star game may go as high as 8.1 or 8.2.

Hopefully review scores from other sites will be as easy to compile as Gamespot’s were. I don’t know if any other sites have a list of all their reviews like that.

The site will be undergoing a radical transformation. It will no longer be a blog. There will be a different sense of humor to the site. I think it will be even more funny, just because of what it will do to both the game reviewing and game publishing industries.

The joke is this: lots of people love sites like GameRankings and MetaCritic. Publishers and fanboys alike use a game’s average score like a weapon. But really, these sites tend to not provide useful information about a game’s review scores. Why? For a couple of reasons.

Blame falls mostly on the reviewers themselves. Most reviewers tend to artificially inflate a game’s score. This has become known as the 7-9 scale, so named due to the industry’s stubbornness in holding onto a 100-point scale, and it’s further stubbornness in using any parts of it that don’t fall between 72 and 95. A famous “IGN 9.2″ tends to have a lot of noise in its signal.

Further fault, however, lies with GameRankings and its ilk themselves, though. They tend to be more than a little forgiving when it comes to who’s allowed to contribute to the “averaged score”. The criteria for GameRankings comes down to a site needing to have a lot of reviews and having a current output of a certain number of reviews a month. Very egalitarian, but there are way too many sites run by, shall we say, the overeager. There are 34 sites with an average review score of 80 or higher. I’ll rephrase that because it bears repeating: GameRankings has 34 sites— that it uses to generate a game’s metascore— for whom the average game scores an 80. Or higher— the creampuff champ is PSE Magazine, which has 817 reviews, the average score of which is 86.7.

Metafuture will improve the service that sites like GameRankings provide. Put a better way, we will create the service that those sites attempt to provide. First of all, we’ll cut down on the number of review outlets. There’s no need to have a huge number of reviews for a game to provide a useful aggregated score. Magazines and the top websites will be just fine.

Second comes the funny part. Games will still get an average score from all contributing reviews. But a site’s contribution to that average will depend on that site’s own individual normal curve— with the immediate left and right of the bell’s tip signifying three stars on a scale of one to five. Watch the drama as the biggest sweethearts see their 8.4 score for Gun and Car IV get pegged as three stars.

Continue to subscribe to this blog for status updates; the blog will only truly die when the new site is ready to launch, and compiling data is going to take a significant amount of time. If you’re an editor of a review outlet or otherwise have access to raw review data, please contact the site. I won’t blame you if you don’t, though.