WTF is... The Last Federation

[BobTheJanitor]

Sure, different issues, but I was trying to avoid posting an entire dissertation on the problems with reviews in the game industry, which I would probably do if I didn't cut myself short. But yes, the weird weight to scales in game reviews is another issue, but one that I'd argue Metacritic exacerbates in its own special way. If you have a site that tries to give more honest reviews, that gets treated exactly the same as sites that review on the 7+/10 scale. Another reason why blind averaging just doesn't work.

[Misery]

Metacritic is okay for very rough estimation of quality. If it's below 50 that usually means bad, but I'd always look at the actual reviews if I really wanted to know. That's its only real useful feature, linking to a lot of reviews that you can parse with your working brain instead of letting their mindless averaging system tell you what to think.

This is a problem with the 7-10 scale, not with Metacritic.

(Also, note: that averages to an 81, which on the 7-10 scale is "buy":

1-6: do not buy
7: worth a look
8: buy
9: overhyped
10: nothing gets a 10)

I'd say it's a problem with both.

The scale doesnt help, but it's the review chunks that are at the core of it.

With a normal review, you know it's one guy doing it, and it's made very clear as to how the things he's saying are directly creating the number given at the end.  You can also consider WHO is writing it;  if it's someone that you tend to agree with overall, that can factor in heavily on your decision.  All of this is presented easily, and is very straightforward.

Metacritic, on the other hand, is an absolute and total mess.  Most of the time, you have NO bloody idea what's REALLY going on with it.  Often when I see one of these scores for a game, if I were to check and see what's generating it, it's going to list MANY people and sites, the vast majority of which I'll never, ever have heard of.  For all I know, these people may be an overall negative sort, or they might be so bloody insane that they can have full conversations with their own feet.  Yet even a SERIOUSLY terrible reviewer has a real effect on that final number.  It is, frankly, a deeply idiotic approach.

And that's when you're looking at "proper" reviewers.  It gets even more bloody stupid if you look at general random player reviews.  I shouldnt even have to explain what's wrong with THAT.

This stuff is why I consider the site to be an enormous disaster.

[BobTheJanitor]

They also refuse to remove completely inaccurate reviews from their score average, even when the source of the review pulls it from their site and admits that it was flawed. See http://kotaku.com/5960657/metacritic-refuses-to-pull-negative-review-that-gamespot-admits-was-factually-inaccurate

[Riabi]

I'm trying to figure out what kind of scoring system would be better than Metacritic.

I'm actually baffled.

What do you want, an electoral college?

How about... no scoring system? :p

[doctorfrog]

The problem with not having a scoring system is that someone will then go out and make a scoring system.

[Riabi]

The problem with not having a scoring system is that someone will then go out and make a scoring system.

That's fine, except if they do, I'm not likely to read that review.

[KingIsaacLinksr]

The death of scored reviews cannot come soon enough. Summing up an entire videogame with a made-up scale that has many different standards feels completely wrong.

[Histidine]

Mostly I just find the notion that game reviews are sufficiently precise and objective to merit the precision implied by a hundred point scale hilarious.

[Riabi]

Mostly I just find the notion that game reviews are sufficiently precise and objective to merit the precision implied by a hundred point scale hilarious.

Or even a 10 point scale.

The closest thing I'll take to a score is a system like "Buy it, burn it, trash it". Basically where you say 1.) Yeah, this is good, 2.) It's okay, maybe you'll like it if you like similar stuff, or 3.) It's awful, avoid.

That's it, just three slots, anything else is silly.

[Draco18s]

Any individual review cannot have the precision of a 100 point system.  However, they can generally review something on a 5 point scale for 5 different metrics, then perform a (weighted?) average.  Which can be mapped to a 100 point system (4 + 4 + 5 + 5 + 3 => 84)

Also, watch this:
https://www.youtube.com/watch?v=iOucwX7Z1HU

[doctorfrog]

The problem with not having a scoring system is that someone will then go out and make a scoring system.

That's fine, except if they do, I'm not likely to read that review.

With respect, what you don't read isn't important. (Or I, for that matter.)

When I was a wee lad, I'd read magazines for game reviews. I got burned a couple times, so I started asking friends their opinions.

Nowadays, I don't read many reviews, unless word of mouth is good, first.

Game publishers can't really reach me as easily as someone who reads game reviews and buys based on scores. Enough buyers do exactly this to make my wishes mostly irrelevant.

Game players want a scoring system because it's a way to be told which game to buy, and to self-congratulate on a purchase already made. ("Well I don't want it!" Me neither, and that's exactly why we're irrelevant. As Mitt Romney would see it, we are folks who wouldn't swing that way anyhow.)

Game publishers want a scoring system because it's a metric they can point at and say "success" or "failure," and take action on. Want to push for that sequel? Point at the Metacritic score. Want to get rid of a troublesome lead developer, who still made the company millions? Point at the Metacritic score. It's a dumb metric but one that swings a big hammer anyway.

So, we can wish for the death of the scoring systems, we can analyze how badly they work, and gnash our teeth about it, but they're not going anywhere. Enough people want them, or at least rely on them. If they didn't exist, someone would invent them.

From the perspective of a person who plays video games, this really doesn't bother me much. I'm not being deprived of good video games. I'm frickin' flush with them. And the majority of them are ones that sail under the radar, and as a whole, the indie scene is getting along fine, even with Metacritic entries that would topple huge titles.

It does, however, put small to medium-sized game companies in a tricky situation. Rather than rely on Metacritic metrics, they have to cultivate a fanbase the old fashioned way. It's not very fair, but name me a sector of business these days where the little guy isn't being constantly tread upon by towering giants?

[Ucchedavada]

Any individual review cannot have the precision of a 100 point system.  However, they can generally review something on a 5 point scale for 5 different metrics, then perform a (weighted?) average.  Which can be mapped to a 100 point system (4 + 4 + 5 + 5 + 3 => 84)

Assuming that those review scores are normally distributed around the "true" score, that would give us a 95% confidence interval of 4.2 +/- 0.7. That is to say a range of scores from 70-98 on a 0-100 scale.

[Draco18s]

That was 5 metrics (gameplay, graphics, sound, etc.) not 5 reviews.

Also for a good sampling you'd want 30 or more reviews.

[Ucchedavada]

Ah, my bad. However, my point still applies; you'd still have to account for the error / uncertainty when you aggregate those metrics, as indicated in my previous post. Not to mention that most reviews do not offer such a breakdown in the first place, nor is it clear that such metrics can even meaningfully be summed (conceptually, what does for example 3/5 in Graphics + 4/5 in Gameplay even mean?).

[Misery]

Any individual review cannot have the precision of a 100 point system.  However, they can generally review something on a 5 point scale for 5 different metrics, then perform a (weighted?) average.  Which can be mapped to a 100 point system (4 + 4 + 5 + 5 + 3 => 84)

Also, watch this:
https://www.youtube.com/watch?v=iOucwX7Z1HU

Not good enough.

Part of the problem is that it's the bloody internet.  By default, it's made entirely of crazy.  And worse, it's the GAMING COMMUNITY on the INTERNET.  So you add pointless, stupid raging, trolls with the might of 1000 stars, and the extreme and total insanity, and you get.... well, that's basically what Metacritic is made of.  And worse, some of those reviews are thought of as "professional" (I could seriously rant for 10 pages as to what's wrong with THAT idea).

If I could simply destroy the site myself, I would.   With the virtual equivalent of 300 supernovas going off at once.


If someone wants info on a particular game, go read some basic reviews of it.  Find a reviewer/writer that you like, who seems at least relatively sane and objective, and just read what he/she thinks.  Dont go and just look at the damn score, READ the review.  Pretend the score isnt there.  See some detail on the game from a good writer, and then after that, maybe look at a brief gameplay vid.

And POW, you already know way, way, way, WAY more than bloody Metacritic will ever teach you about said game.  Your chances of missing out on something that you'd otherwise love, simply because Metacritic, is super dramatically lowered.

This is actually one of the core reasons why I think about it the way I do: Because awhile back, that one happened to me.  There was a game, I forget exactly which one.... I buy so damn many.... that I didn't touch because "Oh, it's getting this low score on that site. Okay, must not be very good".  Months later, I play it at a friends' house, and.... Yeah.  Turned out I loved it.  But because I'd just looked at a damn number generated by lots of seperate reviews by reviewers who seriously dont need to even have anything in common, including preferred genres (wow, yet ANOHTER flaw!), and who may or may not be A: insane, B: trolls, or C: insane trolls, well... I'd denied myself that experience for that long.   When all I'd really needed to do was watch a couple of quick videos of the game, maybe see a basic write-up somewhere from someone into that genre.  Instead, nonsensical Metacritic score.  I can tell ya, I wont do THAT again.  That site is now worth a vague laugh, if that, but that's all.