Network now...FEATURE
Why Metacritic Doesn't Matter
There is a prevailing theory that review scores affect sales. It is certainly true that there is a high correlation between aggregate review ratings, like those found on Metacritic.com, and sales; games rated over 90 percent tend to record higher game sales than their lower scoring peers. The question that was posed to the developer panel at DICE Summit 2009 was whether they designed games with reviewers in mind. The answer was an overwhelming no.
"The first person you have to design for is yourself," says Danny Bilson, creative director at THQ. He acknowledges that Metacritic is a powerful force. He's found that he can game Metacritic by improving what he calls "linear assets" that don't affect the actual code, which the team spends most of their time writing. By that he means offering up a better story, better voice acting, and better music. Reviewers have to write, says Bilson, so give them something to write about. By paying attention to such things, you can gain a five to ten point bump because in Bilson's eyes game reviewers care more about story than consumers.
The consensus among panelists was review scores in and of themselves don't matter. Fingers pointed to the success of games like Wii Fit that scored in the 60 percent range. Reviews of such games just aren't relevant to their audience, says Rich Hilleman, chief creative director at Electronic Arts. Social gamers are not the people who are going to be reading hardcore game magazines.
No, the issue isn't the impact of the score on sales, it is how your company uses them, says THQ's Bilson. They are either a touchstone or they are a weapon. Companies use Metacritic scores to project game sales. When teams pitch a game, they're asked to estimate the score. Why would a development team ever shoot for anything less than 100 percent, he asks. The problem is that these projected scores determine game budgets and marketing spends.
Metacritic scores are trailing indicators. Chris Taylor, chief executive of Gas Powered Games, doesn't need a Metacritic score. He knows if he put out a "shitty game." Developers get that information from retail, he says. If it sells well it was good, if it didn't it was bad.
Another problem with Metacritic scores, says Bilson, is they aren't weighted. Reviews from Time Magazine are treated the same as some kid's blog. A low score from a "couple weird sites" can significantly drag down average review scores. [Metacritic's website actually notes that scores are weighted. Thanks to readers for pointing this out.]
Hilleman suggests that game developers don't need Metacritic at all. With telemetric data, they can see what is wrong with all of their products--the answers are in the data. Community, he says, is the most important thing.
Besides, says Bilson, "How can you put a number on art?"
Contact Us / About Us / Advertise With Us / BeltwayBlips / TechBlips / MassiveBlips / GamerBlips / FAQ
Subscribe To Edge Magazine / Corrections / Terms & Conditions / Privacy Policy
Copyright 2006 - 2009 Future Publishing Limited, Beauford Court 30 Monmouth Street Bath, UK BA1 2BW
England and Wales company registration number 2008885
Subscribe To Edge Magazine / Corrections / Terms & Conditions / Privacy Policy
Copyright 2006 - 2009 Future Publishing Limited, Beauford Court 30 Monmouth Street Bath, UK BA1 2BW
England and Wales company registration number 2008885
The problem with Metacritic is that reviewers are idiots. Sure, there's Yahtzee (whose hyperbolic jokes are more informative than 95% of other reviewers' entire careers), and a few of the Edge staff (though their reviews of GTA4, Halo 3, and Resistance 2 seriously need rechecks), but other than that game critics are just average, mostly dumb people. Few ever venture beyond lists of "I liked this" "I didn't like this" in paragraph form with a number attached, not even reliably commenting on the experience, but making sure they give an opinion on every aspect of a game whether or not they carry any relevance. Biases seep in, either through brand name loyalty or advertisement promotional cash, to the point where you just can't trust the words on your screen.
Hopefully a couple of decades from now we'll get real writers/developers to help us choose what games to get, but as of now it's just down to waiting 'till they hit the bargain bin and deciding for myself.
The problem with many game critics is that many of them are worse fanboys than the ones you find on gaming forums. I know of one in particular who posts on a non-gaming related game site that has a small gaming section who's openly 360 biased. He posts all manner of misinformation about the other platforms (but manly about the PS3) and is unbelievably over positive about the 360 to the point that you'd think that each unit come direct from the bosom of the Blessed Virgin Mary. Apparently the fail rates on the PS3 are about the same as the 360 and Fable 2 was revolutionary!!!!!!
The stuff he comes out with would be quite funny if it wasn't for the fact that he's in a position of trust and that his hyperbole could be taken as honest and balanced information by people who aren't as anal with facts as we all are. This guy claims to have written for Future Publishing in the past as well.
As bad as that is, that doesn't apply to as much of the game critic population as just sheer badness. With few exceptions, video game reviewers just aren't good writers, or even good analysts. They're just people who like to play video games and thought it'd be great to get paid for it.
I've never really understood why a lot of video game journalism is so poor. Why are people paying to read some badly written review that they themselves could have done better?
I'll not even get onto the odd scoring scale, just the words filling the magazines are bad enough. Do they not have subs?
Surely one problem is that we often get the reviewers we deserve? The comments above make the point that there are both fans and critics who make no bones about a favouritism that seems to the neutral to be something very like bias. Why would you expect reviewers to be any different than the people they write for?
Journalistic integrity is one thing, but this site showed the flipside, which is that an awful lot of people don't give a hoot for integrity. There's a lot of people who don't like to read things they don't agree with - sometimes to the point of hysteria, it seems. The Glorious Killzone Debacle showed that: it doesn't matter how good, or even how fair a review is, people who disagree label it a 'bad review'. When a lot of the market for reviews is fanboys, you simply won't succeed as a publisher if you produce great, balanced reviews that criticise what the audience loves. Even though they should be published, and even if they're probably true.
I'm not saying it's right, simply that it inevitably happens. If the hand that feeds you is attached to a moron, it doesn't pay to point out how stupid they are. Hence awful publications that make no secret of their zealotry, or worse, terribly biased reviews masquerading as neutral. And also why subs aren't really all that important when you can print any old toss and it'll sell on the basis that it's about a game the market probably already likes...
Metacritic does "weight" scores. But it won't tell you how, so who knows.
“The problem is that these projected scores determine game budgets and marketing spends.”
Surely that’s why EDGE is having a problem with Metacritic. I may be a little naïve here but that’s a hairy prospect.
Metacritic does a good job of compiling and averaging scores from all over the internet. Look at the most recent addition of Street Fighter 4 on 360. I'm not being funny, I like the Street Fighter series as much as the next person. But unless you are that casual about your gaming that you really don't care what you're getting, as long as it's in the top 20%. I'd have trouble justify purchasing this example purely based on the scores give here, however they make me automatically suspicious.
Those scores provide a good platform, especially in terms of generating interest around a product. It just seems a bit ill thought out. Out of all the SF4 reviews on that page, The one I'm most likely to read the one at the bottom to be honest, to see why "The Onion (A.V. Club)" gave it 67%. Of course I'll also be waiting to see what Edge's critical eye will brings to the discussion. (My personal feeling is that it's going to have its work cut out raising that bar above the Rival Schools, Dead or Alive and Soul Calibur games that have gone before it. My person favourite is Street Fighter III 3rd Strike on Dreamcast, but everyone is going to have their own Street Fighter benchmark to compare it with I would imagine.)
It's a good a idea for the most part, just not something I personal like! preferring a broader range opinions and information before I spend my money. If people are happy to use Metacritic as there benchmark that's fine. Although, I find it hard to deny that it relies and caters for the more impulsive and casual side to our gaming. The Lazy side of us that still walks into game station and starts browsing boxes when bored at lunchtime, willing something to jump out at you.
There will never be a completely accurate way of reviewing, because so much of what is written is opinion based. As long as you have people in the Industry that can provide good commentary and can provide a sound understanding of the product in a articulate and interesting way that surely good enough for the majority of us.
What worrys me about metacritic is the fact that it includes sites like gamespot whos dependent on advertising as a major revenue stream. Geff Gertsman need i say more ? .At the time he left gamespot Assasins Creed was released Ubisoft had payed for Assasins Creed to have their whole frontpage ( banners ) and guess what Kevin Vanord said it is a game you will remember for the rest of your life (quote check the video review) and gave it a glowing review with no mention of its pitfalls , his review was one of the most un-constructive peices of rubbish i have ever read in many years of gaming reviews.Of course his review score had nothing to do with all the money Ubi were paying for the adds did it !
I think i remember Edge doing an article on advertising and the fact that it pays for alot of these sites.
Absolutely right.. it's all so insidious isn't it! Fanboy-ism only partly explains it. There is no way of really knowing how much reviewing and or commentary is bankrolled by the distributors themselves, to show their product favourably. I realise offline it's harder for the printed press to misrepresent the facts in a lot of cases (isn't there some sort of commission setup to enforce this?) the difference with the online world where people hide behind lots of cabling and gamer tags. Lots of anonymity to exploit.
Savage ("He is a bloody murdering savage. And he's telling lies", "I never lie. But I am a savage") henry,
BTW, have you played SSIV yet? I found some of the characters are very unbalaced (Abel, Zangief, Ken, Sagat) and some of the characters nerfed (Vega, and really anyone with charge supers/ultras). The cancellation system only really works with characters who's supers and ultras don't rely on charging, and the throw system is totally broken (if you remember in SFII-SFIII if an opponent was standing over your body trying to throw you as you stand up you'd be granted a split second advantage to counter with a throw/uppercut/etc as you stood up. You don't get that advantage anymore. If someone wants to stand over your body and throw you while you stand up, well, they can now). Other than that, it's the tried and true SF of old, just broken now.
Way off Topic... No Not got it yet Nick!... Looking forward to it though, waiting for that time of the month (payday).. I only used Street Fighter as an example to illustrate my point about metacritic. :D Although from what you have said, It would be a shame if they left key elements out, but I guess it's inevitable in some cases. Looking a some of the screens and seeing the gameplay videos it worth I reckon for the visuals and to see what the new characters are like, but I am going to give a week or so and see what the reaction is. It's only like day 2 here in the UK and I am still loving Soul Calibur 4 so much.
You've completely thrown me with this though.. where's that a quote from?"He is a bloody murdering savage. And he's telling lies", "I never lie. But I am a savage"
Yeah, off topic. I've been waiting for my chance to rant about SSIV lol. I have only had it two days so maybe the more I dig into it I'll like it more. The visuals are pretty sick though. it does seem though that some of the moves, namely the Ultra combos, were put in there more for aesthetic purposes (they look awesome) but in practice only really work well in conjunction with cancellations if you don't have to charge them. So you could for example start an F,D,F P with Ken, do the focus attack right away (he stops his uppercut after the first hit and is still standing) then you cancel your focus attack and can go straight into a Super or Ultra combo (D,F,D,F +3 punch buttons) and your opponent is still stunned thus cannot block. Well, charged Ultras/Supers will not work in this fashion so you're at a huge disadvantage if you're favorite character is E. Honda, Guile, Vega, M. Bison etc.. Granted, those are more advanced moves (and there are even more examples where charge characters are at a disadvantage but I'm not going to go into those here) but for me I love Vega and he blows in this game. Ken is a beast. To be fair, I haven't unlocked the 8 additional characters so maybe they're more balanced.
And as far as the quote, it’s from Braveheart. He’s having a meeting in a horse carriage with the Princess and a male escort. William Wallace is telling the Princess that her King had done far worse things to the villagers when he took towns over. After he said that the escort dude told the Princess in Latin that William Wallace is a bloody murdering savage and a liar under the assumption William Wallace couldn’t understand Latin. William Wallace responded in Latin that he never lies but that he is a savage. He then went on to ask if they want him to respond in French if they prefer. Awesome scene.
I have a played through of a couple of the older games (SF2Champ and SF3Zero/3rdStrike) and I noticed similar balancing issues. I have always put this down to the differences in attributes between the characters and their individual fighting styles. It's noticeable with other beat'em ups too. Sometimes is weighted to heavily in favour of certain characters though and that's annoying, but usually that get tidied up with later editions. luckily, these days we have downloadable patches :D
I know people that have been banned from using certain characters (you know who you are!) or at best handicapped with extreme prejudice in the past so it'll be interest to how the balancing will fair. Although having said that, it could just be that's all new and different and takes a bit of getting used to. I bet your timing is a bit off... Let us know if you still feel the same way in a few weeks. I was similar when I start playing Soul Calibur.
From what you are saying, does that mean you can win a round from your first attack? Providing you can time the move right.. That could get interesting, but I imagine it's one thing pull that off against Capcom AI. I can't see anyone pulling that off with any opponent that shows you absolutely no quarter, not without some serious time in.
Some gamers may not have experienced some of the first games previous to Super Street Fighter 2 HD on 360 and PS3, would you say that Street Fighter 4 can stand alone on its own merit without being overshadowed by previous editions? And, is it a worthy addition to a genre that already as some hardened brawlers?
FOR THE WIN!!
I don't think you can win a whole round with your first attack, but with the right ex-focus, focus cancellation, ultra combo you can take off about 75% with one combo that's essentially unblockable after the first attack. My only real complaint is charge characters can't pull off as devastating a combo because the focus cancellation requires you to dash forward or backward to cancel your focus attack (which requires you tap forward or backward twice thus eliminating any accrued charged ultra/super), then you only have about a half second after that to pull off your next move before the stunned enemy topples over and is un-attackable. Non-charge supers/ultras are relatively easy (D,F,D,F PX3 for Ken or Sagat) and can be pulled off before the stunned enemy topples over. Charge supers/ultras on the other hand cannot be pulled off before the character topples over thus eliminating that combo for charge characters altogether. This basically means non-charge characters have many opportunities to pull off unblockable ultras/supers but charge characters have to rely on just the luck of landing one.
You gain the super combo bar by attacking/using special moves and you carry over your super bar into other rounds. Your Ultra combo meter is filled by being attacked but that bar does not carry over to other rounds.
It's somewhat of an oversight and I think presents some significant balance issues. If you're going to play online, be prepared to not only get thrown a lot (throw system as I said before is broken) but to play against Ken quite a bit (and get Ultra-ed by him which looks pretty sweet).
"William Wallace, a scottish paedophile, the worst kind"
http://www.youtube.com/watch?v=ybQCNb4AuW4
The braveheart stuff kicks in around 4 mins :-)
"I'm not saying...."
eeeeekkkk!!! keep diggin'
I like the dude with the beard, in the front.. he looked like he was getting pretty steamed. eventually seeing the funny side though
Pretty funny stuff. I didn't know he was a wealthy noble though...as well as a pedophile!
I think game reviews are turning into what film reviews already are. Take for instance a movie like Transformers. Michael Bay knows that none of the cultural critics will score it higher than 7/10. But he also knows that because of the high-budget special effects and franchise alone, it will gross a massive amount of money and attract more people than any Bergman-movie ever could.
So in other words, reviews stand totally uncorrelated with the financial success of a movie, and what do you think weighs highest -- a billion dollar or a perfect 10 in some newspaper?
I also believe that there are two poles in gaming. One is the casual pole, where people simply don't give a squat what reviewers say because they either don't read, or don't care about them.
The other is the experienced gamer type who has played whole lot, and will most likely turn to sites like GameTrailers and make his own judgment based on gameplay clips, screenshots and even demos at times. To him, reviews are often not needed because he already knows what he's after and knows the kind of games that attract him. Like Guitar Hero, even if it got horrible reviews people would still buy it, because they know what they get at this point. Same concept, different songs. That's more than enough to make people buy it.
In other words, reviews are in large very unnecessary. Just as with movies, they are neither a meter of the upcoming financial success, nor are they necessary nothing more than a reviewer's personal opinion.
"...game reviewers care more about story than consumers."
And if That Review is anything to go by, he's right, although it doesn't explain the gushing of perfect tens all over Metal Gear Solid 4.
*runs away*
Why Metacritic Doesn't Matter?
Well, I find it pretty helpful. I don't always care very much about the metascore as much as reading a summary of opinions from various sources, all conveniently gathered in the same page. The score is a good compass, it tells me if a game is pointing more to the north than the south. Games scored over 70 on Metacritic usually means they're worth a try to me, if they're scored below 60 I won't bother. Reading various comments, both from high-scorers and low-scorers, is always the best way to approach Metacritic.
In all fairness, the developers are saying that scores don't matter to them, not to you. At the end of the day, even a company like EA might give you a bonus for high Metacritic scores, but if your game is a commercial failure, you won't get chance to make another. If you release a low scoring title that results in huge piles of money appearing, you will be given more work.
Bilman, "How can you put a number on art?"
How about asking the gaming magazines that and not Metacritic. Metacritic does not serve developers, it serves us users who are looking to see what many outlets are saying about a game. If the publishers are using that aggregated score for other means, then that is their fault.
If people have a problem with aggregated scores, then tell the reviewers to stop giving scores so more emphasis can be placed on the text. That's to say the text is actually readable and not filled to the brim with hyperbole.
That's why i like reviews with simple score systems. Nobody liked the Next-generation score system and i loved it, just five stars. We don't need 10 or 100 score if we are evaluating something so subjective. I see the score just as a quick reference.
Simple scores are the ideal, 5 stars, no half stars. Product scores work when they answer one, simple, pointed, question: should I spend my money on this?
Readers of a review, not a piece of criticism, look for several things: for a videogame they want information on graphics, style, narration, gameplay mechanics and feel, multiplayer etc. all alongside a description of how much fun the game was and a short summation of the product as a whole. A percentile or a star rating simply isn't complex enough to represent all that, readers will try to apply some logic to single digits, hence all the apoplectic flatus when a game gets 8/10 after a positive appraisal, as was the case for Shawn Elliott's Crysis review for 1up.
If it's made clear that the number is just an easy hot or cold scale then most of this debate fades away. The Guardian's film site is a fine example of how to arrange this: http://www.guardian.co.uk/film scroll down for the stars for the week's releases, click them for a more fulsome description, and for those looking for in depth pieces there are archives of longer critical articles.
Most publications refrain from putting scores of any kind on visual art, visual art shows and books. Books are a particularly valuable example of the limits and dangers of a site like Metacritic, from their website:
"Book reviews are a little trickier than movie, music, and game reviews. With the exception of a tiny handful of publications, none of the publications assign a score to their reviews. This factor, coupled with the nature of book reviews themselves (which tend to be less straightforward than, say, videogame reviews) makes us very reluctant to use an exact quantification system for book reviews. While we are comfortable determinining if a book review is positive or negative (or mixed), we do not presume to know if a critic is suggesting that the book merits an 80/100 rather than a 70/100.
If we add up all of the points for all of the reviews for a given book, and divide that total by the maximum possible points (which would equal 4 times the number of reviews), and multiply the resulting percentage by 100, we have a METASCORE.
Note that individual critics and publications are NOT weighted for books as they are for the other genres; thus, each review counts equally in this equation. (We felt that in did not make sense to weight some reviews higher than others since very few publications use a consistent group of book critics--thus making it difficult to assess one publication's reviews vs. another's.)"
Of course, even when assigning a simple 1-5 score, Metacritic editors do not liaise with the original review authors; this lack of communication coupled with the fact that Metacritic is going against the grain by giving book reviews scores at all puts serious doubt in my mind as to the accuracy and value of the final scores.
Leaving the merits and otherwise of this much filtered metascore, your second point was that reviewers should stop giving scores in order to halt aggregation, but by the evidence here this wouldn't stop Metacritic at all, since it doesn't affect their book scores.
As to Bilson's statement, if we discount Metacritic: you don't put a score on art, you put a score on a product. Are art shows and books products? Yes, but publications seem less willing to treat them as such. The reasons for this are complex, divisive, and are really destined for a topic outside the scope of the Edge comment section. The telling quotation for the videogames industry is:
"...the nature of book reviews themselves (which tend to be less straightforward than, say, videogame reviews)".
What exactly is in the 'nature' of videogame reviews that makes Metacritic editors so happy to give them percentile scores? Even if all publications stopped giving games scores it seems Metacritic would be fine with doing the numbering themselves. Perhaps this goes further than reviews but actually extends to the way people think about videogames, as opposed to art and literature.
There is a movement towards more games criticism and fewer games reviews, I believe there is room for both in the same way that there is space given over to film criticism and film reviews. Again, reach is beginning to exceed grasp with this so I'll sign off for now.
"...the nature of book reviews themselves (which tend to be less straightforward than, say, videogame reviews)".
What he meant is obviously not that video-games are any lesser pieces of art than books. But a video-game (and movies as well) present numerous technical aspects that aren't in a book. Graphical and sound effects, loading screens, presentation and controls just to name a few. Imagine if books would occassionally contain spelling errors, tiny letter fonts you can't read or author-made illustrations.
If that was the case, I'd gladly cross-compare book and video-game ratings. But a book only presents a description, which is both the beauty and the limit of the medium.
I wasn't suggesting that Metacritic thinks of videogames as lesser works of art, though even if they do I can't speak for them. It was more my intention to question whether they and we think of them in different terms, some quality that allows for percentile scoring where it is shied away from in art and literature.
I disagree that books are only descriptions, they contain allusions, implications and explanations and sometimes illustrations too. There is a sense in which one could call all writing descriptive, but I'm not sure how useful that would be and in any case it's not the way critics talk about language and writing. Also, books and any talk or writing about books uses all sorts of technical vocabulary and concepts: metaphors, similes, poetic meter, even chapter titling or the lack of it; experimental works like Cortazar's Hopscotch, which only suggests an order to the reader, even intentional spelling errors, neologisms and writing to reflect speech mannerisms.
I absolutely understand what you mean, videogames reviews might talk about hardware, which is much more important than hardcover over paperback, cross platform comparisons and other really architectural concerns. If anything these additional elements suggest that videogame reviews should be less straightforward rather than more. If I had to pick an interpretation of the quotation, I'd say Metacritic editors find videogames reviews more simplistic, and they aren't explicitly reflecting on the nature of videogames, whereas they see book reviews as better developed, more complex. The slight, if there is one, is against games journalists not games developers.
I believe this simplicity, this feeling of being comfortable with giving games percentages is bound up with how we see them. Some quality in their presentation, their history or just what they are, and hence my original question. So I do think it can be of value to cross compare, if only to work out the difference. I'm not sure it's something easy to name, but I might very well be missing the obvious. It's certainly worth discussing.
"The METASCORE is considered a weighted average because we assign more significance, or weight, to some critics and publications than we do to others, based on the overall stature and quality of those critics and publications." Metacritic.
Take that Mr. Bilman
Is it Bilman or Bilson?
I think it's possible he was thinking of Gamerankings, but in either case, he should have made sure he knew what he was talking about.