Network now...FEATURE
Is Metacritic Damaging the Games Industry?
Indeed, many PRs are strongly affected by a greater use of metascores as well. Certainly, we’ve spoken with a PR for a major publisher who expressed huge frustration with the stress their companies place on getting the best they can. “PRs who haven’t been in the industry very long will get angry when certain new publications that I know haven’t earned their reputation aren’t included, or some local daily paper,” says Doyle.
“I say, ‘Guys, they haven’t made it yet’ – I try to be as kind as I can because I know sometimes they have clauses in their contracts that make them affected by metascores in terms of bonuses or penalties.”
The practice could be subtly changing the relationship between developers and publishers, too. “You really want a producer to focus on doing everything that’s right for the game, not to be focused unduly with the review score it’s going to achieve,” says Splash Damage’s Paul Wedgwood [above].
“If his bonus is wound up at a score of 70 or 80, he might be tempted to err on the side of caution rather than taking risks and pushing for an 85 or a 90. Look at projects like BioShock, for example – on paper that isn’t something any sane producer would take on, but it’s obviously well justified by its review scores.”
Further to this, many developers of games for broader markets feel that metascores are unrepresentative of their work. “If you look at family games and kids’ games, they consistently score as many as ten review points lower for, dare I say, what’s similar quality,” says Frontier’s David Braben. “It’s actually really hard to make a really
good kids’ game.”
For Wedgwood, developers can be more directly affected by a poor metascore. “The negative side is if developers are penalised for achieving low scores despite not having control over the resources and schedule for the project.”
Doyle thinks so too: “If they’re having to achieve a certain metascore with the same budget, that’s disturbing.” But it depends on the relationship between developer and publisher as well, as Wedgwood says: “Obviously, if the developer is wholly or partially funding a game or has a strong relationship with the publisher and can determine when it’s going to be released and how much it’s going to cost to make, it’s their responsibility. And I think in that situation it’s quite common for a publisher to have an expectation for quality.”
Contact Us / About Us / Advertise With Us / BeltwayBlips / TechBlips / MassiveBlips / GamerBlips / FAQ
Subscribe To Edge Magazine / Corrections / Terms & Conditions / Privacy Policy
Copyright 2006 - 2009 Future Publishing Limited, Beauford Court 30 Monmouth Street Bath, UK BA1 2BW
England and Wales company registration number 2008885
Subscribe To Edge Magazine / Corrections / Terms & Conditions / Privacy Policy
Copyright 2006 - 2009 Future Publishing Limited, Beauford Court 30 Monmouth Street Bath, UK BA1 2BW
England and Wales company registration number 2008885
well I know that quality review titles like people to place trust in the review itself, not the numbers at the end. OK, fair point, but unfortunately the review industry burnt that bridge of trust with the public years ago. Anyone recall the xbox magazine that gave Halo 2 a perfect 10? Official or not, professional lying is damn right immoral, when the magazine in question has a serious price tag attached. Then we had 'Drivergate', and the mother of all con tricks from Sony; a certain mr David Manning. To quote the BBC "A judge has finalised a settlement in which film studio Sony will pay $1.5m (£850,000) to film fans after using a fake critic to praise its movies.
In 2001, ads for films including Hollow Man and A Knight's Tale quoted praise from a reviewer called David Manning, who was exposed as being invented." The fact Sony Entertainment was targeting movie goers is irrelevant, it is the same ass clowns at the top calling the shots and we know it. What continues to amaze me is the fact that large corporations and their printed review poodles never seem to seriously consider that the buying public swap notes on their work and have done since the birth of the net. Necessity is the mother of invention and I am afraid that Metacritic is the resulting offspring. Enjoy your child.
If there was an ounce of credibility in game reviews then metacritic would serve a purpose.
It includes such biased sources like the official Playstation magazine and Nintendo Power. Not that the independent guys are much better as proven with the gamespot Kane & Lynch disaster or as anyone would see reading Kotaku.
To top it off the guy doesn't even give honest average scores but has some cocked up method of doing things with some mags being worth more than others (yet another bias in the system) and he won't even explain how it works.
It's correct that Edge's scores are often out from the other guys but that's why I trust Edge rather than official console magazines and less than moral companies like Gamespot.
Game reviews in general are about as honest as the Weekly World news and its Bat boy stories. Anyone with an ounce of intelligence can see that.
Nintendo sold off Nintendo Power to Future (publisher of Edge, PC Gamer US/UK, OXM) about a year back.
I'm aware of that and the fact that Future do the official Nintendo mag in the UK too. I enjoy those magazines and have been a NP subscriber in the past but their reviews, like any "official" magazine, are slightly biased.
I would base my purchasing of games on both Edge reviews and metacritic score, but a 6 in edge is 60-70 euro staying in my back pocket. And yes, metacritic is a great indication of how good or bad a game may be.Developers can't please everyone but they will have to produce some top quality games to get them scores up because everybody is watching. No pressure!
In lieu of certain recent Edge reviews...this article = a touch defensive?
I don't see it that way at all. They end the article by suggesting it's a change for the better!
Is Metacritic Damaging the Games Industry? Well no, I don't see how it could.
If more publishers are looking towards a metacritic score as a target then good for them and good for us. It means publishers will be more focused on producing games of higher quality, and I can't see how that could possibly be a bad thing. There do seem to be two main issues at hand though.
The first is this: as long Boyle's criteria for the weighting of certain publications is hidden there will always be cause for suspicion over the reliability of his methods. The fact that he's the sole editor of the site doesn't help his claim towards being objective or reliable.
The second isn't actually a fault of metacritic, but the result of a bizarre pheomenon that seems to have occured within the field of games critique. A surprising amount of reviewers will use the seventy-mark as their average point (out of a hundred) instead of using the logical median. Others, and I would mention Edge and probably Eurogamer as included within this bracket, seem to start with five as the average -- the median point. As a result you have a magazine like Edge or Eurogamer giving a game a five out of ten score which would then equate to a seventy-plus score in another magazine.
And then you have publications which use an out-of-five scale to determine scores. So now you have to contend with three diferent scoring systems: the out-of-hundred, the out-of-ten and the out-of-five scale. With all these different ways of scoring games there are bound to be discrepencies which aren't incorporated into the metacritic calculation.
But in general, yeah, metacritic is a boon to the consumer and a fairly reliable guide to the quality of a game.
Agreed. I understand why he might want to keep his methods hidden for fear of copycats, but they are fairly well established now. he would be better off being transparent with the methods.
The different scoring systems are not a fault of Metacritic, but they do show the difficulties in attempting such a thing. It should only really be used as an at a glance guide to see what general consensus is. I'd hate to think anyone made their purchasing decisions based solely off it.
Let me get this right.
Step 1 – You take subjective opinions of reviewers and the arbitrary scores they decide to give games.
Step 2 – You filter these through some convoluted, and also completely subjective ‘formula’ some guy has concocted on his own.
Step 3 – You somehow arrive at an ‘objective’ score for a game.
How could this not be harmful to the industry? Surely all you have done is take the subjectivity inherent to the reviewing process and compounded it?
I remember Edge went through a stage when they scrapped review scores altogether. I absolutely loved those issues. Edge has always stood out from the rest of games journalism for me because they touch on aspects of games that others do not. Maybe it is time Edge scrapped those scores and stood out from more prominently from the piles of sludge out there?
If you are interested in how metacritic compares to gamerankings point for point, there's a video report at:
http://www.gamequarry.com
It goes over all of that. There's also a video report of how metacritics weighting system has been applied, the related scoring swings, etc. The vids are also on youtube, just search under metacritic.
Cheers,
Tim
Are game reviewers hurting the games industry by handing out scores that feed the metacritic system?
Metacritic and Game Rankings are the best way to get a general consensus from professional reviewers about what games to pay attention to. If Edge gave out a 7 to a game, and some other magazine gave that same game a 9, I want to know what the difference is and what others think. If the majority of publications are handing out 9's to this game and there is only 1 magazine out of 30+ others that gave it a 7...
Good thing I have a central location to see what everyone has to say about a game instead of putting faith in a single publication. Or everyone can just abolish game scores all together and give out a Buy, Rent or Sell label.
Sounds to me like Edge resents Metacritic, because Metacritic proves how out of touch Edge's reviewers are with the rest of the gaming community.
thank god for Metacritic. It has exposed Edge for being "so far away" from everyone else that it shows who has the agenda. It also takes the power away from reviewers, to damage a game with a "final say", as opposed to being just another number in the bunch.
Edge, stop trying to change the industry to what you want it to be. Consumers run this industry, and if consumers want, need, and like what Metacritic does, then fall in line or become a fashion mag.
Someone here said that "Metacritic is killing originality"....are you kidding me? If being original is being a troll fanboy website then yes, Metacritic is killing originality.
But lets face it, the ps3 games with the worst review scores from Edge, those 20-30 points away from the Metacritic, all did something original, or tried to. Only once has Edge given a ps3 game with an original idea a good score, LBP. The other games, like Eye of Judgment, Lair, Resistance 2, Killzone 2, Folklore, all tried to be innovative, but Edge didn't like that innovation, and punished the game accordingly.
So the innovation and originality are there, it's just that Edge rewards innovation "it likes" and punishes what "it doesn't." But it doesn't punish Halo X, Gears X, GTA X, so again, we are back to the double standard.
Funny that this article is going to be included (or was already) to convince your own readers that its "Metacritic" that's ruining the industry, and not some fallout shelter for fanboys trying to ruin the hard work of developers who are pushing the limits and trying new things.
Good job Edge, the A+ content just keeps coming.
Oh and by the way, Marc Doyle isn't going to see things your way, or mine. He makes his own judgments based on what is happening in the industry, and when him and I spoke on the phone, the impression I got from him was that (and I am paraphrasing) "if there is a respected publication that carries a lot of weight with their scores, and they are not connecting with the public or are not being balanced, their weight can be adjusted down, and vice versa. It's all about tweaking."
Metacritic is here to stay, so stop your crying Edge and get used to it. Yes, there is a website that tracks what you do over time. I guess accountability is something new to this publication.
thank god for Metacritic. It has exposed Edge for being "so far away" from everyone else that it shows who has the agenda.
So if you don't agree with metacritic your just plain wrong?
also doesnt the GOY award for LBP kind of sink the edge ps3 bias argument?
It's actually the other magazines that have a problem with their scores, not EDGE. Most of them have a vested interest in selling their magazines. Usually the covers feature "exclusive review" and other such commercial hooks. They ramp the score up beyond what it deserves - because thats what you check when browsing on the news stand. they need to do that regularly to maintain sales. EDGE almost never features a cover concerning a reviewed game.
Metacritic is full of reviewers who DO NOT play through the game from start to finish and some have openly admitted to being able to pick a score within 15 mins of playing the game. (usually because they are the same all the way through from beginning to end - but there are exceptions).
The only recent EDGE score I take issue with is Mirror's Edge. But it didn't stop me buying or thoroughly enjoying (most of) it.
"thank god for Metacritic. It has exposed Edge for being "so far away" from everyone else that it shows who has the agenda."
I always thought that Edge (and a few other magazines) based there rating scale differently than most others.
While most review sites/magazines would regard a 70% rating as 'an average game', for Edge the equivalent score would be '50%'.
So on that basis it didn't really come as any surprise that Edge scores average below the Metacritic scores.
Still, I can't agree that Metacritic is harmful to the industry any more than the whole concept of rating games on a linear scale. The games industry would still make the same decisions based on game ratings even if Metacritic didn't exist, it just happens to be something that makes it easier for them get a aggregated score.
It's not like the industry has been short of controversy over publishers 'buying' good review scores.
Mate no one needed Metacritic to know that EDGE is 'so far away' from everyone else.
You'll find that's why 30,000 buy it every month.
"You'll find that's why 30,000 buy it every month."
If you are focusing on the numbers, you've lost the point. You are taking this way too seriously, and obviously since you are reacting, it shows you are a fanboy. /troll
This comment just shows how arrogant, and ignorant you guys really are. why don't you try gaining respect through honesty, integrity, and consistency, instead of throwing around numbers and claims of "most respected"?
I am sure there is respect for Edge, but respect is subjective. 30,000 is probably a good enough number to feed your own ego, but to me, it just looks like another way to validate your already misplaced intentions.
I guess the only way you guys will see the writing on the wall is when that 30,000 is something lower, far enough lower that you will take notice, that something is wrong, it doesn't make sense, why is this new number so much lower than all the other numbers?
you'll get it someday Edge. I have faith in you.
pswi60
Are planing to punish edge by overloading their internet traffic?
:)
At the end of the day, all I know is that I love games and one poxy website with an average score is not going to influence my purchasing decisions. I will happily give KZ2 a blast at a mate's house when they get hold of it and make up my own mind, but I'd rather play something original, daring, provocative and exciting. We all know KZ3 is on the cards but what saddens me is that something like Okami 2 or Rez 2 isn't. Reviews only play a small part in a game's success. The rest is down to advertising. Just look at Assassin's Creed.
Metacritic means nowt, except perhaps the slow painful death of originality, daring and risk-taking by developers, and that saddens me deeply because I truly believe games can be so much more. Hey, at least GTA4 is top of the pile. That game contains the above elements mentioned in the first paragraph and it has clearly paid off handsomely.
If Heavy Rain (I really should stop mentioning it shouldn't I?) is as amazing at it has the potential to be and sells lots, I really think it could be a turning point in the production of games and for the industry as a whole and that can only be a good thing.
here i have nice article here
http://thatgamingsite.com/articlenav-115-page-1.html
The best system for this sort of thing in my opinion is the Rotten Tomatoes model where you pool everyone, then break out the "real" critics vs the enthusiast critics. You can see an overall score or a score based on what people who are educated in criticism rate it.
Of course, I doubt there are a whole lot of people educated in criticism reviewing games. Gaming lacks a Pauline Kael or a James Agee who can bring up the discourse and help make someone's career with their reviews.
Brian
www.brianwoods.com
Metacritic scores are quite damaging to the industry, but the service that it provides to prospective buyers as a clearing house of different reviews is valuable and benign. Unfortunately, this genie has left the bottle.
metacritic is an example of why democracy isn't exactly the best idea for everything. The masses are simply too stupid to really do most things properly, which is perhaps why enemies of democracy arose *in* the first democracy (Plato, anyone?).
Metacritic is also a sign of the decay of criticism. Everything comes down to a number on a linear scale of 1 to 100. How incredibly lame and inaccruate -- we demand games to be called Art and yet we judge them with the depth of a lima bean.
"Metacritic is also a sign of the decay of criticism. Everything comes down to a number on a linear scale of 1 to 100. How incredibly lame and inaccruate -- we demand games to be called Art and yet we judge them with the depth of a lima bean."
Well said. The whole linear scale is the fundamental problem of reviewing (not just in the games industry)
For example, Half Life Episode 2 might be a fantastic game, but there are some days that I just can't be bothered to look at it, yet a blast on MK Vs DC (or something else deemed 'average' or 'uninspired') would be quite appealing. Different games appeal on different levels.
In much the same way that movies like Crank or Shoot 'Em Up are great fun to watch, but in a completely different way to an movie like Burn After Reading.
The problem starts with the reviews, not at Metacritic. By reducing a game experience to a single number, the review is insinuating that all games can be expressed through a lowest common denominator; literally a number. What a load of BS.
Reviews should focus on explaining which taste buds a game tingles. If you hate shooters, then the 10/10 new fps hotness won't matter to you. Even if you look at one genre only, it can have a lot of design decisions that make or break the game. I like the driving in Burnout Paradise, I hate the driving in NfS. A plain number cannot express to me which game is for me. Each genre is littered with game mechanics which specifically cater to one audience.
Whining about Metacritic is not going to change that. The review culture has to change. A reviewer has to able to explain to me in words why the mechanics, presentation and overall entertaining value of a game is worth my while and who might not like it. If you review books you got to have an understanding of literature science to compete. If you review games, you just have to cough up the hypetalk of the developer, combine it with juvenile excitement and slap a number on it. In the end all, we get is paraphrased ads. Such a course will naturally lead to all review reading like the ones in the burger King magazine. 5 Stars out of 5, every time, it's hilarious.
Id argue that reviewers by and large do that anyway. What's important about the review is the text, but scores can provide a useful at a glance guide.
Yes, there is a lot of sloppy journalism out there, but it's not always that bad, and there are people out there who do more than just regurgitate press releases.
I'll admit - I prefer GameRankings as it's purely an average of all review scores (not weighted). The junky small time press / sites could artifically inflate / deflate a score for a short time, but overall the average evens out and you can say "if it's an 80+ on GameRankings, it's worth looking further into"
"Is Metacritic Damaging the Games Industry?"
Think so, yes.
Do I care?
Not much. There are sad people who just can't be happy with their games (with anything basically), so they visit these places to splash some bile. See the user "reviews" there or the Killzone 2 "scandal" here.
Edit: it was off topic, I know, but I'm tired, tired, tired.
I've always had issues with metacritic, but they tend to be related to games im passionate about , and i fear is the same for everybody else.
But hidden beneath my bias is a certain degree of truth. I remember looking at LBP and Deadspace scores and finding "Variety" the publication consistently at the bottom of the scale (80 for LBP, 70 for deadspace) [numbers that deviate too far away from the mean].
LBP has a total of 95 and Deadspace 88, and i wonder how much impact the variety score had, but thats they issue isnt it, i can only wonder... the easy answer is that a metacritic score is a weighted average.
(as a side note the only game rated higher than LBP [GTA IV] has a 98, and has no score from variety... this leads me to my other point)
The other issue is the inconsistent amount of reviews included, any given high profile game will have a higher amount of reviews, making the apples to apples comparison nearly impossible, add the fact that we have a limited pool of reliable sources and i start to conclude that the more reviews you have the less reliable the final score is.
Any set of empirical data to be considered reliable has to be controlled/consistent and able to be replicated. (now i dont think Metacritic has ever claimed their data to be empirical... in fact i would think they have said it is anything but)
So here are my fixes for the Metacritic formula (eventhough nobody asked me for it)
- Cap the amount of reviews to a set number (10, 20 whatever)... this makes it more of an apples to apples comparison. You know you will have a good set of reliable sources from the big names in the gaming world (1up, Edge, Kotaku, etc) throw in some reliable non gaming magazine sources and i think you have a good set of "empirical" data.
- Include the rest of reviews as anecdotal... This reduces the possibility that the smaller lesser know sources taint the sample, however you want everyone to know what everyone is saying, a wide range of opinions is good. Heck it can even be that nobody knows which sources are part of the empirical set, but let us know that ther is such a system.
- Make it more transparent... don't give up all your secrets but reliable data is that which can be verified an replicated, tell me more on how things are done.
- Drop the highest and the lowest score... (i know this presents problems on a weighed average system but the capping should "reduce" the need for weighing) by doing this you eliminate that crazy high and that crazy low score, you know a game is getting all 80's and all of a sudden wham! a 100... or how about that game getting 95's and all of a sudden there goes that 70?
*a variation of the above
- Drop scores that deviate too far from the mean... theres a threshold there; theres no way a games whose "reliable" scores range from 100 to 85, should include a 60 in its average, that may not representative of the game but the methodology with which it was reviewed or that the sample is indeed tainted, that 60 is important don't get me wrong just not empirically.
* going off the deep end
- A multiple scores/grade system... tell me the controlled or empirical number 80 along with an all inclusive number 65.
Your final score should look like 80/65 this tells me there more to the story. (It would be interesting to see how close or far these end up)
"Is Metacritic Damaging the Games Industry?"
The simple answer to that is: No.
As long as there's bullshit reviews around like Edge's KZ2, then there will always be a place for a site like Metacritic where consumers can get a more informed opinion from various sources.
How do you know the review is bullshit?
Remember when GTAIV came out, after all the glowing reviews were out, people started to point out that the game had been over hyped and for game of that scale the reviewers had very little time to actually go a thorough review - a situation many believe was manufactured in order to bring out favourable reviews from reviews still riding the hype wave.
From what I've seen, sounds like a similar situation is happening with KZ2 - reviews are pouring out raving about how great multiplayer is, yet access to multiplayer has been restricted (if I recall, there have only been 3 press MP sessions..), which came to light in the other KZ2 'controversy' when a couple of other review sites didn't publish reviews on the grounds that they hadn't been provide sufficient time to properly review the MP aspect of the game.
With the amount that is riding on KZ2, you have to expect that these things have been thought about and planned for by Sony.
Is the Edge review crap? Well, it reads pretty favourable to me.
Are they right? I don't know (and don't really care... but it's hard not to get drawn into arguments of principal!!), at the end of the day all that will matter is how many people are playing it online in 6 months time. If it's the most played game on PSN, then it's obviously the hit Sony wanted, but if everyone is back playing COD, then maybe Edge were right.
Finally, everyone remember to out side and get some fresh air... this is just a game, and these is so much more to life ;)
Steviepunk said:
How do you know the review is bullshit?
Dude have you read the review?
Finally, everyone remember to out side and get some fresh air... this is just a game, and these is so much more to life ;)
You sound like a Mormon.
Let's just ignore the fact that KZ2 had a Beta for it's multi-player that many in the gaming press have played already.
Can we dig a little deeper here and let's discuss some other games where reviewers did not play the multi-player, gave out perfect scores and then when the gamers get it the damn thing is broken online for months....but yeah it had great multiplayer 10, 10, 10.
You bring up a point I have an extreme issue with. Reviewing games that they have yet to even finish. If you have not completed the game (I don't care if it's a massive 100hr RPG), do not call it a review and do not give out a score. Of course management won't like that but you can't fully review something if you have not fully played everything. I extend this to multi-player games as well. They should at least be broken into 2 part reviews with a score (I hate scores) released after the second review. Perform a review for the single player, and then a month after the game release, perform a review for the multi-player. This will never happen though.
This Doesn't inspire confidence though does it?.. Metacritic based on one man's opinion?
What absolute nonsense - metacritic holds no more power that edge does for it's review scores.
It's about perception from the reader - and the perception is that an average of review scores is the way forward, and a natural progression of this is smoothing less renowned publications scores be they high or low in a total average score.
What edge fail to extrapolate on, and it's quite astounding the lack of intelligence on display, is that all the constituent scores are shown.
I, personally, would trust a metacritic average score way in advance of an edge score. It's less personal, more consistent and is not prone to bouts of egotistical behaviour.
Oh and let's not forget. "Let's Tap!"
Bunch of twats.
The only problem with MetaCritic/GameRankings is that people don't understand reviews. DigitalStorm's obsession with "Let's Tap!" getting a higher score is a perfect example of that. Does he, and other readers, realize that every website/mag scores a game in relation to it's platform? Just because a game like Let's Tap gets a higher score than a game that shows up on PS3/360 doesn't mean it's a better game - it just means that it is a better game than another title on that system with a lower score.
I was just expressing my concerns given that Edge are a much bigger organisation compared to that of Metacritic... Give how I have been reading Edge now for a number of years and know how seriously they take their gaming.. I'm inclined to believe them over a bunch of fanboys or lonewolf lawyers.
There is no need at all to be insulting, you have a right your opinion and I respect that. but it works both ways. So maybe you should be a little more respectful.
if it wasnt for people being colourful and giving a different view these comments sections would be boring as...
"I say old bean that was a damn fine comment wasn't it"
"Oh yes - I whole-heartedly agree."
"Well that's wrapped up that then. Pimms anyone?"
You can be colourful without being insulting or prejudice ! Try not starting your posts "Oi Cunto" and not ending your post "Bunch of twats"! That would be a great start..
Good Article... Thank You :D
This Doesn't inspire confidence though does it?.. Metacritic based on one man's opinion? or did I just read that wrong?!. My personal opinion of this is, if we want to use sites like these as the basis for scoring a games quality then people like Doyle are going to have to be more transparent about how they go about scoring these games and where they get the information from.
The Killzone 2 review thread stands testament to the confusion caused by trying to compare metacritics scores with independent analysis.. Let hope that people realise this sooner rather than later.
Until that happens I'll always be looking to independent reviewers for their opinions, it's important to me that all information comes from as close to game industry as possible but without being completely biased.
Until that happens I'll always be looking to independent reviewers for their opinions, it's important to me that all information comes from as close to game industry as possible but without being completely biased.
dumbass. Keep us all informed as to when the cell has divides - then post with some intelligence.
Great comeback...
isnt it just.
Excellent article!
The thing is your really can't trust game totaly trust games sites. Gamespot is a good exmaple their known to be ani pc. Gamespot also has a track record of their writers being paid under the table. I find IGN to be creadable. It all comes down to creadability.
as pc gamer I go bywhat gamer say about the game. It the scores reflect what other gamers are saying about the game. Then I will buy or not based on what gamers say and scores.
Speak English, boy.
you get the point :) here is a good exmple
http://kotaku.com/gaming/rumor/gamespot-editor-fired-over-kane--lynch-re...
good call