With a US$300 million budget, a three-month IMAX run, and the hopes of the stereoscopic 3D movement on its shoulders, all eyes are on James Cameron’s Avatar. Not just moviegoers’, either. The videogame tie-in, in development at Ubisoft Montreal, hopes to spearhead 3D’s move from the big screen to the television. But as industry insiders are discovering, the movie isn’t just a pioneer for live-action 3D, but for next-gen motion capture as well. This, the director of Aliens and Titanic tells us, will benefit avatars of every description.

How involved have you been on a day-to-day basis with the Avatar game?
I wouldn’t say day-to-day but month-to-month. They’d come back with art, storylines, characters, and concepts for how they wanted the gameplay to work. And they’d either come in person or they’d send it to me and I’d make comments. Usually, it was the visits where they brought in the development kit and we’d watch it, play it, and talk about it. But this isn’t every day. I’m making my movie every day; they’re making their game every day.

I gave them a lot of latitude. I created some boundaries for areas I didn’t want to go, if not to protect the movie then to protect the sequel possibilities – stuff like that. It was a really good collaboration. They were really respectful. They wanted to make a great game, but they also weren’t inhibited. They came up with some great ideas I wouldn’t have thought of, where I thought, ‘if I can’t work them into this film, I can certainly work them into a sequel’.

Do you play games yourself?
I’m not super into it in that I sit and play for hours on end, but I have five kids. My brother and his two sons are avid gamers – we’re talking 100 hours a week. So I use them to get the pulse of the business and what the serious gamers really like. In fact, I incorporated him on this project because I wanted a consultant who really knew what people wanted to see.

So my brother Dave came to me and said the first thing they showed him was a firstperson shooter. And they all loved it and it was really cool shooting a Na'vi bow, but you might not have a good sense of yourself as a Na'vi. So they switched it to thirdperson, where you kind of switch to right over the shoulder when you shoot.

Was the plot of the film a natural fit for a game?
Absolutely. You’ve got an antagonism between philosophies: you’ve got technological humans and the Na'vi, who live on a Neolithic level but are very harmonious with their environment. They can harvest from it what they need in a time of great crisis. It’s a conflict that will play out over time and over an entire world. And we’ve seen this conflict – with the Europeans coming to the Americas, and in our own history – and we know it tends to end badly for the indigenous people. But the Na'vi aren’t to be discounted. They’ve certainly got some tricks up their sleeves.

The creature designs are, I think, some of the best I’ve ever seen. And we drove the creature design group to not only do something really good but incredibly detailed in terms of the lifecycles of the animals and how they move. Their structure is very well thought out, and the flight dynamics: the way they breathe, the way they move, the way they perch. So a lot of that just got dropped straight into the game. There’s a richness of imagination where [Ubisoft] started with that foundation and just took off from there.

What were their faces like when you said you wanted full 3D in the game?
To be honest, they were hesitant – they didn’t even think it was possible. But I planted the seed. I didn’t say, ‘Guys, you’ve got to do this. It’s deal or no deal.’ But I said they should really think about trying to do this in stereo because it’s the right way to do it. They went away, and a couple of months later they came back and showed me a stereo demo that convinced us all that was it.

Will 3D have the same impact on games as it’s having on cinema?
These changes are led by individual titles, not ideas or philosophies on new technology. The idea of stereoscopic digital projection has been around for about seven or eight years. It’s only taking off now because there’ve been a number of good films to drive it.

What we’re doing with this game is really important, because if the game’s successful and other people try to emulate it, then stereo game development will become a major thing. And if it does, it’s going to drive stereo in the home. It’s going to drive TV companies to build the big screens. We already have Xbox 360 in stereo; the others will have to adopt. I think it’s going to be huge.

How will the relationship between games and movies have developed five years from now?
You’ll still have the straight licence deals and they’ll be OK – as good as the game guys can do. But if we do our jobs well with this one then maybe others will look and say, ‘Oh yeah, this is how you do it.’ I would look at a sequel to Avatar being something where I sit in a room with their creative people and we think about cool ideas that should be in game and movie simultaneously. So there might be a portal or place in the game that you can’t pass through, but there’ll be clues in the movie, and vice-versa.

You’ve been quoted as saying Avatar’s motion capture technology was a lot like a game engine.
It was a game engine. A game authoring package.

What kind of shot does it allow you to do?
It’s not so much that we searched for a strange or wacky CG shot. I actually pulled back from that. I think a lot of stuff looks like CG because it doesn’t look like there was a camera there. So by introducing a camera instead of just watching mouse-clicks, it actually takes on a very naturalistic style.

That’s much of the strength of what we’re doing. I did all the live shots myself – except the Steadicam shots ‘cause I don’t operate a Steadicam – and operated 100 per cent of the virtual photography. So the same body language was used in both media, and I think they converge in a way where it looks like one piece. So, curiously enough, we were working using game authoring tools to shoot the movie.

What about the uncanny valley phenomenon?
We talked about it a lot. Game cinematics are honestly still in the uncanny valley, but they’ll get through it. The limit is in the speed of real time rendering. A given frame in Avatar might take 50 to 100 hours to render. That’s per frame, per processor, so they have 6,000 processors running in parallel at a render farm so they can do five or six shots per night. But I think our fingernails are locked in the cliff on the other side of the uncanny valley; sometimes we climb up, sometimes we slip back.

If you watch the movie, is every shot out of the uncanny valley? I think one in ten slips back, but at that point the narrative is propelling you along and it doesn’t matter. And interestingly enough, we first thought we were safe because these are blue guys on another planet. I’ve since realised that it’s much harder to light blue skin than human skin – a lot of the tricks don’t work. So, in that sense, I think we’re a lot closer to doing a human than a Na'vi. I actually think you could do a photoreal human right now, no problem.

But it’s a huge investment. The thing that people don’t realise is that the technology is not based in how accurate the data set is, but how good the rigging of the model is that the data is then applied to. Yes, you still need good data, and our image-based facial is working very, very well. We’ve got great eye and eyelid data, and all the facial muscles are properly accounted for. So we’re getting extremely accurate performances out the other end. But it’s the twelve to 14 months of rigging that model in between that make the characters real.

The first thing you see is a little bit ghastly. But you study it and make little tweaks to the push and pull of the facial muscles, and after a while it all comes into focus. It’s an amazing process that could also be applied to the cinematics in games, but it’s not an automated one, and I don’t think it ever quite will be. Not until we go to a level beyond just scanning, and beyond image-based facial capture, to some kind of realtime MRI scan of the facial muscles firing – something that may not exist for 20 years. It’s muscles behind the surfaces that make the expressions. You can’t just take a scan, map it to your fantasy character and expect it to work.