Part 2: Inside the HUD with Tony Stark

 

Christmann: Was Iron Man the most extensive project you’ve been involved with? What was your role on the film?

Seki: For me, yes. It was 19 months — the longest I’ve been on any one show. It was also the most I’ve been involved in the production and that heavily embedded in a show. So it was a great learning experience. Everyone was really generous and collaborative, and it was great. It was very difficult, too. I would say that if I had to evaluate Iron Man, it was the most difficult project I have every worked on, yet the most rewarding. It’s that irony that sort keeps you going, I think. Something that’s worthwhile is usually pretty difficult. My title on Iron Man was Visualization/HUD Effects Supervisor.

Collaborating with Hundreds of Contributors

Christmann: Of all of the UIs that were displayed in the movie, which ones did you specifically work on? I have heard the word POV and HUD mentioned in various circumstances, what is the distinction between those two?

Seki: HUD or H-U-D (heads-up display) stands for all the ones when Tony has the mask on and is looking out. We tried to distinguish between when it’s a face shot of Tony in the mask with graphics in front, and an actual true POV. So we would say the HUD is Tony’s face with the graphics in front and the POV is his point of view out — how Tony sees the world when he has the mask on. That’s the distinction we drew so we could talk about them in an easy manner.

As far as the rest of the displays in the movie go, several people worked on them, and there are tons of UIs besides Tony’s own interface. Alex Sears did all the on-set monitors and graphics. A lot of the ones you see in Tony’s workshop that are behind him or when he’s working on his car engine and he says, do an explodedview, Alex Sears and his team made those and actually played on-set with Robert Downey, Jr.

 

Then there were sets of graphics on very specific monitors. One example: when Tony is modifying his Mark II suit after first flying, he says, “Put the gold titanium alloy on.” Those monitors were done by a company called Prologue, Kyle Cooper’s company. I don’t know if you are familiar with Kyle Cooper, but he is most famously accredited with doing the opening titles to Se7en.

I think Kyle reinvigorated the titles industry and motion graphics in general with those titles, which he did when he worked at R/GA. Then he went to co-found Imaginary Forces, and left there to form Prologue. We had Prologue do several of the other monitors that are featured heavily in the movie, as well as the holographic display that Tony tinkers with. It appears twice — once when he’s taking the Mark II and paring it down and again when he’s creating the gauntlet that goes on his arm sticks his arm inside that holographic gauntlet and tests it. Prologue did all those graphics. Also, they did the really cool end titles as well.

The Orphanage did all the HUD and POV shots as well as some of the interactive glass interfaces in the house — like when Pepper touches the door to Tony’s workshop. Prologue also did the big glass bay window when Leslie Bibb’s character wakes up and sees the view, the weather, and everything else about Malibu on that window. Amazing work. PLF designed and composited the graphics in the Combined Aerospace Operations Center (CAOC) during the movie’s “dogfight.” We also designed and composited the screens during the sequence in which Pepper downloads the manifest information in Tony’s office.

Christmann: You mentioned The Orphanage working on the HUD and POV. How did you collaborate with The Orphanage on those?

Seki: The Orphanage was the vendor that the shots were awarded to. They ultimately finished all of those shots, so we should be very clear about the credit. They deserve tremendous credit for the work they did. I was asked to come on to work with them, to guide that process. When they were awarded the project, Jonathan Rothbart, the visual effects supervisor at The Orphanage, took on that work.

Since my background was in graphic design, I was sort of a natural fit and I began doing some development as they were working on it and it became increasingly clear that the level of the detail in the HUDs was going to be really large. John Nelson, the visual effects supervisor, very generously offered to get my involvement overseeing those details under his guidance. So I started talking with them and we developed a partnership working together.

The main liaison at The Orphanage for the HUDs was Dav Rauch, and he was fantastic.  I can’t stress this enough. A lot of the graphics were due to his direct involvement in the design process. He took a lot of ownership. He made the great observation that the HUD was a character in the film and that it played an important story role that interacted in concert with Robert Downey, Jr., and that the HUD was a part of the suit.

The Orphanage team came down for a big download session with Jon Favreau and the rest of us at production. You could see Dav getting energized during that discussion, and we walked away with a very good understanding of where we wanted to go from that meeting. They took the ball and really ran with it. They had a very small specific team dedicated to the HUDs and Dav really drove it home, I have to say. It was really great working with him. I had a fun time.

We started working together remotely. We would see work from them once or twice a week. I would comment on it and we would use CineSynch software to look at the same QuickTime content and draw on it and talk to each other about how we wanted to update it.

Near the end as we got close to delivery in January, I started flying up to San Francisco once a week to sit in and work with their artists and talk to them and sit in their dailies and comment on how the shots were going to come together.

 

 

Sticking to the Story

Seki: Jon Favreau directed us like this: “The most important thing is that the graphics hit home on story points that I care about. I can tell you what those are, and you can fill in the blanks.” So we said to ourselves: Well, in the concept art for the show, there are small cameras embedded inside the mask. So what we’re seeing is a tiled version of Tony’s face with graphics laid on top.

But then the next question was: Well, who’s looking at this? So we came up with the story that it’s Jarvis, or anyone else who’d like to check in with Tony on a computer interface, so it’s a virtual version of Tony. That freed us up to decide how to lay the graphics in.

When you watch the movie, you see that the graphics are all on the eyepieces. They’re not actually in front of Tony’s entire face. We created a representation of Tony’s world inside the mask. It’s a pretty high concept that asks the viewers to take a big leap.

That was another reason why we got so excited about those shots in the movie — if the HUD failed, it would fail miserably. But if it succeeded, it would be one of those amazing transitional moments that are conceptually very high, but people bought.

Christmann: I would like to find out a little bit about the process you go through. We are both creating interfaces but also kind of live in different worlds. At EffectiveUI, we go through one approach to develop innovative projects and rich experiences and bring them to market — through architecture, design and development. I’m curious what your steps are and how you go about creating innovative and rich experiences — from brainstorming to final production.

Seki: Well, on the HUD specifically, it really started with John Nelson getting tons and tons of reference. He found all kinds of present day HUDs, user interfaces that seemed to relate to what we were going to try to do, including the Terminator movies and anything we could find. We would download and have a plethora of things that we could always draw upon as far as looking at how it’s been done in the past. This is what current thinking is on it.

Christmann: Did you look at current military material?

Seki: Yes. We found tons of F22 visuals and other targeting information. In particular, one film that was highly influential was Appleseed, the Japanese anime film. The point being that we looked at all these things and decided to cherry pick what we liked about all of them. Then once we knew what we liked, we set about doing tests and style frame.

The Orphanage did a great test early to pitch the job. They took a Robert Downey, Jr., stand-in, shot some footage and put some graphics in front of it. Based on that very promising test, John Nelson decided to award the work to them.

Christmann: By a test, do you mean a quick little design and laying it out there as if it was the eventual design? Or is it more of an iteration?

Seki: The tests are free, no money changes hands. So, they did stills representing a more complicated version and then they did a simple version of a test in motion. That’s generally how tests like that work — a little bit of give and take. You can’t ask for tons of iterations and tons of extra work, but generally tests give you enough flavor of what the team is thinking. At the same time, they’ll do some stills to further explore more deeply what the visuals are going to look like.

One of the major challenges with the HUDs is that it was so open-ended, it could have been a lot of things. As soon as that happened, the problem was that any one person’s idea of what that thing should be is different from everybody else’s ideas. Graphics are really a difficult thing to talk about. The language we need to talk about graphics is so subjective that you can go down many paths without hitting the right one. I find that this project was a little different from visual effects, when mostly you have to ask if the effect looks real. So you can talk about the elements that make something look real or fake. But what design is a good design? There’s some wiggle room, and you can debate about what good design is. Regardless, the background of the story matters most and there are a lot of things that can factor into design success.

Christmann: Did the team iterate through a wide range of designs? How many different design approaches did you take?

Seki: I think The Orphanage took iterations into the triple digits.

Christmann: Wow.

Seki: I think one of the shots versioned well into to the hundreds, with something different in each one. So it wasn’t 100 completely different versions. But at one point they sent us 15 different still frames of possibilities for the HUD and asked for our thinking and for direction. The difficulty there is when you go to an a la carte menu, I want to choose part A of this and part B of that approach …

Christmann: Sure.

Seki: The danger is that part A and part B seemed like a good idea separately, but when you put them together — not so hot. Design has to be cohesive and it can be very tricky blending and balancing the composition.

Christmann: How did you ultimately whittle it down to a final design for the HUD?

Seki: Much of it was through trial and error. Then the widget designer at The Orphanage Heath Kraynak, and their lead compositor, Kyle McCulloch, worked really closely with Dav and they came out with something that they felt very strongly about and that suited the rules that we set for them, as well.

Then it became much clearer — when we wanted something vignetted we knew how to handle it as far as the framing, Z depth and sizing. Then things start to really come together. What’s interesting though, is that the POVs came together much more quickly than the actual HUDs. Actually, that makes sense because the POVs are something that you can easily relate to. You’ve seen something like it before in other movies. But the HUD is so different that it was a much more difficult beast to wrestle to the ground.

Christmann: If I’m understanding you right, you gave the Orphanage a set design frameworks to work in, right?

Seki: I would say that’s a fairly accurate statement, but that framework was open for debate. I’m not a big fan of dictating the way things must be. No one person has all the answers. It’s a testament to Dav that he was able to look at the rules that I would send and apply them sensibly. By the end, it was a real exchange of design ideas that involved a lot of mutual trust — a level of trust that comes from working with someone over time and knowing they are just committed to that work as you are.

Christmann: Do you feel the spirit of collaboration was really what made this project successful?

Seki: For sure. The Orphanage really embraced collaboration. They took it town. They deserve so much credit, and I hope they get the recognition from this project that I seem to be receiving at this point. They did a fantastic job.

Jon Favreau was very keen on the story. He would always be clear about the story we’re trying to tell. John Nelson, the visual effects supervisor, kept an overarching view on the picture as a whole, as well. He would come back to me said there were three things the HUDs need to do:

1. They need to be driven by Robert’s performance. You’ve got to remember that it’s all about Robert. It’s his face. It’s what he’s doing. You must let that drive it. You can obscure the performance with the graphics, but at the same time let the performance drive where the graphics go.
   
   
2. You need to let the Z-access space really work because that’s what makes the HUD shots successful. That is, the notion that there’s much more beyond the camera but we’re so intimate that you can’t see those things. And,
   
   
3. You should have an alpha event happen in each of the shots. What he meant by that was there needs to be a story point. Boom, you hit home with a graphic. Maybe it’s lost in the viewer because the shot is so quick, but if you were to watch it again, a third or fourth time, you would notice that the event happened. Robert would say, “Leave it up on the screen.” Boom, all of sudden, the power meter stays up on the screen. You don’t think about it when you’re watching it, but it does happen.

Christmann: Right!

Seki: Another thing that Favreau was very keen on was this notion of the HUD being in a dialogue. Sometimes, Tony Stark is asking for information. He’s calling for it to come to him. Other times, it’s Jarvis saying, “What about this? Have you considered this?” So that plays back into Dav’s notion of the HUD being a character. That sentient portion of it is a character.

 

Downey Soars in Previs

Christmann: Interesting. I’d like to explore those three points some more. How did you go about merging the script elements and Robert’s acting with the visualizations to make sure they were all in perfect synch with each other?

Seki: That was a back-and-forth with the director and visual effects. There was a point at which we needed to go through all the HUDs right now and look at them determine what they’re supposed to be doing. So we went through shot by shot and saw them in context to understand how they work.

They filmed the HUD face shots of Robert twice. The first time was on the last day of shooting and at that point we weren’t really sure what it was going to look like. We had an idea, but things hadn’t been set yet. So cinematographer Matthew Libatique shot it out in 65mm so we’d have a larger film back to work from and we could scan at 4K instead of 2K. That would allow the compositors to push in and move it around wherever you wanted to.

The second time we did it, we said, “Robert, we’re going to put these cards up that represent things you’re going to look at when you need information.” We showed him the test and Jon Favreau started working with him about what’s happening inside the mask. He would let Robert improvise on that, but Jon had a better idea of what things were going to look like than Robert did.

Christmann: Interesting. So you mocked up paper cards, if I understand it right, as little samples in a way. You placed them in front of Robert to test his reaction to use as a reference point.

Seki: Yes, the cards were static. For example, one said “power,” one said “diagnostic.” So before the sequence was filmed, Favreau would talk to Robert about what was going on and so he had a background story on what was going to happen. Then Robert ran with it.

I think that the success of those headshots is because Robert is so compelling as an actor. One thing that actually worked very well is when they showed Robert our previs of him on a teleprompter of him flying. So his reaction in the movie when he’s flying out of the garage — we call it the “woo-hoo moment” when the dowels are spinning and the flight bubble is completely rotating just one after another — Robert is actually reacting to watching himself fly. So the performance is very true to what was supposed to be happening at that moment in the movie, which I think is great.

Christmann: Oh, wow. So he was looking your previs sequence of himself flying while he was being filmed?

Seki: Yes, Robert was reacting to the previs version that was our first pass on what those shots were going to look like.

Christmann: Oh, very cool.

Seki: Basically, he was looking at himself as Iron Man flying out of the garage and he was astounded.

Christmann: So that child-like moment of wonder was real!

Seki: When I watched the movie I had a sense of relief at that moment. Tony had a lot of heavy things happen before that and when he flies out of that garage, everyone can relate to that moment of release that he’s flying.

The thought of anybody really flying is just — it’s a big deal. In this movie, it’s not like in other superhero movies where it’s just easy to fly. Here, it’s hard. It’s not an easy process. You need skill and there’s some jeopardy involved. I think that’s one of the big successes of the film.

Random-Free Zone

Christmann: About that “woo-hoo” excitement … In our industry we often have to balance really fun aspects and wow moments with something that’s useful, that goes beyond that momentary excitement and interest and still be grounded enough so people will get something out of the application and out of the UI. Sometimes it’s about balancing the two purposes; sometimes it’s about choosing one versus the other.

Ultimately we try to maintain both over a long period of time for the end user. So when you were designing the HUD and the POV for Tony, how you were thinking through the wow factor for the audience, along with useful and usable controls for Tony?

Seki: We agreed that nothing random would be in a shot. Everything should have a purpose. So if I asked the designer what a certain visual element is doing, it had to represent a function like pitch, or other information about his suit.

When you can justify all the visual elements, then you know you’ve done your job because you don’t have to worry about anything being random. We took out all randomness and made sure we had a purpose to everything. That way, when we consciously decided to do something that is random, we knew we were doing it for a reason.

For example, when Tony is flying up out of the garage and you see the HUD shot, there’s some code that’s just sort of flying around on the left and right side of the frame. We purposely said, “at this point, his suit is overloaded,” so it’s showing random bits of code because it’s trying to sort itself out. We wanted to limit that kind of thing until there was a reason for randomness, like when his suit was malfunctioning.

At the same time, you want to make cool graphics. But, if it just looks cool and doesn’t really work, that’s not so great.

Progressive Adaptation

Christmann: There was a nice balance of where the POV did adapt to the situation with really meaningful information. So while it was definitely a “woo-hoo moment” for the audience, I did notice a lot of elements that were really interesting about how the suit adapted itself to the situation. Sometimes it was a flying suit with the visual displays of an airplane cockpit. Other times it was a targeting system for weapons use that that had a whole different interface. What were you thinking around designing the suit to adapt and progress?

Seki: We knew we wanted the HUD to reflect one of the main themes of the film, which was technological progression. When Tony builds something, he continually tinkers and makes it better. If you notice when he takes off in the Mark II, you’re 100 percent right that the interface switches. In the garage he’s in what we call “analysis mode” and then he starts to power down and says to wait a minute, do a weather check and ATC check.

Then he changes into flight mode when the flight bubble comes up and the UI definitely shifts around and becomes flight mode. We decided that after he does the initial flight with the Mark II, he’s made some decisions. He no longer needed two separate modes, only one. In the Mark III suit he has one mode that we call “battle HUD,” for better or for worse. It sounds a little bit more aggressive than it really has to be. Now, Tony can access all functions through that one mode.

So in the Mark III, the big jump up there was that the interface became white, if you noticed, whereas the Mark II was basically cyan with some other colors. That idea came from the additional VFX supervisor on the show, Wesley Sewell. He came to me early on and suggested that white would be really cool for the Mark III. He said, “Think about how graphics 10-20 years were green monographs because of the monochromatic monitors. Today in movies, monitors are all pretty much cyan. What’s it going to be tomorrow? White, clean white with color accents for attention.” We ran with it. We thought that was a great idea and it definitely worked.

So your observation about the HUD adapting in the middle of the sequence is correct. Tony adapts the HUD to his needs, as well. You’ll notice on the bottom of the Mark II HUD, there’s what we call the “dock.” It’s like a dashboard on the Macintosh or on a PC. It has like five icons on the bottom and each one stands for either diagnostic, flight, power, radar or navigation, targeting, and they all have purpose. Tony would look down and activate that function and then it would turn into that mode.

You can see every so often those bottom dock icons lighting up, pulsing when active and turning into the next mode. We figured that by the time he left the Mark III, he didn’t need that anymore. So we put him into what we call “expert mode” where the things will be accessed from what The Orphanage team invented called the “omega widget” — a circular widget that is activated by Tony’s eye motions. Why would he any longer need the dock down at the bottom and have to look so far down to change mode if he knows the suit really well?

So we used the Mark II as the introduction to look like a computer interface and resembles a dashboard. But by the time he gets to the Mark III, Tony doesn’t need all that stuff. If you notice in the Mark III there’s a small dock at the bottom, which is mainly just for show just so he can quickly look down and to see that he’s in flight, targeting, or diagnostic mode. But he doesn’t have to look down to access the device; he accesses it directly from the omega widget.

 

From Screen to Real-World

Christmann: So his eyes were the input into the system and the system was able to recognize eye movement.

Seki: Yes, that was the idea that we had — plus we used voice access. We brought in the futurist John Underkoffler from the Tangible Media group at MIT to look at how we were progressing with the HUD and to give us guidance in case we were headed off base in terms of behaviors. He brought up this really interesting concept called “artificial foveation.”

 

Christmann: It’s interesting to see how films inspire people who will eventually create these realities. There’s a definite synergy between older science fiction films and ultimately what comes into being. In our industry, the movie The Minority Report is referenced a lot — especially around the emerging Microsoft Surface Table technology. I’m curious about your thinking that what you were doing on screen might influence future realities. Do you see any kind of real-world use for this technology anytime soon?

Seki: I don’t really know when this technology is going to exist. We didn’t really consider that too much. I can say that in general terms, we felt that the technology had to feel just a little bit futuristic, slightly more than you could possibly get now. But it couldn’t be too far out because then the movie would seem inaccessible. We didn’t want it to become a crazy future thing with flying cars and the like.

The film has to feel grounded even though the technology was advanced — but not too far advanced. We kept bringing ourselves back to the idea that it had to be accessible. John Nelson, the visual effects supervisor, brought up the word “intuitive” time and time again. It had to feel intuitive. While I’m not a futurist, I can speculate as to what the technology can be used for — military, flight, deep sea exploration, deep space exploration, gaming, simulation, training, fabrication and sports.

Can you imagine a cool heads-up display for cycling when you’re riding your bike? Instead of looking at your computer on your bike, you’re looking right through your helmet or your sunglasses. In football, auto racing, even the sport of crew, you could have a little readout that tells you how many strokes per minute, time lapse, a stroke counter — you can have all that stuff in a HUD so you don’t have to look at an external device.

So there are real-world applications for it. It’s definitely an intellectual discussion that you can have. Dav made an interesting observation one day that there aren’t many good HUDs around, and that we really needed to make ones a lot nicer. For example, I don’t really know what’s going on in the new military HUDs, but the guy that’s flying the plane sure as heck does. We had to make something that was advanced but still accessible. It was a balancing act that we had to find.