The Future of VFX During COVID-19: Digital Face Replacement, Social-Distancing Doubles

Film and TV projects got the green light to resume production in LA County last week — but the coronavirus still rages on while producers and studios/networks scramble to find ways to keep cast and crew safe. That’s where VFX will come into play, at a time when location scouting, intimate scenes and large crowd shots are discouraged. “We’ve already seen a big change of how our services are being utilized,” Nhat Phong Tran, VFX Supervisor at Pixomondo and two-time Emmy nominee for his visual effects work on HBO’s “Westworld” and Fox’s “The Orville,” told TheWrap. “With this whole COVID situation, there are problems arising from different angles and several different levels. On the creative side, it’s much more difficult; creative can’t be as intuitive anymore because of the physical limitations due to the safety measures. We recently got a lot more interest in how we can utilize VFX to support productions and bring people back on set and bring them back to work safely.”  Tran, whose credits also include “The Fate of the Furious,” ‘Crazy Rich Asians,” “Fear the Walking Dead” and “True Detective,” said his team has already started working on a multitude of solutions to combat arising issues. Dialogue scenes will be shot separately and stitched together in post-production, and digital face replacement techniques will be used for action and intimacy scenes. Action scenes, for example, can be done with an actor’s stunt double whose face would be replaced in post. Or, the person fighting opposite the actor could wear a protective suit that would be removed in post. A “social distancing double” who is more comfortable in intimate scenes can be used to replace a star who isn’t as comfortable with close quarters, and their face would also be replaced in post. Non-speaking characters might be asked to wear masks that will be removed in post, said Sam Winkler, a visual effects artists whose credits include “The Call of the Wild,” “Westworld” and most recently, Andy Samberg’s “Palm Springs.” David Gidali, director, VFX supervisor and the founder of Outpost VFX, agreed that face replacement is a valuable method. “For intimacy scenes, one approach to reduce risk is to use digital face replacement. We happen to be right on the cusp of being able to do that seamlessly,” Gidali said. “A recent example is ILM’s de-aging work for the film ‘The Irishman.’ It would obviously require casting the right body doubles, preferably couples who social-distance jointly, but it’s becoming more and more doable.” But Winkler doesn’t see how intimate scenes can be shot under social distancing guidelines: “The best solution might be to simply shoot with more suggestive angles that don’t require physical contact. It’s very common these days for post to do split screens to combine separate takes of actors into one shot, but even then, there are two people working together, not one actor talking to a tennis ball unless it’s a CG character. Would you want to do that with a serious drama?” Tran and Jordi Matsumoto, a director and producer at digital production company Marching Penguin, say that planning a shoot will have to be a lot more methodological from now on to anticipate all issues that would have to be fixed with visual effects down the line. VFX crews need to be on board early on to identify potential problems. “It starts in the writers room — as filmmakers, we have to think of it holistically and that’s the reality of it,” Matsumoto said. “Moving forward, you have to pressure test the need for scenes with extras or intimacy, but the beauty of filmmaking is we can be creative of how to shoot it differently.” Gidali agreed that VFX teams should be early collaborators on projects, especially to keep costs from skyrocketing. “This wisdom is already embraced by big-budget and VFX-oriented productions, but it will likely spread fast across other productions such as reality TV and more grounded half-hour comedies and dramas.” But even with precise planning, there are still scenes that might be more complicated to shoot with solely VFX. Tran explained that large crowds, of 100+, for example, will require a lot more planning. “Treating VFX as an afterthought has always made things more expensive and complicated, but coronavirus precautions will make this even more true,” Winkler said.  Even pre-Covid, Gidali explained, a common and cost-effective approach was “crowd tiling,” which means filming a small group of extras in one place a bunch of times, swapping them around to create variety and then duplicating and tiling those plates in the final shot. CG humans would also be used to allow the cameras to move more dynamically, but this is more expensive.

Now, Tran explained, you could shoot people, then give them a wardrobe change, and then shoot them again and stitch those scenes together. However, in bigger crowd shots — take the Colosseum scene in “Gladiator,” for example — you would need digital doubles. You take 10 actors you filmed live-action, scan them and get their 3D “agent” which has their look, but alter it enough so they look different. But you also have to get them to act and behave differently so not everyone in a crowd shoot is doing the same motion — this is done with AI simulation by building a “behavioral network” for the crowds. 

“For things like crowd shots, intimacy scenes, location scouting, etc., in somewhat of a coincidence several technologies recently reached a level of maturity and became production-ready,” Gidali added. “Fully digital crowd has been used already in shows like ‘Game of Thrones’ and ‘The Walking Dead.’  While historically limited to wide shots, advancements in rendering technology allow us to get much closer to ‘digital actors’ without losing fidelity.”

But where it gets trickier, Matsumoto explains, is with those shots that are in the middle between VFX and reality.

“Let’s say you are in a restaurant and there are people in the background and waiters walking back and forth — with social distancing and planning, you might be able to do those,” Matsumoto said. “But in a crowded bar, that’s where it’s going to be dicier, if you have shots that are tracking with lots of people in the background. Then we’d have to decide, do we shoot it on a longer lens and then compress the frame? That’s where the discussion is going to be more intense.” Gidali added: “Probably to even greater effect, screenplays will be altered and shooting styles revised to avoid or minimize certain scenarios. While we may see a wide establishing image with hundreds of digital extras, we may not be gradually following the protagonist through the crowd, but rather cut immediately to a tighter angle of the protagonist with the surroundings a fuzzy blur of shallow depth-of-field. There will certainly be many practical limitations even with the helpful hand of VFX, but Hollywood is an industry well-versed with self-imposed limitations as is, so there’s absolutely no doubt that productions will jump right into action and quickly outthink these limitations.” Location scouting is something that can also be done virtually, and VFX can be a great tool for that, as well. “In terms of visual production, how can directors and creatives actually still be doing their job? The only way to really do that without being present on set with a large crew is to have a virtual production environment, which is something we have done before,” Tran explained. “We had the location scout go with the VFX crew, which was like three people, to Iceland where we made virtual scans of a 300-meter radius, and then we take that data back and what we do is we build that environment and everyone can sit at home and scout Iceland.” However, Gidali doesn’t think producers will replace physical location scouting with virtual quite yet, given the costs involved. “Instead I think we’ll see a greater reliance on green screens and digital backdrops, or more immersive technologies like (again) ILM’s ‘Stagecraft’ which was utilized on Disney’s ‘The Mandalorian,’” he said. And speaking of costs, will budgets balloon now that more VFX is required? Matsumoto doesn’t think so given that budgets are established early to account for visual effects, while Tran says budgets will have to be reevaluated to account for more time spent on visual effects in terms of labor. But Gidari disagrees, saying that studios will see an inflation in budgets but that studios and networks will quickly find a way to balance costs. “As productions begin to resume, the increase in reliance on VFX will definitely cause a spike in associated costs,” Gidali said. “In addition, some studios may be short on cash because of the ‘aftershock’ caused by canceled or paused shows that would have landed on their desks by now but haven’t due to the pandemic.  I believe things will balance themselves out rather quickly. Studios will quickly develop increasingly efficient techniques in order to remain competitive, and producers will become smarter with how they shoot to avoid excessive reliance on VFX.” Winkler added: “VFX budgets will definitely increase since a higher percentage of shots will become VFX shots. But as the quantity of VFX has increased over the last few decades there has also been enormous pressure to decrease the cost of VFX labor through outsourcing and tax incentives.” Overall, all four experts agree that technology has evolved enough to handle most every VFX need for future productions as long as there is early communication between teams as opposed to bringing in VFX groups as a “last-act rescue” team. “We have all the technology we need — it’s just about finding and identifying the problems and where we can assist in a timely and economical matter,” Tran said.  “We do have incredible technology now,” Matsumoto said. “We’re at a level we’ve never been at before and shooting at resolutions we’ve never shot at before. With so much data, it’s hard to imagine that we can’t piece something together.” Winkler said low-budget TV and indie films will face some challenges with this new normal since “you’re not going to be able to cut corners the way you used to. You’re going to have to pay for safety supervisors, staggered call times, more space to keep crew distant from each other and VFX for scenes that normally wouldn’t have it in the budget.” But Gidali says believability is still the biggest challenge — after all, VFX still has troubles with effectively rendering close-up shots of faces.

“Even on ‘The Irishman’ where the technology cost millions to develop and implement, the results at times fall short of being 100% convincing,” he said. “The general rule of thumb is the bigger a digital face appears on screen, the harder it is to convince us it’s real. There are many creative workarounds, and it’s not rare for directors to intentionally opt to hide faces to great emotional effect. But the bottom line is, while we’re technologically closer than we’ve ever been to being able to create realistic faces digitally, we’re not at a point where most productions could expect to rely on it successfully.”