Virtual production gets real: Bringing real-time visual effects onto the set

Virtual production offers greater freedom and flexibility. Using technologies such as game engines, LED volumes, and augmented reality, virtual production can bring real-time computer-generated graphics and visual effects directly into the production process, enabling everyone on set to see and interact with them. Sets can be built inside a game engine or captured using photogrammetry—for example, by scanning the surfaces of a Paris neighborhood to create a 3D model. The virtual set and its effects can then be rendered in high-resolution on LED volumes—walls and ceilings made of LED screens that surround the physical set—transforming a sound stage in Burbank into that Paris neighborhood. Actors can see digital sets and elements and react to them more naturally. Cameras with augmented reality layers let operators and cinematographers see more dynamic digital assets such as virtual characters driven by motion-capture actors. Stage cameras can align with virtual cameras to precisely track movements across physical and virtual spaces.

And virtual production’s benefits can extend far beyond what it enables on set. For many producers, it’s an invaluable tool for making the filmmaking process much more flexible. For example, objects in the digital set can be repositioned on demand, and lighting between real and virtual elements can be more easily matched. Production time can be shortened, costs can be reduced, time and location constraints can be removed, and creativity can become more unbounded.¹

As studios support more streaming video services—or launch their own—many highly competitive production pipelines are taking advantage of virtual production. For instance, using virtual production, ILM turned a few soundstages and a backlot into 2019’s The Mandalorian, bringing an alien universe with cinema-quality effects to streaming TV.² The show’s success has been cited as a catalyst for virtual-production adoption.³

There’s certainly no shortage of demand for content with spectacular—and often otherworldly—visuals. Over the past decade, adventure and action movies have become the most successful film genres, followed closely by fantasy and sci fi. Such stories are a perfect stage for CGI and visual effects, and they increasingly require virtual production capabilities that enable the impossible and transport actors and audiences into other worlds.⁴ Leading streaming video services are also filling their programming with these genres, with virtual production being central to developing effects and experiences that satisfy their demanding audiences.⁵

The COVID-19 pandemic helped underscore virtual production’s value and boosted its adoption.⁶ Closures and social distancing shifted audiences from theaters and mobile to in-home viewing just as leading media and entertainment providers were launching their streaming video services. The need for streamers and studios to fill their catalogs with original content without sacrificing quantity or quality pushed studios toward virtual production and visual effects, especially for the more popular visual effects–rich genres. The pandemic also required mostly colocated production teams to adopt remote and distributed workflows. Instead of flying people to filming locations or bringing them together on sets, studios turned to virtual production solutions that could replicate those production elements.

Advances in developing hyperrealistic video games have been a key enabler of virtual production. The Unreal Engine from Epic Games has been recognized as a pioneer in using high-performance computing, 3D modeling, and physics to enable the shift from laborious and time-consuming postproduction to photorealistic and real-time rendering on-set—advancing their own games’ capabilities in the process.⁷ Other game engine leaders, like Unity Technologies, are also enabling the convergence of 3D gaming experiences and 2D storytelling, which will likely spark more movement of intellectual property (IP), audiences, and entertainment between them. On the way to the promised land of the metaverse, the collaboration between film and gaming could unleash new forms of entertainment that mix cinematic storytelling, immersive and social gaming, and real-time broadcasts.