There is a huge buzz right now around “virtual production” in filmmaking. But what is it?
Well firstly, despite what you might think, the technique of virtual production is not new. The term is quite broad but is generally used to describe the process of using in–camera visual effects in real–time so that what the director and camera operators see either on set or in live monitors, is much, much closer to the final edit, and gives filmmakers the ability to a shoot a scene which would not be possible on location or by using practical effects.
This contrasts with the prevailing VFX workflow of the last few decades which has involved using green screens behind the filming of live–action footage of actors, where the final effect is not seen until weeks, months or even years of post–production work have been completed. With Virtual Production, a large amount of the VFX work is completed ahead of the shoot, leading to a much shorter post–production process.
Virtual production could arguably be said to date back to the 1930s when projection screens were used behind static vehicles to create the impression of a location shoot with a moving backdrop. Over time many new techniques have been pioneered, using not just filmed footage but also CGI VFX. More recently the use of powerful game engines has dramatically changed what’s possible – so much so that now an entire environment and set can be created in the digital world and rendered in real–time on set.
Advanced game engine and graphics capabilities have opened the tantalising prospect too of modifying the background in real–time as the camera is moved, to create much greater realism and sense of perspective. Of course, this requires another key technology – camera tracking – which can accurately capture the exact position of the camera as it is moved or panned around the virtual set.
This technology can be combined with green screen, but it has really come of age with the advent of high resolution LED displays. It’s in no small part down to the innovative use of these displays on TV shows like the Mandalorian that the world of film and television is taking such a huge interest in this new workflow, and increasingly choosing to use LED screens for virtual production.
Productions are now also having to deal with the impact of COVID, both in terms of travel restrictions and how many personnel you can have working on a set at any given time. This means virtual production, particularly when using LED screens instead of green screens*, becomes increasingly appealing, as it gives the flexibility of taking the shoot “anywhere in the world” without having to leave the safe confines of a protected sound stage.
Virtual production with LED combines an array of different technologies working closely together in sync in order to be successful. How complex that setup is, dictates how many of the available technologies you might need. It’s a myth that you have to have a blockbuster budget to do virtual production – it’s definitely a scalable workflow!
A simple set up might just be an LED screen placed behind a set element – let’s say a train carriage window to simulate countryside flashing by. If the camera position is fixed, then camera tracking may not be necessary either.
However, if your intention is to create a large scale “volume”, (an entire set which can be generated and manipulated in real-time using graphics engines), you may need some if not all of the following:
- In-camera LED effects using LED panels to display the content that’s going to be shot on camera
- Off camera LED panels (typically a roof and/or foreground setup) to provide lighting and reflections – some of which are not possible to achieve using conventional lighting
- LED processors (like our Tessera processors) to enable the panels to correctly display the content and communicate properly with the other technologies within the pipeline
- Motion capture and camera tracking technology for setups when the camera is moving, so that the motion of the background on the LED wall looks realistic in relation to the camera and the other elements on set
- Media servers for playback of pre-recorded content
- A real-time graphics workflow tool to sync live motion VFX
- A graphics processing engine to manage both the tracking information and VFX and get it through the pipeline and onto the screen
- As well as, of course, all of the standard film set technologies (e.g. camera, lighting, set dec, art and VFX, etc)
The success of any virtual production – whether it’s a small scale “through a window” effect or a giant immersive LED soundstage where the walls, floor and ceiling are all displaying digital content – depends upon every single technology working well together. When it does, magic happens!
*If you’re interested in learning more, you might like to read our previous blog on The Pros and Cons of LED Screens as Green Screen Replacement.
Image credit: Nathan Reinds / Faber AV
