Virtual Production with Frame Remapping

We’re pleased to be the first in the industry to introduce an exciting new feature that will be a game-changer for virtual production: Frame Remapping.

Frame Remapping offers two major benefits to virtual production:

• Frame Remapping allows you to display different content streams a frame apart so that several cameras* can capture different background perspectives on the same LED screen simultaneously
• Interleaving content can be used to interleave solid colour frames with any 8-bit colour – whether it’s blue or green to allow post-production keying or black to reduce motion blur

For virtual production shoots using complex CGI content, most setups use one camera per shot. Occasionally it’s possible to use several cameras, but it depends on how the original content was captured. Real-world content that was captured in 360 (e.g. from a car) can have several cameras shooting from different angles, but running CGI content from a game engine usually only renders one camera’s perspective within a frustum. It’s locked to the camera tracking so it moves realistically with the live-action.

Frame Remapping opens up a world of new possibilities when shooting with LED screens, allowing for at least *two cameras per setup (up to four depending on frame rate you’re shooting at) by genlocking each camera to the same frame rate as the content coming in. It’s easy to setup too since the different content is automatically split up and stitched together by your Tessera processor, which can display it in any order you want.

In the example we have shot, we are using two cameras running at 50fps with a shutter angle of 1/100, the processor is receiving content at 50hz and driving the LED @ 200hz, using frame remapping to sample the left half of the content for two frames and then the right half for two frames.

Camera 1 is in sync with the processor, however, the camera 2 has had a genlock phase offset applied, resulting in camera 1 capturing one portion of the content, and camera 2 capturing another portion of the content. It is essential that phase offsets are applied to ensure each camera is picking up the correct frames of content.

The Tessera S8 and SX40 processors can also generate frames of any 8-bit colour to be stitched between the frames of your content, e.g. blue or green frames. The benefit of interleaving these plain coloured frames it that you can simultaneously shoot your generated content and a green screen background on your LED at the same time with a single camera.

This gives the freedom to make changes much more easily during the post-production process, allowing you to shoot the live-action with the confidence that any of the pre-shot background content can be completely swapped out later. Editing footage that’s been shot on LED can be challenging and interleaved green frames make editing out unwanted content in post much easier.

Interleaving green frames can also be used for multi-cam setups if you have problems with your tracking, or you only have the right perspective for one camera on the day. Simply edit the green screen frames at a later date when you have the additional perspective ready.

To have something upstream doing this would add a considerable amount of latency, but we’re able to offer Frame Remapping alongside all of our other Tessera processing features at just 2 frames of end-to-end latency and one frame end to end when using Ultra Low Latency mode. We want to keep opening doors for creatives and making virtual production workflows even more powerful and flexible for filming, and Frame Remapping does just that.