Junjie Chen | Integrated Marketing Portfolio

When entering the virtual production studio for the first time, many people will have a question:

Isn’t it just a background video playing on the LED wall? Why is the final picture really like an actor standing in another city, another coast, or even a completely non-existent world?

The answer is: the LED background in virtual production is not an ordinary video, but a digital scene that will be adjusted in real time with the change of the camera.

The LED wall is not “putting the background”, but building the space.

In traditional shooting, if you need to present different scenes, the team often has to go to the real location to shoot, or let the actors stand in front of the green screen, and then synthesize the background in the later stage.

Virtual production presents the digital scene directly on the LED wall.

This digital scene may be a hotel, a future street, a desert, or a completely fictional world created by 3D software. The actor stands in front of the LED wall, and the camera can directly shoot the picture of the combination of the characters and the background.

But the key to making the picture really have a sense of space is not the LED wall itself, but camera tracking and real-time rendering technology.

As the camera moves, the digital scene will also change synchronously.

In the virtual production studio, the position, angle and movement trajectory of the camera will be recorded in real time by the tracking system.

When the camera moves to the left, the digital scene on the LED wall will also recalculate the picture according to the position of the camera; when the camera moves forward, the objects in the background will produce corresponding distance changes.

This process is similar to the visual feeling in our real life.

For example, when we approach a window, the nearby buildings outside the window will move faster, while the distant sky and mountains will change more slowly. This relative movement between different distances is called “parallax”.

The virtual production system will simulate this parallax in real time, making the two-dimensional picture on the LED wall look like a real three-dimensional space in the camera.

That is to say, the camera is not taking a fixed background picture, but watching a digital world that will change with the lens.

Real-time rendering allows the “post-production picture” to appear on the scene in advance.

Traditional film and television production usually waits until the shooting is over before the scene synthesis and visual effects can be completed in the later stage.

Virtual production brings the original post-production part to the shooting site in advance through the real-time rendering engine.

During shooting, the director and photographer can see the effect close to the final film directly on the monitor. If you need to adjust the scene, you can immediately change the weather, time, lighting and building position in the digital environment.

For example, the same scene can change from dawn to dusk in a few minutes, or switch from the streets of New York to seaside resorts. The team does not need to wait for the weather to change, nor does it need to change the shooting location frequently.

This real-time adjustment ability makes the creative discussion no longer stay in words, sketches or imagination, but can directly turn into pictures that can be seen on the spot.

The LED wall also assumes part of the real lighting function.

There is another very important difference between virtual production and green screen shooting, that is, the LED wall itself can provide real light for actors and objects.

Assuming that the background is a sunset, the orange and gold light emitted by the LED wall will naturally shine on the actor’s face, costumes and objects in the scene.

If you take pictures of glass, cars, metal or other reflective materials, the environment on the LED wall will naturally appear on the surface of the object.

In traditional green screen shooting, these rays and reflections usually need to be remade in the later stage. With a little improper handling, the characters and backgrounds are easy to look like two different pictures.

The LED wall allows the characters, props and digital environment to share the same set of light at the shooting stage, making the final picture more natural.

Behind an LED wall, there is the synchronous cooperation of multiple links.

It seems to be just a camera shooting, but the virtual production site actually requires multiple systems to run at the same time:

Digital scenes need to be designed and built in advance, camera parameters need to be matched with the rendering system, tracking equipment needs to accurately judge the lens position, and LED images, on-site lighting and photographic exposure should also be coordinated.

As long as there is a deviation in one of the links, it may affect the realism of the picture.

Therefore, virtual production is not simply “shooting in front of the screen”, but integrates photography, lighting, real-time rendering, scene design and technical control into the same production process.

From “post-synthesis” to “on-site completion”

The most important change in virtual production is that digital content no longer exists only in post-production.

It brings the digital scene to the actors, and also allows the director, photographer and client to see, discuss and adjust the final effect together on the shooting site.

In the virtual production process of WLAB, we hope to make the camera truly enter the digital world through the cooperation of technology and creativity, and also make the originally complex scene shooting more controllable, flexible and intuitive.

The next time you see a character standing in a magnificent landscape or future space, it may not be a real location, nor may it be an ordinary background.

It may be a world that is changing with the camera in real time.

Junjie Chen