NERF – Neural Radiance Fields

Making narrative film just keeps getting easier. What once took a studio is now within reach of the dedicated hobbyist. And Neural Radiance Fields are making it a dramatic step easier. The guys from [Corridor Crew] give an early peek.

Filming and editing have reached the cell phone and laptop stage of easy. But sets, costumes, actors, lighting, and so on haven’t gotten substantially cheaper, and making your own short film is still a major project.

Enter 3D graphics. With a good gaming laptop, anybody can make a photorealistic scene in Blender and place live action actors in it. But it takes both a lot of skill and work. And often, the scene you’re making is available as  a real place, but you can’t get permission to film or haul actors, props, crew, and so on to the set.

A new technology, NERF, for “NEural Radiance Fields”, has decreased the headaches a lot.  Instead of making a 3D model of the scene and using that to predict what reaches the camera, the software starts with video of the scene and machine learns a “radiance field” – a model of how light is reflected by the scene.

If you use the radiance field to predict the light that falls on a 3D model, the software can render the 3D model as if it was lit inside the scene. So if your actor stands near a red wall, the red reflection will show on their face.

The result is dramatic – live video of actors converted to 3D models by photogrammetry and dropped into the radiance field look like live action video shot on set. 3D models dropped into the scene look eerily real. Camera motion from tracking the actor’s video can be applied to a camera in the radiance field, so the camera can move during the shot.

The elements are CG objects, so they can be moved or scaled. In the video they use this to insert an actor as a giant towering over a warehouse. You can adjust camera motion after the fact, so no more shaky camera moves or regrets.

It’s a powerful new tool for low budget film makers. If you like VFx this is a real advance. NERF is new, but we’ve covered photogrammetry many times, including this neat “donut” version of a turntable.

---