vfx doesn't start in post, it can start before the screenplay is even locked

Preface: let's talk pipeline.

I suppose now is a good time to take a pause on going through all the landmarks, and actually talk about what is a VFX pipeline. We keep saying this movie invented the pipeline, that movie re-invented the pipeline, another movie established the modern day VFX pipeline. When we say pipeline, what are we referring to? Pipeline is just a fancy way to say “standard procedure”, where what goes in, and what comes out the other end of the step, and it goes into the next step down the line. Sort of like an assembly line, but each step takes a bit longer and a bit more artistry than an assembly line. 

So the pipeline usually goes like this. Typically, the R&D, which stands for research and development phase of the VFX production happens before the screenplay is even locked. A visual effects producer will read the not-yet-locked screenplay, and do a breakdown with the VFX supervisor. It’s like taking a highlighter and highlighting anything in the screenplay that might require visual effects, such as a giant robot, or a dinosaur, or environment extension, fake fire, snow, lightning, or some sort of graphics. They usually fall under the categories of creatures, special effects, hair, movement, environment, etc etc. Even though the screenplay is not locked, the major elements of the story have been decided, the director has some idea about how he or she wants the elements to look like. The VFX houses can start researching and doing mockups to present to the director, however, this is not common. Usually the R&D process has its design done by concept art, and not the VFX department, but I did work on movies that had VFX artists to design effects from the get-go, before the movie even started shooting.

For VFX-heavy movies, people typically do what's called a previz, short for pre-visualization. It looks like something like this. It's like a moving storyboard for where VFX will be, where live action will be, and what the camera movement is, if the ceiling is high enough for the cranes or the given movement that sort of thing. There's also techviz and postviz. You can think of it as different storyboards for different stages of production.

Typically, this is done with a game engine, Unreal, or Maya that sort of things. Back in the day, the biggest company / previz provider was this studio called "The Third Floor" (https://thethirdfloorinc.com/what-we-do/visualization). Looks like the company still exists and is flourishing, unlike technicolor, which went bankrupt. And Rhythm and Hues, which also went bankrupt a long time ago.

Previz costs money, which is why some movies / studios are hesitate to hire previz vendors, some think it's not a necessary step in the production pipeline, usually the ones that are relatively new to VFX-heavy productions. Though they quickly realize, that not having previz might end up costing more, especially when you are on a tight schedule. You will come to realize that without a basis for each department to discuss how each shoot should go, communication becomes impossible. Everyone has a different idea of how it should go, how it will go, lots of time is wasted on set on trying to understand each other's vision alone. Previz, techviz, postviz solve these problems. It provides a workable blueprint for production meetings, if people have different ideas, they can visualize it, and test it, to make sure this will go as planned on set, when all departments are the clock, time is money.

After discussing the storyboards, previz, and production plan, all departments arrive at some what of a common ground, then the shooting can start. Typically, a on-set VFX supervisor is hired, to make sure everything needed in post gets collected on set. There's quite a list of things we need to collect. First, there's lens grids / distortion charts for accurate undistort/redistort. What this means is that each lens we use to shoot on set has its own distortion, that's just one of the lens' properties. When you shoot something with that lens, and you add a CG element to the same shot, you want the CG element to have the same distortion as the things shot on the lens. This is why you must record the lens distortion, so you can apply it in post, onto the CG elements. This is same as the idea from the last post, where the water tentacle had to morph into a lady's face, and ILM used a grid projected onto the actress's face to record her face shape. Same idea for recording the lens distortion, using a grid. This is usually done before shooting starts, VFX department will coordinate with the camera department to get all their lens.

HDRI / chrome-gray ball for lighting reference is a strange one when I first started working on set, I quickly learned how genius of an idea this is. Light and objects reflections and refractions with their surrounding lights are the most difficult thing to imitate. Just like the lens distortion, we want the lighting of each CG element to match the exact lighting environment that was on set. Therefore, we must record the lighting environment on set somehow. We need all the lighting information on a single plate, so we can know how bright every single point is compare to any other point in the same environment, it has to be on one picture. There's a easy way to do this, is to take a 360 picture of a perfectly reflective ball, we call chrome ball, or chrome-grey ball, depending on if half of it is grey or not (https://www.vfxball.com).

Before each take, we take a photo of the chrome/grey ball, which records the lighting information of its environment in that take, before the camera rolls. I'm going to ask Claude to help me explain this part. So, HDRI stands for High Dynamic Range Imaging. The human eye can adjust to see detail in both shadows and bright highlights at the same time. A standard photograph can't — it exposes for one range and clips everything outside it. An HDR photograph solves this by taking the same shot at multiple exposures and merging them, capturing the full spectrum of light in a scene from the darkest shadow to the brightest highlight. On set, a chrome ball and a grey ball are placed at the position where a digital element will live in the shot. The chrome ball is mirror-reflective — it captures a 360-degree panorama of the entire lighting environment in a single sphere, including every light source, every color, every direction light is coming from. The grey ball is diffuse — it shows how that light falls on a neutral, non-reflective surface. Together, photographed in HDR at multiple exposures, they give lighting artists a complete map of the real-world illumination at that exact position in space.When a lighting artist later needs to place a digital creature in that shot, they load the HDR as the light source in their 3D environment. The creature gets lit by the actual light that was on set that day. Not an approximation of it. The actual thing. That's the difference between a digital element that feels like it belongs in the frame and one that feels like it was photographed somewhere else and pasted in.

Survey / lidar / photogrammetry for set geometry are three different methods for capturing the geometry of a real-world space and bringing it into the computer as a three-dimensional structure. They're often used together.

A survey is the most basic version — measuring key dimensions and positions of a location manually, recording distances, heights, angles. Fast and cheap, but limited in resolution and detail. For surveying outdoor sets, we used DJI drones to fly over them, and build a map from it to know the exact size and location of each section, so we can recreate it as a CG environment, for the shots we need these to be CG.

LIDAR uses laser pulses. A scanner fires millions of laser beams in all directions and measures the time it takes for each one to bounce back. From those measurements, it builds a point cloud — a dense three-dimensional map of every surface in range, accurate to within millimeters. Walk a LIDAR scanner through a practical set and you have a precise digital copy of that set that artists can navigate, model against, and place digital elements inside. A digital character running through a corridor can cast the correct shadow on the real geometry of the floor. A digital vehicle can navigate a real street with the correct physical scale.

Photogrammetry works differently — it uses photographs. You photograph an object or environment from dozens or hundreds of angles, and software analyzes the overlap between those images to reconstruct the three-dimensional geometry. It's slower to process but can capture surface texture and color in a way raw LIDAR doesn't. On set, photogrammetry is commonly used for props, vehicles, and actor bodies — anything that needs to become a digital double with accurate surface detail.

For a large-scale VFX production, a location might get all three: survey measurements for key dimensions, LIDAR for full spatial geometry, and photogrammetry for surface detail. The result is a digital environment that artists can work inside with confidence that what they build will line up with the footage.

We also had witness cams for performance capture or complex motion. Witness cameras are any camera running on set that isn't the primary production camera. Their job is documentation. They record the full context of a shot — what the set looked like, where everything was positioned, how the actors moved, what the lighting rig was doing, what was happening outside the frame. I believe I once used a pretty simple Nikon as a witness camera.

In standard VFX shooting, witness cameras give post teams the visual evidence they need to reconstruct decisions made months earlier. But on shots involving performance capture — actors in motion capture suits performing alongside live-action footage — witness cameras become structurally essential. The motion capture data records the body movement, but it doesn't capture performance nuance, spatial relationship, eyeline, or the physical reality of how the actor was interacting with the set and the other performers. Witness cameras do. Animators and compositors use that footage frame by frame to keep the digital performance grounded in what actually happened on the day.

On complex motion — wire work, stunt sequences, multi-camera action with elements that will be replaced or augmented — witness cameras are the paper trail. If a rig needs to be removed, the witness cameras show the compositor what was behind it. If a stunt double needs to become a hero actor in post, the witness cameras show exactly how that body moved through space. Without them, post teams are reconstructing reality from insufficient evidence.

Other than movement, size, geometry, lighting, color is also vital. Color charts to keep the pipeline anchored to consistent transforms. A color chart — the most common being the X-Rite ColorChecker, sometimes called a Macbeth chart — is a physical card printed with patches of precisely defined, internationally standardized colors. It gets photographed on set at every significant lighting change. Its purpose is to anchor the entire post pipeline to a consistent, verifiable color truth.

Camera sensors don't all see color the same way. Different cameras, different lenses, different lighting conditions, different log formats — all of these introduce variation in how color is recorded. When you're combining live-action footage with digital elements rendered in a 3D environment, and grading all of it in a DI suite, every link in that chain needs to speak the same color language. The color chart is how you establish that language.

When a digital intermediate colorist or a compositor receives footage, they can photograph the chart under known conditions, compare it to the standardized values, and derive a precise mathematical transform that corrects the camera's response to match a known baseline. That transform then gets applied consistently across every shot from that camera in that setup. Digital elements get rendered to match the same baseline. The result is a pipeline where color decisions made by the director of photography on set survive intact all the way through to the final grade — instead of drifting, accumulating error at every handoff, and forcing artists to make corrections by eye that should have been solved mathematically from the start. More on this in the next post.

🔥