I built a pipeline to create matte paintings and set extensions for shots where the camera doesn’t move much, or there is very little parallax shift.

The workflow is model-agnostic, but I’m using Flux Klein on these examples. Everything is built in ComfyUI and runs locally on my machine. Producing high-quality images that can later be used for compositing using traditional tools and techniques that we have been using in VFX for years.

This is a contact sheet with shots from The Last of Us HBO series. I used these as references to create a couple of shots that could live in this universe. The second shot in the first row and the first shot in the second row are the ones created to test this workflow.

I’m using Flux Klein and control nets, including depth, canny and masks to work on each building independently.

This shot uses the same technique designed for a locked-off camera, but in this case, the camera moves.

With the AI-assisted workflow, I’m creating matte paintings that I can later composite in Nuke with camera projection techniques. Edges are a bit off as I’m not the best compositing artist, nor do I spend enough time on this, but hopefully it illustrates the idea.

What you are seeing here is the original live-action plate.

I built a tiny tool that uses photogrammetry under the hood to automatically track cameras. It works very well, but if the camera moves very fast or there is a lot of motion blur or depth of field, this tool won’t work. But there are plenty of shots that I can track completely automatically and with a lot of accuracy using this tool.

Tracking results visualized in Nuke as a pointcloud.

Below is the workflow that produced the matte paintings in ComfyUI, followed by some of the layers created to compose the final matte painting assembled in Nuke.
I work on each element individually to have more control.

Below the projection setup in Nuke. This is regular projection techniques that we usually do in visual effects to solve many shots, specially environment work and set extensions.

Before and after.

Final shot.

This is another test; this time it is a bigger environment, a wide shot. Same technique, using the same steps along the way.

Final shot.