This pipeline that I built was designed to mix traditional visual effects and AI-generated images.

The idea is to be able to create the look of 3D assets as efficiently as possible but with as much control as possible. I don’t want the AI to do all the job for me; I want to be able to design and build a 3D world with all the visual effects techniques that I know and I’m comfortable with. But I want to use AI to speed up certain steps of the process.

At the end, I’m creating 3D assets that I can place in a virtual scene to create shots. I’m able to reuse assets to keep consistency between shots and sequences. Everything goes through a render engine at the end, where I can create materials and looks, and change the lighting at any given time. But thanks to these hybrid techniques, I can get to the final result much quicker, without losing any control and being able to accommodate feedback from a director.

This is how the final 3D asset looks once rendered. On the left, references from The Last of Us HBO series. The final asset needs to hold up to the standards that we have in VFX productions.

I wanted to start as minimalistic as possible, so this is my 3D asset. Basically a cube with a couple of extrusions to make it look like a building.
Once I had this building, I just did a simple simulation in Houdini to destroy it, creating more complex shapes. It is still extremely basic, but it should be good enough to create a mid-background asset that can be used in a film or series. This will be one of many that will be needed to create a library that can be used to populate a post-apocalyptic city.

Below you can see what we call clay renders, in gray. Basically, I’m creating a camera for each side of the building. Then, the render is used in ComfyUI together with control nets for the AI to understand the shape of the building. And on the right side, you can see the final camera projections that later will be dictating the look of the 3D asset.

This is the Comfyui workflow that creates the images for matte painting by inputting the clay renders.

Camera projection setup in Blender. As I want to create library elements for the city, I’m baking the matte paintings to UV space, so I can simply connect the textures to the shaders and have a very simple setup to drive the entire look of the city elements, with shaders for different materilas, like bricks, metal, plaster, etc.

Back in Houdini, I can keep working on my assets, in this case scattering ivy and overgrown vegetation to match the look of The Last of Us.

To see how well this technique would scale, I also worked on bigger buildings. The approach is the same, but to get enough resolution in the textures, I’m creating more camera projections. For bigger buildings like this, instead of 4 projections, I’m dividing the building into 3 sections and creating 12 projections in total. The number of projections depends on how close the building is going to be to the camera. For a hero building, more projections would be needed; for background assets, 1 to 4 projections would suffice. The important thing here is the amount of flexibility that this workflow provides to visual effects artists.

Bottom section with 4 camera projections.

Middle section with 4 camera projections.

Final renders of the building under different lighting conditions.

Quick test duplicating the same building a bunch of times. Because this is a real 3D render, lighting and materials react as realistic as possible, way better than traditional matte painting where we usually get this painterly look that doesn’t work that well.

Close up to demonstrate that there is enough detail in the texture maps, even if this is just a rough test, it could be better if done properly.

Below another test, in this case a set extension. The bottom floor is a photogrammetry scan of a real building, and I did the set extension using the same techniques to complete the upper floors.

Final renders below, different camera distances.

Everything that I showed so far was by modelling the buildings by hand. But there is a world where we can also model the building using AI. This wouldn’t hold up for close-ups, but for a project like The Last of Us, where potentially hundreds of buildings would have to be created, we can AI-generate geometry for buildings and use them as background assets.

But again, this is a hybrid pipeline, so creating shaders, scattering vegetation, and lighting the buildings is still the best way to go. But you get there much faster.

All the images below are AI generated, the geometry is AI generated and the textures are created using the same techniques as before. Then taken into Houdini for lookdev, assembly and lighting.

Because buildings worked really well for these tests, I also wanted to generate other elements for the city. The Last of Us also needs a lot of abandoned cars.

These cars were generated using the same techniques: AI for geometry, AI for camera projections. Then I set up a procedural workflow for remeshing the cars to get a better topology, and then I baked the camera projections into UV space to get the final textures. Then I put everything together in Houdini to get the final look and get the asset ready for shot production.