How do you use custom depth as a mask in unreal?

Custom Depth in Unreal Engine is a powerful tool for creating masks that enable a wide array of visual effects, from subtle outlines to complex selective post-processing and advanced rendering techniques. It allows you to render specific objects into a separate depth buffer, which can then be sampled and manipulated in your materials.

How to Use Custom Depth as a Mask in Unreal Engine

To utilize custom depth as a mask, you'll need to enable the feature, assign objects to render into the custom depth buffer, and then access and process this data, typically within a Post-Process Material.

1. Enabling the Custom Depth Buffer

The first step is to ensure the Custom Depth buffer is active in your project settings. This allocates the necessary resources for Unreal Engine to render objects into this special buffer.

Navigate to Edit > Project Settings.
In the left-hand panel, under the Engine section, select Rendering.
Scroll down to the Postprocessing category.
Locate the Custom Depth-Stencil Pass setting.
Set it to Enabled if you only need depth information, or Enabled, with Stencil if you plan to use the custom stencil buffer for more refined masking (e.g., multiple mask IDs).

2. Assigning Actors to Custom Depth Pass

Once the Custom Depth buffer is enabled, you need to specify which actors in your scene should contribute to it. Only objects explicitly enabled for Custom Depth will appear in this buffer.

Select the desired Static Mesh Actor, Skeletal Mesh Actor, or other renderable components in your Level Editor.
In the Details panel for the selected actor, search for "Custom Depth".
Check the Render CustomDepth Pass checkbox.
(Optional, if using Stencil): If you set Custom Depth-Stencil Pass to Enabled, with Stencil, you can also set a Custom Depth Stencil Value (an integer from 0 to 255). This allows you to assign unique IDs to different groups of objects, creating multiple distinct masks from a single Custom Depth pass.

3. Accessing Custom Depth in Materials (Post-Process Materials)

The most common and flexible way to use Custom Depth as a mask is within a Post-Process Material. These materials are applied to the entire screen after all other rendering has occurred, giving you access to various scene buffers, including Custom Depth.

Creating a Post-Process Material

In your Content Browser, right-click and select Material. Give it a descriptive name (e.g., M_CustomDepthMaskEffect).
Open the newly created material.
In the Details panel (under the Material category), change the Material Domain to Post Process.

Sampling the Custom Depth Buffer

Inside your Post-Process Material, you'll use the SceneTexture node to sample the Custom Depth buffer:

Right-click in the material graph and search for SceneTexture. Add this node.
In the SceneTexture node's Details panel, set the Scene Texture Id to CustomDepth.
The output of the SceneTexture node for CustomDepth (specifically the R channel for normalized depth, or RGB for raw depth) will serve as your mask. Objects rendered into Custom Depth will have a depth value, while areas not rendered into Custom Depth will typically be black (0).
You can then use standard material nodes (e.g., Subtract, If, Lerp, Multiply, Power, Desaturate) to refine, invert, or manipulate this mask to achieve your desired visual effect. For example, 1 - CustomDepth can invert the mask, making objects rendered into custom depth transparent and everything else opaque.

Practical Applications and Examples

Custom Depth masks unlock a vast range of visual effects and capabilities in Unreal Engine:

Object Outlines: By sampling the Custom Depth buffer at slightly offset screen positions and comparing the depth values, you can detect edges of objects rendered into Custom Depth and draw an outline.
X-Ray / See-Through Effects: Make specific objects visible through walls or other geometry. This is achieved by rendering the desired objects to Custom Depth, and then in a Post-Process Material, blending them with a different material or color only where they would normally be occluded.
Selective Post-Processing: Apply specific visual effects (like blur, color grading, or stylized filters) only to particular objects or groups of objects. For instance, you could blur only the character while keeping the background sharp.
Custom Culling/Occlusion: Implement unique culling logic where objects within or outside a certain volume (defined by Custom Depth) are hidden, faded, or otherwise altered.
VST Masking: As demonstrated in advanced virtual production workflows, you can utilize the custom depth buffer to create precise Visual Studio Technology (VST) masks. This enables accurate blending and compositing of virtual elements with live footage or other rendered layers, ensuring correct depth relationships and seamless integration.
Interaction Highlighting: When a player character or cursor hovers over an interactive object, that object can be rendered into Custom Depth, and a highlight effect applied via a Post-Process Material.

Leveraging Custom Stencil (Advanced)

When Custom Depth-Stencil Pass is set to Enabled, with Stencil, you gain access to an additional 8-bit buffer that runs alongside Custom Depth. This allows for even more control:

Assigning Stencil Values: In an actor's Details panel, alongside checking Render CustomDepth Pass, you can set a Custom Depth Stencil Value (0-255).
Sampling Stencil: In your Post-Process Material, use the SceneTexture node with Scene Texture Id set to CustomDepthStencil. The stencil value is typically accessible via the Stencil output pin of the SceneTexture node, or sometimes packed into the G or B channel.
Creating Multiple Masks: By comparing the sampled stencil value to specific IDs using If nodes, you can create multiple, distinct masks for different object groups, all from a single Custom Depth-Stencil pass. This is incredibly efficient for complex scenes.

By mastering the use of Custom Depth and Custom Stencil, you unlock a powerful layer of control over your Unreal Engine projects, enabling sophisticated visual effects and precise rendering techniques.