How the Stencil Buffer Works

Direct3D performs a test on the contents of the stencil buffer on a pixel-by-pixel basis. For each pixel in the target surface, it performs a test using the corresponding value in the stencil buffer, a stencil reference value, and a stencil mask value. If the test passes, Direct3D performs an action. The test is performed using the following steps:

1. Bitwise AND the stencil reference value with the stencil mask.
2. Bitwise AND the stencil buffer value for the current pixel with the stencil mask.
3. Compare the result of step 1 to the result of step 2 using the comparison function.

Written in pseudocode, these steps would look like this:

(StencilRef & StencilMask) CompFunc (StencilBufferValue & StencilMask)

Where StencilBufferValue is the contents of the stencil buffer for the current pixel. This pseudocode uses the & symbol to represent the bitwise AND operation. StencilMask represents the value of the stencil mask, and StencilRef represents the stencil reference value. CompFunc is the comparison function.

The current pixel is written to the target surface if the stencil test passes, and ignored otherwise. The default comparison behavior is to write the pixel no matter how each of the bitwise operations turn-out (D3DCMP_ALWAYS). You can change this behavior by changing the value of the D3DRENDERSTATE_STENCILFUNC render state, passing one of the members of the D3DCMPFUNC enumerated type to identify the desired comparison function.