Goal 5 - Shader Debugging Tutorial

This tutorial will launch the debugger and show you how to debug a vertex shader.

If you are having trouble getting the debugger to work, see Goal 5 - Shader Debugging Tutorial to understand the tools you must have installed. Once you have completed the tutorial, many of the debugger features are listed in the Getting Started Debugging Shaders.

The shader debugger tutorial is divided into several tasks:

  1. Task 1: Launching Visual Studio .NET
  2. Task 2: Opening a Visual Studio .NET project
  3. Task 3: Building a Debug Project
  4. Task 4: Starting the debugger
  5. Task 5: Setting a breakpoint
  6. Task 6: Viewing the disassembly code
  7. Task 7: Setting conditional breakpoints

Task 1: Launching Visual Studio .NET

Launch Visual Studio .NET (either version 2002 or 2003).

If neither of these versions is available, you need to install Visual Studio .NET. Go to Setting Up the Tools for install instructions.

Task 2: Opening a Visual Studio .NET project

  1. From the File menu, click Open Solution.
  2. In the file open dialog box, open the file BasicHLSL which is installed by the SDK into the following default location: 
    C:\DX90SDK\Samples\C++\Direct3D\BasicHLSL\BasicHLSL

    If you cannot find BasicHLSL you may need to install the latest version of the DirectX SDK. For instructions, see Install DirectX9.

Task 3: Building a Debug Project

  1. From the Build menu, click Configuration Manager to make sure Debug is selected.
  2. Remove the first comments in the following two lines: 
    //#define DEBUG_VS   // Uncomment this line to debug vertex shaders 
//#define DEBUG_PS   // Uncomment this line to debug pixel shaders

    The lines should now look like this:

    #define DEBUG_VS   // Uncomment this line to debug vertex shaders 
#define DEBUG_PS   // Uncomment this line to debug pixel shaders

    This sets the correct shader flags so that the shaders can be debugged.

  3. From the Build menu, click Build Solution to build the debug project.

Task 4: Starting the debugger

From the Debug menu, click Direct3D then Start With Direct3D Debugging to launch the debugger.

Visual Studio .NET window with Direct3D Debugging selected

This is also available with shortcut keys CTRL+ALT+F5.

Task 5: Setting a breakpoint

A breakpoint can be set in the code window.

  1. From the File menu, click Open.
  2. Select the BasicHLSL.fx file.
  3. Once the file has opened, click on a line of code (line 92 in this example) where you want to break and press the F9 key.

    This is what you will see when the debugger breaks. The breakpoint is marked by the filled circle to the left of line 92.

    You can now step through one line of code at a time by hitting F10. You can check the contents of any variable or register by placing the mouse over the variable and waiting for the value to pop up.

See Viewing Register Contents or Viewing_Device_State for more options for viewing state.

Task 6: Viewing the disassembly code

Disassembly code is a combination of the HLSL shader statements, and the assembly instructions generated by the HLSL compiler. From the Debug menu, click Windows then Disassembly to open the disassembly window.

With the shader paused at the breakpoint, you can hold the mouse over individual registers to see the contents, or even drag and drop a register into the watch window to look at the contents.

Task 7: Setting conditional breakpoints

Once a breakpoint is set, you may need a mechanism for stopping at this breakpoint only if a certain condition is met; for example, after some number of vertices have been processed, or only if a certain normal vector value is seen. This is called a conditional breakpoint because the break only occurs on this condition. Here is an example of how to set one.

  1. First right click on the breakpoint and select Properties.
  2. When the Breakpoint Properties dialog box opens, click the Condition... button.

  3. In the text box on the Breakpoint Conditions dialog box, enter an expression that evaluates to a Boolean result.

The conditional expression in this example is (Output.Diffuse.a == 1), which means that the debugger will only break on line 92 when the alpha component of the diffuse color is 1. (Since we set alpha to this value, this will always cause a break.) Conditional breakpoints offer a powerful method for fine tuning breakpoints, so that you can look at a particular subset of conditions while debugging.

See Getting Started Debugging Shaders for more options for setting breakpoints and for debugging shader and Direct3D code simultaneously.