[This is preliminary documentation and subject to change.]
To enable the debugging of a custom action, set the environment variable MsiBreak to the name of the designated action. For DLL custom actions, a user breakpoint is called just prior to the entry point call. If the current session is not running under a debugger, the standard exception dialog box is invoked with an option to debug. This brings up the debugger that is registered for Just In Time debugging.
To enable source-level debugging, the program database (PDB) file must be available, either at the original build location for the DLL or in the directory where the DLL is now running. Custom actions stored in the Binary table are executed from the system Temp directory. The mechanism is similar with EXE custom actions except that the debug break occurs just prior to the CreateProcess call. You cannot step into the EXE code, and you cannot attach the debugger to the process until it is running.