PRB: WOW System Hooks Limit Number of Concurrent MFC Apps

The information in this article applies to:

• The Microsoft Foundation Classes (MFC) included with: Microsoft Visual C++ for Windows, versions 1.0, 1.5, 1.51, and 1.52 when running under the WOW Subsystem of Windows NT 3.1 and 3.5

SYMPTOMS

There is a limit to the number of 16-bit MFC applications that can be run under the WOW subsystem of Windows at one time. You will see one of two symptoms depending on whether the application manifesting the problem is a release or debug build:

• If the application manifesting the problem is a release build, it will cause a general protection (GP) fault before the main window is displayed.

  -or-

• If the application manifesting the problem is a debug build, an assertion failure will occur, ASSERT(pWnd != NULL), in the AfxWndProc function. Each of the following lines denotes where assert can be found in WINCORE.CPP:

  MFC 2.00 asserts at line 161. MFC 2.50 asserts at line 210. MFC 2.51 asserts at line 210. MFC 2.52 asserts at line 210.

CAUSE

MFC uses a WH_CALLWNDPROC hook during the creation of a window to connect a CWnd object to the window. If the hook cannot be set, the window is not properly added to MFC's permanent handle map. When the window receives a message, MFC attempts to retrieve a pointer to the corresponding CWnd object from the permanent handle map. Because the window is not in the map, a NULL pointer is retrieved. In release builds, the pointer is used, causing an access violation, which results in a GP fault. In debug builds, the pointer is trapped by the assertion.

The WOW subsystem of Windows NT provides a limited number of system hooks. Windows NT version 3.1 has 16 hooks available. Windows NT version 3.5 has 32 hooks available. When the hooks are exhausted, an attempt to set a hook fails without reporting an error, so MFC is unable to detect that the hook was not set and recover gracefully.

MFC applications are particularly susceptible to exhausting the available system hooks because they typically install two hooks (one for F1 help, the other for gray dialogs) at application startup. These two hooks last for the lifetime of the application. Applications that use the autosubclassing feature of CTL3D consume an additional hook.

RESOLUTION

To run more 16-bit MFC applications under the WOW subsystem, you must reduce the number of hooks consumed by each application.

The following are three common uses of hooks by MFC applications that you need to prevent if you want to execute more applications:

• Providing F1 Help.
• Setting the background color of dialogs.
• Using CTL3D to generate 3D effects for controls and dialogs.

Removing the F1 Help Hook

The F1 Help hook is installed whether or not your application uses F1 Help. If you do not support F1 help, you can remove this hook by following these steps:

1. Add the MFC source directory to your Include Files path, typically <drive>:\msvc\mfc\src.

2. Include AUXDATA.H and the function definition for _AfxMsgFilterHook so that it can be used by your CWinApp derived class:

      #include "auxdata.h"
   
      extern LRESULT CALLBACK AFX_EXPORT
        _AfxMsgFilterHook(int code, WPARAM wParam, LPARAM lParam);
   
   

3. At the beginning of your derived CWinApp's InitInstance() function, add the following code to unhook the msg filter hook in your application:

      // Remove system hook for F1 Help
      if (_afxHHookOldMsgFilter != NULL) {
         if (!afxData.bWin31) {
            ::UnhookWindowsHook(WH_MSGFILTER,
                             (HOOKPROC)_AfxMsgFilterHook);
         }else{
            ::UnhookWindowsHookEx(_afxHHookOldMsgFilter);
         }
         _afxHHookOldMsgFilter = NULL;
      }
   
   

Removing the Dialog Background Color Hook

The Dialog Background Color hook is only installed if you call CWinApp::SetDialogBkColor(). AppWizard places a call to SetDialogBkColor() at the start of your CWinApp derived classes InitInstance() function. To remove the hook, remove this and all other calls to SetDialogBkColor() from your application.

NOTE: If you are using the CTL3D library, it will paint the background of dialogs, so you do not need to call SetDialogBkColor().

Removing the CTL3D Hook

If you are using the autosubclassing feature of CTL3D to add 3D effects to your application's controls and dialogs, a hook is used. You can still use CTL3D without consuming a hook if you explicitly subclass each dialog by using the Ctl3dSubclassDlgEx() function. See the CTL3D documentation for more information on how to explicitly subclass dialogs using CTL3D.

STATUS

This behavior is by design.