PRB: OnCtlColor Not Called When Using CTL3D.DLL

The information in this article applies to:

• The Microsoft Foundation Classes (MFC) included with:

      - Microsoft C/C++ for MS-DOS, version 7.0
      - Microsoft Visual C++ for Windows, versions 1.0 and 1.5
  

SYMPTOMS

The Microsoft Developer Network (MSDN) CD contains a dynamic-link library (DLL) called CTL3D.DLL. This DLL provides functions that allow 3-D painting of Windows controls to give applications a 3-D look. If an application uses the auto-subclassing feature of CTL3D.DLL, the OnCtlColor() handlers for CDialogs and CFormViews won't be called.

Failure to handle the WM_CTLCOLOR message can cause problems with controls, especially VBX controls.

CAUSE

When an application uses Ctl3dAutoSubclass(), a Windows hook is created to trap the creation process for dialog boxes. This hook will subclass the window procedure for the dialog box and call the Ctl3dDlgProc() function. The following are a few lines of code from the hook function as shown in the CTL3D documentation:

   BOOL fSubclass;
   fSubclass=fTrue;
   SendMessage((HWND) hwndHookDlg, WM_DLGSUBCLASS,0,
               (LPARAM)(WORD FAR *) &fSubclass);
   if (fSubclass)
     {
       SubclassWindow((HWND) hwndHookDlg, (FARPROC) Ctl3dDlgProc);
     }

The new window procedure, Ctl3dDlgProc(), installed by SubclassWindow(), does everything that is needed to use CTL3D.DLL. The function is shown in the CTL3D documentation. Note that it traps messages such as WM_INITDIALOG, WM_NCDESTROY, and WM_CTLCOLOR. Below is a portion of the code for handling WM_CTLCOLOR (please see the CTL3D documentation for the full code):

   case WM_CTLCOLOR:
        (FARPROC) lpfnDlgProc = (FARPROC)
                     GetWindowLong(hwnd, DWL_DLGPROC);

        if (lpfnDlgProc == NULL) {
           hBrush = Ctl3dCtlColorEx(wm,wParam,lParam);
           }
        else {
           hbrush = (HBRUSH) (*lpfnDlgProc)(hwnd,wm,wParam,lParam);
           if (hbrush == (HBRUSH) fFalse ||
               hbrush == (HBRUSH) 1)
                hbrush = Ctl3dCtlColorEx(wm, wParam, lParam);
        }

        if (hbrush != (HBRUSH) fFalse)
            return (LRESULT) hbrush;

RESOLUTION

There are several techniques that can be used to prevent the problem of not receiving the WM_CTLCOLOR message:

• Do not use the auto-subclassing feature of CTL3D; instead, subclass each dialog box using the technique described in the CTL3D documentation. You must use Ctl3dSubclassDlg() rather than Ctl3dSubclassDlgEx(); otherwise, the same problem will occur.

  -or-

• Continue to use the auto-subclassing feature; however, for CDialogs or CFormViews that must receive the WM_CTLCOLOR message, trap the WM_DLGSUBCLASS message and set the value pointed to by lparam to CTL3D_NOSUBCLASS. Then, subclass the dialog box manually [using Ctl3DSubclassDlg()] just as you would if you were not using auto- subclassing but wanted 3-D dialog boxes. See the section below titled "Using Auto-Subclassing and WM_DLGSUBCLASS" for more information about this technique.

  -or-

• Continue to use auto-subclassing but additionally subclass the dialog box procedure in the OnInitDialog() function handler such that the new dialog box procedure that you define is called before the dialog box's regular dialog box procedure. In your new dialog box procedure, check whether WM_CTLCOLOR is sent. If it is, call the OnCtlColor() handler for the dialog box; else, call the regular dialog box procedure for the dialog box. See the section below titled "Trapping WM_CTLCOLOR with Additional Subclassing" for more information about this technique.

MORE INFORMATION

This section describes in more detail the last two techniques in the RESOLUTION section.

Using Auto-Subclassing and WM_DLGSUBCLASS

There may be times when a programmer wants to use CTL3D's auto-subclassing feature [that is, call Ctl3dAutoSubclass() in OnInitInstance()] but has a few dialog boxes that need to receive the WM_CTLCOLOR message. For those dialog boxes that need to receive the WM_CTLCOLOR message, first write a routine that will trap the WM_DLGSUBCLASS message and set the variable, pointed to by the LPARAM, to FALSE. Secondly, call Ctl3dSubclassDlg() in your OnInitDialog() and add any other code described in the CTL3D documentation that is recommended to handle messages. Note, do not use Ctl3dSubclassDlgEx(). You must use Ctl3dSubclassDlg().

To give an example of using this technique, the following is sample code one might have for the CDialog-derived class that needs to trap WM_CTLCOLOR:

   LRESULT CAboutDlg::OnDlgSubclass(UINT wParam, LONG lParam)
   {
       *((int FAR *)lParam)= CTL3D_NOSUBCLASS;
       return 0;
   }

   BOOL CAboutDlg::OnInitDialog()
   {
       CDialog::OnInitDialog();
       Ctl3dSubclassDlg(m_hWnd,CTL3D_ALL);
       return TRUE;
   }

   HBRUSH CAboutDlg::OnCtlColor(CDC* pDC, CWnd* pWnd, UINT nCtlColor)
   {
       if( nCtlColor == CTLCOLOR_EDIT)
       {
           pDC->SetBkColor(RGB(127,127,127));
           return (HBRUSH)::GetStockObject(GRAY_BRUSH);
       }
       CDialog::OnCtlColor(pDC, pWnd, nCtlColor);

       const MSG * pMsg=GetCurrentMessage();
       return Ctl3dCtlColorEx(pMsg->message,pMsg->wParam,pMsg->lParam);
   }

   void CAboutDlg::OnNcPaint()
   {
       const MSG * pMsg=GetCurrentMessage();
       ::SetWindowLong(pMsg->hwnd, DWL_MSGRESULT,
           Ctl3dDlgFramePaint(pMsg->hwnd,pMsg->message,
           pMsg->wParam, pMsg->lParam));
   }

   BOOL CAboutDlg::OnNcActivate(BOOL bActive)
   {
       const MSG * pMsg=GetCurrentMessage();
       ::SetWindowLong(pMsg->hwnd, DWL_MSGRESULT,
           Ctl3dDlgFramePaint(pMsg->hwnd, pMsg->message,
           pMsg->wParam, pMsg->lParam));
       return TRUE;
   }

Trapping WM_CTLCOLOR with Additional Subclassing

Because the CTL3D window procedure calls the dialog box procedure (not the next window procedure) after it processes WM_CTLCOLOR, you can trap the WM_CTLCOLOR message by subclassing the dialog box procedure in the OnInitDialog() member function for a CDialog-derived class or in the OnInitialUpdate() member function for a CFormView-derived class. This technique allows CDialogs and CFormViews to use the auto-subclassing feature of CTL3D.DLL without adding much code.

The code might resemble the following:

// The CDialog's OnCtlColor() function needs to be
// modified a little. The base class CDialog::OnCtlColor() must not
// be called (like shown above). CDialog::OnCtlColor() calls
// Default() which can get the program into a recursive loop.
// The revised version of OnCtlColor shown below must use afxDlgBrush,
// which requires the AUXDATA.H file from the \MSVC\MFC\SRC directory.

#include "c:\msvc\mfc\src\auxdata.h"

   {
    if( nCtlColor == CTLCOLOR_EDIT)
       {
       pDC->SetBkColor(RGB(127,127,127));
       return (HBRUSH)::GetStockObject(GRAY_BRUSH);
       }

    LRESULT lResult;
    if (pWnd->SendChildNotifyLastMsg(&lResult))
        return (HBRUSH)lResult;     // eat it

    if (!GrayCtlColor(pDC->m_hDC, pWnd->GetSafeHwnd(), nCtlColor,
      afxDlgBkBrush, afxDlgTextClr))
        return NULL;  // Don't call Default() because that will
                      // get us into a recursive loop. The
                      // dialog window procedure calls the dialog
                      // procedure.
    return afxDlgBkBrush;
   }

// This is the new dialog box procedure that will be used.
// Call OnCtlColor() for any dialog boxes that need to have it.
// Be sure to define the OnCtlColor() function as public.

                            WPARAM wParam,LPARAM lParam)
{
   CAboutDlg* pWnd = (CAboutDlg*)CWnd::FromHandlePermanent(hWnd);

   if(msg==WM_CTLCOLOR)
         return (LRESULT) (UINT) pWnd->OnCtlColor(
             CDC::FromHandle((HDC)wParam),
            CWnd::FromHandle((HWND)LOWORD(lParam)),
            (UINT)HIWORD(lParam));
   else
         return ::CallWindowProc(pWnd->m_oldDlgProc, hWnd, msg,
                                 wParam, lParam);
}

// This is a sample of how to subclass the dialog box procedure for
// a dialog box so that the dialog box procedure above can be used to
// forward the WM_CTLCOLOR message to CDialogs or CFormViews.
// m_oldDlgProc can be defined as a data member of the dialog box or
// CFormView. It is defined as:
 //      WNDPROC m_oldDlgProc;

    CDialog::OnInitDialog();

    // Subclass the dialog procedure and save original procedure.
    m_oldDlgProc = (WNDPROC)::SetWindowLong(GetSafeHwnd(),
                                 DWL_DLGPROC,(LONG)&MyDlgProc);

    return TRUE;