Your own window procedure

Writing your own window procedure and hooking it up to a form or control window is a lot easier than juggling knives, but not much safer. The technique is known as subclassing a window. You simply replace the existing window procedure with your own. Usually your window procedure will identify and handle some specific messages, but defer other messages back to the original window procedure.

So let’s start by throwing a few knives. After we’ve caught them, we’ll look at some of the bad things that could have happened had we missed (as I did when writing this code). Here’s a window procedure for capturing the About menu item that we put on the system menu in the last section. You can see this code in TSYSMENU.VBP:

Public procOld As Long
Public Const IDM_ABOUT As Long = 1010

Public Function SysMenuProc(ByVal hWnd As Long, ByVal iMsg As Long, _
                            ByVal wParam As Long, lParam As Long) As Long
    ‘ Ignore everything but system commands
    If iMsg = WM_SYSCOMMAND Then
        ‘ Check for one special menu item
        If wParam = IDM_ABOUT Then
            MsgBox “Callback Test"
            Exit Function
        End If
    End If
    ‘ Let old window procedure handle other messages
    SysMenuProc = CallWindowProc(procOld, hWnd, iMsg, wParam, lParam)
End Function

This code goes in TSYSMENU.BAS. You might prefer to keep everything in the form module, but you can’t. Windows procedures, like other callback functions, must be in standard modules so that there will be one and only one copy. Anything that must be accessed by other modules, such as forms, must be public. This issue was summarized in “The Ultimate Hack: Procedure Pointers” in Chapter 2.

The code that attaches the window procedure can be in any module. This code is in the Form_Load procedure of TSYSMENU.FRM:

‘ Install system menu window procedure
procOld = SetWindowLong(hWnd, GWL_WNDPROC, AddressOf SysMenuProc)

Here’s how you restore the old window procedure in Form_Unload:

Call SetWindowLong(hWnd, GWL_WNDPROC, procOld)

The first SetWindowLong installs the address of SysMenuProc as the window procedure for the form. Visual Basic has already provided a window procedure for the form, but we’re replacing it with our new window procedure. At the same time, we store the address of the old window procedure in procOld. Notice that the procOld variable is public so that it can be accessed by either the form or the window procedure. The form needs this variable to restore the old window procedure in Form_Unload. The window procedure needs it to pass off most of the work.

As soon as our window procedure, SysMenuProc, is installed, it starts getting messages at a furious pace—perhaps hundreds per second. But we care about only one message—WM_SYSCOMMAND. If you look up WM_SYSCOMMAND, you’ll see that it responds not only to system menu commands, but also to other menu commands, to accelerator commands, and to other situations that don’t concern us. But we care about only one command—IDM_ABOUT. This is the menu ID that we installed for our new About item in the last section. We could have a Select statement here and handle other menu commands, such as SC_MOVE, SC_SIZE, SC_MAXIMIZE, and SC_MINIMIZE. But we don’t want to. The form already knows how to do that stuff. If it’s our menu item, we handle it and exit. Otherwise, we pass it on.

The CallWindowProc API function lets us pass control back to the original window procedure, which knows how to do normal form processing. We let that procedure do its thing, and we receive its return value so that we can return it to whoever subclassed us. Keep in mind that you aren’t the only programmer in the world who might be subclassing your window. There might be a whole chain of subclassers, and you had better be a polite member of the chain.

Notice that the lParam parameter of the window procedure is ByRef, while all the others are ByVal. This is because the CallWindowProc API function is defined in the Windows API type library as taking a void * (equivalent to As Any by reference). So if CallWindowProc is receiving ByRef, then SysMenuProc had better also receive ByRef. You could change SysMenuProc to receive lParam ByVal, but if you did, you’d have to change how you pass lParam to CallWindowProc. For example, you could call like this:

SysMenuProc = CallWindowProc(procOld, hWnd, iMsg, wParam, ByVal lParam)

Either way works for SysMenuProc because the one menu message it handles doesn’t use lParam. The important thing is that you have to pass on exactly the same bits you receive. If you get your types mixed up, you’ll probably crash or hang instantly. You won’t be able to step through your code and find the bug. The compiler won’t warn you. You won’t be able to use Debug.Print. You won’t pass Go. You won’t collect $200. You won’t even go to jail. If you try to pass garbage on the stack several hundred times a second, you won’t be forgiven.
I didn’t learn this from a textbook.