Intercepting Windows Messages in Visual Basic

Deborah L. Cooper

The AddressOf operator, which returns the address of a VB prodecure, was introduced in VB5, letting programmers supply "callback" addresses that had required use of third-party tools like Message Blaster. In this short tutorial, Debbie shows you how to use AddressOf.

Many Windows API functions require a fixed address or pointer. You can use the AddressOf operator to supply the address and to subclass specific Windows messages.

Subclassing refers to the method used to intercept a message sent to a specific window by the operating system. For instance, when you click on the system menu, Windows sends a WM_SYSCOMMAND message to that window. In addition, if you click on one of the menu items within the system menu, Windows sends a menu item ID value to the window, which tells the system the exact menu item you selected. When you subclass a specific Windows message, your program can perform a task not normally done with regards to that Windows message. With the help of a few API functions and the AddressOf operator, you can insert your own routine into the message stream for a window.

In our demonstration program, we’ll add a new menu item called "Our Function" to the system menu. Next, we’ll intercept all messages sent to the system menu to see whether our new menu item was selected. A message box is displayed if our menu item was indeed selected -- otherwise, we let Windows process the user’s selection in its usual fashion.

To intercept a Windows message, you need to know how to write a callback function and how to call it. A callback function is one that you’ve written in your VB application. Here’s how it works: When you call an API function that requires a pointer to an address in your program, the API function is, of course, immediately executed. The API function, in turn, calls your callback function. The callback function might need to return information to the calling API function, and it always returns control to the API function. After the API function has completed its work, control returns to your VB app.

As I said previously, some API functions require that you pass them a pointer to a function. This pointer is simply the address of the function you want to execute. In VB, you can pass the address of the function by using the AddressOf operator. In other words, you use AddressOf to pass the callback function’s address to the API function you want to call.

The secret to intercepting Windows messages, however, is in knowing how to use the AddressOf operator to subclass a particular message. To intercept a Windows message, you need to insert the address of your routine that will process the incoming message into the message stream. It’s important to note that, after intercepting a Windows message with your own routine to process it, you must, at some point in your application, restore the address of the original routine. In this way, you sort of "toggle" the interception of specific messages you want to examine on or off, as needed. Most likely, you’d reinstall the original routine when your application is terminated, as was done in the sample program later in the article.

The sample program lets you intercept all messages sent to the system window of the program’s Form1 form (window). To do this, you need to call the SetWindowLong function to modify the address of the window’s internal structure. To do this, you’ll call the SetWindowLong function with the second argument set to the GWL_WNDPROC constant, which tells Windows you want to change the address of the window function for this particular window to that of your own routine. The new routine you’ll want to use to process system menu messages is in the WindowProc routine, as shown here:

Public Sub hook()
   lpPrevWndProc = SetWindowLong(gWH, GWL_WNDPROC, _
      AddressOf WindowProc)
End Sub

As you can see, you’ve passed the address of the new message handler, WindowProc, by using the AddressOf operator to the SetWindowLong function, which informs Windows of the change. When your application finishes, you’ll need to restore the original window message routine by calling the Unhook procedure:

Public Sub Unhook()
   Dim temp As Long
   temp = SetWindowLong(gWH, GWL_WNDPROC, _
      lpPrevWndProc)
End Sub

To restore messages for your application’s system menu to their original handler, you can simply call the SetWindowLong function with the third argument set to the address of the original message routine, lpPrevWndProc.

The CallWindowProc function is used in applications that need to process a window’s message -- that is, to subclass the messages the specified window receives. In the WindowProc function, we test to see whether the window received a WM_SYSCOMMAND message, which means the user selected the system menu. If this message was received, we do another test to see whether the new menu item (SC_NEWMENU) was selected. If it was, we’ll display a message box to that effect. If not, we’ll pass control to the original message’s routine via the CallWindowProc function. The demonstration program that follows shows how to use the new AddressOf operator to trap system menu messages in VB.

1. Create a new VB project. Form1 is created by default. Add the following code to the Form_Load event:

Private Sub Form_Load()
   gWH = Me.hWnd
   Dim hw As Long
   Dim hMenu As Long
   Dim X As Long   
   'First we need to add our own menu item
   'to the System Menu
   hw = Me.hWnd
   hMenu = GetSystemMenu(hw, False)
   X = AppendMenu(hMenu, MF_STRING, SC_NEWMENU, _
      "&Our Function") 
End Sub

2. Add a Command Button control to Form1. Command1 is created by default. Set its Caption property to "Hook". Add the following code to the Click event for Command1:

Private Sub Command1_Click()
    hook
End Sub

3. Add a second Command Button control to Form1. Command2 is created by default. Set its Caption property to "Unhook". Add the following code to the Click event for Command2:

Private Sub Command2_Click()
    Unhook
End Sub

4. From VB’s Project menu, select "Add Module" to create a new module. Module1.Bas is created by default. Add the following code to Module1:

Option Explicit
Declare Function CallWindowProc Lib "user32" Alias _
   "CallWindowProcA" (ByVal lpPrevWndFunc As Long, _
   ByVal hWnd As Long, ByVal Msg As Long, ByVal _
   wParam As Long, ByVal lParam As Long) As Long
Declare Function SetWindowLong Lib "user32" Alias _
   "SetWindowLongA" (ByVal hWnd As Long, ByVal nIndex _
   As Long, ByVal dwNewLong As Long) As Long
Declare Function AppendMenu Lib "user32" Alias _
   "AppendMenuA" (ByVal hMenu As Long, ByVal wFlags _
   As Long, ByVal wIDNewItem As Long, ByVal _
   lpNewItem As String) As Long
Declare Function GetSystemMenu Lib "user32" (ByVal _
   hWnd As Long, ByVal bRevert As Long) As Long
Public Const WM_SYSCOMMAND = &H112
Public Const MF_STRING = &H0
Public Const SC_NEWMENU = 1
Public Const GWL_WNDPROC = -4
Global lpPrevWndProc As Long
Global gWH As Long
Public Sub hook()
   lpPrevWndProc = SetWindowLong(gWH, GWL_WNDPROC, _
      AddressOf WindowProc)
End Sub
Public Sub Unhook()
   Dim temp As Long
   temp = SetWindowLong(gWH, GWL_WNDPROC, lpPrevWndProc)
End Sub
Function WindowProc(ByVal hw As Long, ByVal uMsg As _
   Long, ByVal wParam As Long, ByVal lParam As Long) _
   As Long
  'We need to trap the WM_SYSCOMMAND message to
  'determine when the user has clicked on our
  'new menu item. The message is stored in uMsg.
  'Our new menu item is identified as SC_NEWMENU
   If uMsg = WM_SYSCOMMAND Then
   Select Case wParam And &HFFFF&
      Case SC_NEWMENU
         MsgBox "The Our-Function menu item was selected"
   End Select
   End If
   'Always call the original handler when done
   WindowProc = CallWindowProc(lpPrevWndProc, hw, _
      uMsg, wParam, lParam)
End Function

Execute the demonstration program by pressing the F5 function key. The system menu now includes a new menu item called "Our Function." However, when you click on this new menu item, nothing happens. Click on the "Hook" command button. Now, select the "Our Function" menu item. The program responds by displaying the message "The Our-Function menu item was selected." Click on the "Unhook" command button to restore the system menu to its original state (no action is performed when you select the new menu item).

Download TRAPMSG.exe

Deborah L. Cooper, author of Developing Utilities in Assembly Language and Developing Utilities in Visual Basic 4.0, is a contract technical writer and programmer. She’s also the author of the "Visual Basic Tips Articles" series on MSDN and maintains a VB tips site at http://www.geocities.com/SiliconValley/Park/2740. debbie_cooper@infomatch.com.