PRB: Pointer Functions in MASM Can Hang Real Mode

ID Number: Q68537

3.00

WINDOWS

SYMPTOMS

Windows applications written using the Microsoft Macro Assembler

(MASM) can fail when run under real mode if pointers to FAR

functions are used in the code.

CAUSE

This failure is caused by the following sequence of events:

1. MASM breaks a FAR pointer into separate segment and offset fixup

records in the OBJ file.

2. The linker incorrectly resolves the offset record to point to the

wrong location in Windows call thunk table.

3. When the FAR function is called through the thunk table, invalid

code is executed. This hangs the system.

RESOLUTION

Define a double-word variable in the application's data segment and

initialize it to the function address. Instead of using the

function pointer directly in any code, use the value stored in the

variable. MASM will correctly create a FAR pointer fixup record for

the variable. This record is handled entirely by the loader and

results in correct operation.

More Information:

Normally, when a FAR function is referenced, a fixup record for the

function pointer is placed in the EXE file. The fixup record is

resolved at load time by the Windows Kernel to point to a call thunk.

A call thunk is a short piece of code used in Windows real mode to

check if the code segment containing the called function is currently

in memory, and to load it from disk if necessary.

When a FAR function is used in an assembly program as a separate

segment and offset, MASM creates two fixup records: a segment that is

resolved by the loader and an offset, which is incorrectly resolved by

the linker to point to the incorrect offset in the call thunk table.

For example, an assembly program may contain a function pointer

reference in the form:

;

; Assume wc is a WNDCLASS structure, and MAINWNDPROC is the

; main window procedure for the application.

;

mov WORD PTR wc.clsLpfnWndProc, OFFSET MAINWNDPROC

mov WORD PTR wc.clsLpfnWndProc+2, SEG MAINWNDPROC

Since there are two separate references to the function, MASM

generates two separate fixup records. The value for OFFSET MAINWNDPROC

is incorrectly resolved by the linker.

To generate the correct fixup record, it is necessary to create a

variable in the application's data segment that references the

function. Always use that variable to load memory or registers with

the function address.

;

; Define a pointer variable and initialize to the function

; address.

;

data segment

var_MAINWNDPROC dd MAINWNDPROC

data ends

...

;

; Make all references to the function through the variable.

;

mov WORD PTR wc.clsLpfnWndProc, var_MAINWNDPROC.0

mov WORD PTR wc.clsLpfnWndProc+2, var_MAINWNDPROC.2