Windows cannot operate in an environment of mixed segment types (including both 16:16 and 16:32 segments). As a result, the stack selector size must match the corresponding code selector size. When the processor is executing code in a 16:32 (USE32) code segment, the selector in the SS register must contain a 16:32 selector. When the processor is executing code in a 16:16 (USE16) segment, the SS register must contain a 16:16 selector.
When the 80386 or 80486 processor is executing on a USE16 stack segment, it uses the low-order 16 bits of the ESP register as the SP register. Because only the low-order 16 bits are of use when the processor is running on a USE16 stack segment, the processor does not control how the high-order 16 bits of the ESP register are set. As a result, the high-order 16 bits are set at random. When an application switches to a USE32 stack segment, the ESP register contains a corrupted pointer unless the high-order 16 bits of ESP are set properly.
Suppose that a Windows application has a USE32 code segment and a USE16 helper segment, but (improperly) only a USE32 stack. When the application calls from its USE32 code into the USE16 segment, the application continues to use its USE32 stack. The USE16 code segment calls a Windows function, which changes the selector in the SS register to a USE16 selector. Because the stack is now USE16, the high-order 16 bits of the ESP register are set at random. The code that originally switched stacks then restores the original selector in SS and, lacking the information that the selector referred to a USE32 stack, restores the 16-bit SP register instead of the full 32 bits of the ESP register. As a result, the USE32 stack now has an invalid pointer in the ESP register.
There are a number of ways to deal with this problem. One solution is for an application to maintain two separate stacks, one USE16 and the other USE32. Maintaining separate stacks requires you to include extra code—for example, you must copy parameters for stack-calling conventions such as that used in C. Another solution is to maintain one stack but two stack selectors, one USE16 and the other USE32, both of which point to the same memory. This requires the USE32 stack to be restricted to ESP values less than or equal to FFFFh.
In either case, the USE16 code segment must switch to the USE32 stack immediately before calling into a USE32 code segment. When control returns from the USE32 code segment to the USE16 code segment, the USE16 segment must switch back to the USE16 stack before doing anything else.
Because the problem with stack switching is the corruption of the high 16 bits of ESP, a Windows application with 16:32 code must make sure that it sets the high 16 bits of ESP when it is switching to the USE32 stack selector. It sets these bits by placing the selector into the SS register, as shown in the following example:
mov ss,word ptr [Use32StackSel]
mov esp,dword ptr [Use32StackOffset]
mov ss,word ptr [Use32StackSel]
movzx esp,word ptr [Use32StackOffset]
mov ss,word ptr [Use32StackSel]
movzx esp,sp