About the time you read this, the Win32 API will
reach its fifth anniversary in the public eye. Most
new development is Win32-based (although there's still some 16-bit Windows-based development going on), and most programmers are near the end of its learning curve. Although new language features and OS extensions keep programmers busy, the tectonic plates of Windows-based development haven't shifted recently.
Did you just feel a tremor? The landscape of Windows-based development will be changing soon, and there will be new fundamental concepts to master. I'm of course referring to the 64-bit version of the Windows API. Why do we need a 64-bit version of Windows NT®? In the corporate enterprise world, the 4GB limits of 32 bits are just too confining, especially when you look at industrial strength databases like Oracle, DB2, or Microsoft SQL Server™.
Microsoft is a relatively late arrival to the 64-bit party. The DEC Alpha chip is 64-bit, and there are 64-bit versions of Unix that run on the Alpha. Under Windows NT 4.0 and Windows NT 5.0, the top 32 bits of the address space aren't used (at least not without special help). Intel's Merced chip will be a native 64-bit processor that also supports the x86 instruction set. Many people believe that the availability of x86 and 64-bit instructions on a single chip will make Merced a huge commercial success, accelerating the adoption of 64-bit capable chips and programs.
If you read my article, "A Programmer's Perspective on New System DLL Features in Windows NT 5.0, Part 1" (MSJ, November 1997Figure), you may recall that I described a set of APIs (currently implemented only on the DEC Alpha) that let Win32 programs use memory addresses greater than 4GB. Figure 1 lists the VLM APIs. The semantics of these APIs are the same as the original non-VLM versions. They just happen to work with memory addresses above 4GB.
As you can see, the VLM APIs don't constitute true 64-bit computing. Sure, you can allocate and use this memory if it's physically present, meaning that virtual memory doesn't work with these addresses. But 99.44 percent of the Win32 API can't work with addresses above 4GB, so it's just you and your 64-bit pointers. Think of it as frontier territory with no newspapers, running water, or phone lines.
Clearly, true 64-bit computing will require a significant new version of Windows. It's still too early for me to talk about the specifics, although no doubt you'll be reading about things such as the 64-bit address space in due time. But I can go over the fundamentals now, independent of things like new APIs and executable formats.
Data Types
The first and most obvious consideration about a shift to 64-bit computing is that of data types. I'll use C/C++ as my reference language because Microsoft's operating systems continue to be written in C++. Sure, Java and Visual Basic® are becoming increasingly powerful, but C++ is still the lingua franca of Windows-based programming.
Before getting into Windows-specific types, let's drop back a bit and do a quick review of the C++ type system. For storing integral types, C++ has four basic types: char, short int, int, and long int. The short int and long int types can also be referred to as just short and long. These four integral types are implicitly signed. The unsigned modifier gives you another four types: unsigned char, unsigned short, unsigned int, and unsigned long. The unsigned form occupies the same number of bits as the signed form.
The C++ language has been implemented on many processors. A few rules on the integral C types do make it easier (but not easy!) to write portable code. The size of a character is considered to be 1 byte and, by tradition, 1 byte is 8 bits.
C++ also has relational rules for integral types that allow the language to be tailored to a particular CPU. A short is always bigger than or equal in size to a char. An int is guaranteed to be the same size or bigger than a short. The long must be larger than or the same size as an integer. For a more precise definition, see the ANSI C++ standard, section 3.6.1.
Another informal rule is that an int is traditionally the native size that the machine does integer operations most efficiently with. For example, on the 286 and earlier Intel CPUs, the native size was 16 bits. On the 80386 and later, it's 32 bits. For the DEC Alpha, it's 64 bits. See "Does Size Matter?" (accompanying this article in this month's magazine) for more details on the native size on the x86 architecture.
Now let's consider the C++ integral types and their relationship to pointers. C++ doesn't guarantee a relationship. On many machines, the tradition is that the size of a pointer is the size of an int. This assumption has even crept somewhat into Windows. It's entirely conceivable that you could create a CPU with a 48-bit address space, but that only works with integral data in 16-bit chunks. Not likely, but possible, and the C++ standard allows for this. In general, it's a bad idea to cast pointers to integral types and vice versa. However, it's easier to preach this virtue than to practice it.
Under 16-bit Windows and MS-DOS®, the short and int data types are both 16 bits and functionally equivalent. A long is 32 bits. Pointers? Don't get me started! There are near pointers, far pointers, and just plain pointers. Near pointers are 16 bits, far pointers are 32 bits. Forget to specify near or far? In this case the pointer size depends on the memory model (small, medium, compact, large, and huge). What a mess!
Under Win32 things changed slightly, but at least the hassle of near versus far pointers is removed. A short is still 16 bits. The int and long types are both 32 bits. Pointers are also 32 bits. The only real change is that the int type went from 16 to 32 bits. Figure 2 summarizes all of these details for both 16 and 32-bit Windows. Beyond standard C++ types, the Microsoft® Win32 compiler lets you specify the exact size of an integral type ( __int8, __int16, __int32, and __int64), although these are obviously not ANSI-standard.
While I've been talking about shorts, ints, and longs, a typical Windows program often doesn't directly use the C++ types. In Figure 3, you'll see a portion of the WINDEF.H system header. This section declares a series of typedefs to insulate you from the underlying C++ types, which might change from one CPU to the next. After providing Windows equivalents for longs, ints, chars, and shorts, the remaining typedefs create pointers to these basic types.
The DWORD and LONG types are the most commonly used Windows types in situations where a C++ int or
long would otherwise be used. BYTEs are used for 1-byte
integers while SHORTs are used for 2-byte integers. A DWORD is the de facto choice when you want to use as big an integer as possible.
As an aside, notice that most of the Windows pointer types have both P and LP forms (for example, PWORD and LPWORD). I try to use the P form, if available. The P stands for pointer, while LP means a long pointer (AKA a far pointer.) Since Win32 finally released 16-bit programmers from the insanity of near and far pointers, I try to avoid the anachronisms of 16-bit coding in my Win32 programs.
Livin' Large
With all this discussion of the Windows integer types fresh in your head, picture yourself as the lead architect for a 64-bit version of Windows NT. You have choices to make. How big is an int? How big is a long? How big is a pointer? Remember, you need to take advantage of the chip's 64-bit capabilities, but hopefully not break Win32 code that makes assumptions about type sizes.
The size of a pointer is a nonissue. It simply must be 64 bits. The size of an int and long is still up for grabs, though. If you ponder a while, and remember the C++ type size constraints, there are three scenarios to consider.
To make things easier, I'll first introduce some acronyms that will help keep the potential data models straight. If a data type is 64 bits, its first character will be included in the data model acronym. A "64" tacked on to the end of the acronym reinforces the 64-bit model. In this notation, ILP64 means ints, longs, and pointers are 64 bits, LP64 means that longs and pointers are 64 bits, while ints remain 32 bits. The P64 model has pointers as 64 bits, leaving ints and longs as 32 bits.
The first option that comes to mind is what I call the testosterone model, or ILP64. You've got 64 bits, so use 'em all! Make everything 64-bit: ints, longs, DWORDs, the whole enchilada. The primary problem with this model is data bloat. For the most part, the majority of current integer operations wouldn't come anywhere near to using the full 64 available bits. Expanding ints and longs to 64 bits would mean that the upper 32 bits of most values would be wasted, always containing the value 0.
Beyond the sheer size increase, there are other problems with an all-64-bit model. Consider the slowdown that would come about when using DCOM or other technologies using RPC to transfer data over a wire. Speaking of RPC in an all-64-bit world, your IDL code would need to be redone to use explicitly sized types.
The second scenario for a 64-bit type model is LP64. That is, leave the int as 32 bits, while using 64 bits for longs and pointers. This is the model that 64-bit Unix implementations use. For Unix, LP64 is a good choice from the portability perspective, since most Unix APIs use int parameters, rather than longs.
Windows, on the other hand, uses lots of types that ultimately resolve down to a long, and hence would be 64 bits in this model. For this reason, an LP64-based Windows implementation would have many of the same problems as the ILP64 model. In addition, code that currently gets away with assuming that ints and longs are equivalent would break under LP64.
The final data model to consider is P64. In this schema, only pointers are widened to 64 bits, while ints and longs remain the same as under Win32. The fact that ints and longs remain the same size as the Win32 data model means that vast amounts of code wouldn't need to change. In theory, correctly written Win32-based code should be simply recompiled with a 64-bit compiler to run under this data model. Of course, as I'll show later, there are various places that would need to be cleaned up.
The biggest hurdle to using the P64 model is code that assumes that ints and longs are the same size as a pointer. Almost everyone of us is guilty of coercing a pointer into a DWORD because it makes things easier in some piece of code. Another issue with the P64 data model is that if you truly wanted a 64-bit integer, you'd have to use an explicit 64-bit type, rather than using int, long, DWORD, or LONG.
I can now tell you that the 64-bit version of Windows will use the P64 model. In addition, the 64-bit version of Windows will be built from the same sources as the Win32 version. Since the Windows code will compile to both 32- and 64-bit versions, you can infer that there won't be a new programming model or hundreds of new APIs to learn.
Knowing the data model, what kind of problems will crop up when moving Win32 code to Win64? An obvious place to begin looking is the system header files. Find those places where an API or structure assumes that a pointer fits into 32 bits and you've got a potential problem area.
Some problem areas are easy to spot, while others involving polymorphic data can be subtle. An example of polymorphic data would be an API with a parameter that is interpreted as either an integer or a pointer, based upon the value of other parameters. Let's take a look at some of these problem areas.
Many APIs in Windows take parameter sets that consist of a pointer to a memory region, followed by a length para-meter. Three such APIs are VirtualAlloc, VirtualLock, and MapViewOfFile. In each case, the length of the region is passed as a DWORD parameter. Obviously, under a P64 model, that DWORD won't cut it. This parameter should really be a type that's somehow correlated to the size of a pointer on the system the code is being compiled for. Future system header files will include a typedef that correlates to the size required to store a pointer on the target system.
An example of an API that assumes that pointers and integers are the same size is InterlockedCompareExchange. Both the value to be compared and the exchange value are specified as PVOIDs. In real use, I'd bet that this API is used most often with DWORDs or LONGs that the user has cast to a PVOID to prevent compiler errors.
Yet another API that has problems if ints and longs aren't the same size as pointers is RaiseException. The last parameter is an array of DWORD arguments to pass to the exception handler. Consider this parameter array in the case of an EXCEPTION_ACCESS_VIOLATION. The second element of the array is supposed to specify the virtual address of the inaccessible data. In a 64-bit world, this 64-bit address can't squeeze into a 32-bit DWORD.
Moving into polymorphic data types, consider those
classic APIs, GetWindowLong and GetClassLong. Their very name implies that they return a long. Now, consider calling GetWindowLong with the GWL_WNDPROC or GWL_HIN-STANCE options. Or think about GetClassLong with the GCL_HMODULE or GCL_WNDPROC options. These values are all pointers or pointer typedefs, so they wouldn't fit into a 32-bit long.
Even more subtle is SendDlgItemMessage. This API returns a LONG, and is just a fancy version of SendMessage. The values that SendMessage returns may be integers or pointers. Looking at SendMessage, it returns an LRESULT, which equates to a LONG. In a 64-bit world, the LRESULT type could be changed to represent a 64-bit value to avoid portability problems. Send-DlgItemMessage doesn't get off so easy. It explicitly returns a LONG, and under the P64 model, it wouldn't be able to return pointers correctly.
Speaking of LRESULTs, they're just one of a variety of Windows typedefs that abstract you from the underlying C++ types, but that need to be reexamined in a 64-bit world. Another example is the HMODULE. It's well known that an HMODULE is really a pointer to the beginning of an EXE or DLL module mapped into a process address space (at least on Windows NT and Windows 95). In a 64-bit world, an HMODULE will have to be 64 bits to prevent lots of code from breaking. The WPARAM and LPARAM types are classic polymorphic types. How many times have you seen cases where an LPARAM needs to be cast to a pointer? These types will also presumably need to be 64 bits to prevent massive code breakage in a P64 model.
In the past, the system header files have used typedefs and conditionally defined types to keep API and structure definitions as portable as possible. In keeping with tradition, you can probably expect the system header files to be cleaned up or extended so that existing code will have to change as little as possible. There will probably also be a few new APIs for those places where header file trickery doesn't cut it. It's reasonable to assume that they'll want to make it easy to compile for both Win32 and Win64 with a minimum of hassle.
Have a question about programming in Windows? Send it to Matt at mpietrek@tiac.com.
From the July 1998 issue of Microsoft Systems Journal.
|
