Microsoft Corporation
Last updated: August 1999
Summary: This technical white paper discusses Microsoft® Interface Definition Language (MIDL) compiler and Remote Procedure Call (RPC) issues related to 64-bit porting efforts. Both old techniques and new 64-bit features that can facilitate porting applications from 32-bit environments are described.
Introduction
Issues
Tips for Modifying RPC Interfaces
MIDL64—New Features for 64-Bit Windows
Frequently Asked Questions
Open Issues
For More Information
This technical white paper discusses Microsoft® Interface Definition Language (MIDL) compiler and Remote Procedure Call (RPC) issues related to 64-bit porting efforts. Both old techniques and new 64-bit features that can facilitate porting applications from 32-bit environments are described.
With the coming availability of 64-bit Windows® there is a need for a MIDL/RPC guide outlining what can and cannot be done with regards to porting legacy applications.
Terminology Note There is some confusion among IT staff regarding the terms local, inproc and remote. An inproc call in RPC means an RPC call that is executed within the same process. For example, different DCOM apartments can execute as separate threads in a process. RPC is optimized for in-process calls; however, the arguments of the call still go through the marshaler. Therefore, all the usual restrictions of RPC calls, such as no void *, apply. The inproc call category has been introduced for performance reasons, as some handles make sense on both sides of the call. Remote call usually refers to a non-inproc call, because there is not a much difference between executing an RPC call from process to process or from machine to machine. Both cases involve two different address spaces requiring marshaling of all the arguments. Sometimes, especially in contrast to a local call, a remote call means any call that goes through RPC and so requires marshaling and unmarshaling of arguments. In RPC context, a local call means a call within the same process where RPC is actually not used at all—the call goes directly from client layer to server layer of the application. In particular, a [local] attribute of RPC means that the procedure or interface flagged with it is never used in a remote call. Sometimes an interface is specified as [local] just to get its C definition in a convenient .h file.
Also note, that when more information about specific MIDL attributes is desired, MSDN is the best up to date source for pre-64-bit functionality RPC documentation.
This section focuses on 64-bit specific problems; discussion of 32-bit specifics has been added where appropriate.
When porting an RPC application, the key feature to consider is the backward wire compatibility; that is, the ability of the original and the ported versions to talk in the same wire format or protocol, in this case the NDR (see Endnote 1) wire representation of transmitted data. Backward wire compatibility may be an issue whenever one of the following happens:
Usually, methods are added to the interface as a result of expanding the server's functionality for a new release of a product. The wire incompatibility may happen if the new client is not prevented from calling the old server. This type of wire incompatibility is the most obvious one, and this issue is not really a porting issue; see the section "Nonissues" for a more detailed discussion.
Introducing a new data type or modifying an existing data type is the most frequent source of wire incompatibility problems. The core reason for all of these problems is that the OSF/DCE RPC paradigm assumes that the receiving party has preexisting knowledge about the nature of the data coming in; that is, the data is sent without a generic data description. Therefore, whenever a receiver expects a different data type than the sender has put on the wire, the most likely outcome is a bad stub data exception.
The cleanest way of adding new data types to an interface is to use them only in new methods. This doesn't create any problems other than the ones introduced by adding the new methods themselves. Changing data types used in existing methods creates incompatibilities. Typically, there are two ways to add types to an existing interface that can create problems. The explicit way is to change one of the unions controlling data flow by defining a new branch that takes a new type, such as a new info level in some methods returning server information. This is usually a situation that is easy to control. The change, however, may be concealed when an application hides a data type from RPC or cheats on the RPC marshaler. An example of hiding is when an application sends an array of bytes that, in fact, is structured data. An example of cheating is when an application sends a long that means something else. Hiding is actually a form of cheating, and application writers usually encounter trouble when the application talks to a big-endian machine for the first time.
Porting from the 32-bit version of Windows to Win64™ does not create RPC problems in itself. The abstract data type model established for porting (that is, LLP64) has a great advantage for porting RPC applications. Essentially, in this model, all of the integral data types used in the 32-bit world stay at the same size in the 64-bit world, and only the pointer size changes from 32-bit to 64-bit. The RPC programming model has always relied on data sizes being well defined, with all of the integral types being the same size on both sides of the connection. Because the size of the integral types doesn't change, there is no wire incompatibility. On the other hand, pointers are local to each side of the RPC transmission and typically go on the wire only as null or nonnull markers. Thus, marshaler is able to handle different pointer sizes on both ends of a connection in a manner that is transparent to the user.
Changing the size of the pointer itself is not an issue if the interface in question has a pointer specified as an argument. On the wire, pointers are represented by 4-byte markers. When making an RPC, whether it is truly remote or within the same machine, pointers are typically recreated from scratch at the other side, based on the marker. The MIDL and RPC marshaler handle this transparently. For 64-bit porting, this transparent handling is at the expense of performance. Performance extensions to the MIDL compiler and the corresponding marshaler engine are planned for the future.
Also note that a typical pointer for an [in] argument, such as the one shown below:
long Foo( [in] long * Bar );
is a [ref] pointer that does not have any wire representation at all. In the above example, only a long would be sent from client to server.
The context handle is a special RPC concept introduced solely for the purpose of giving a client an opaque handle to a server context held on behalf of the client. Typically, it is a void * pointer marked as a context handle. The type is defined with the [context_handle] attribute and implies the following semantics. A handle must be obtained from the server first, typically by means of a call with handle as an [out] argument. The client indicates the context whenever handle is an [in] argument. At the end, the handle must be closed to clean up the server context. This is done by means of a call with an [in,out] context handle argument when the handle comes back as null. An added bonus is that a context handle can be cleaned up automatically by the RPC runtime when the connection breaks.
Each context is identified by a globally unique identifier (GUID) generated on the fly by the RPC runtime when the handle is created. Context handles are represented on the wire by 20 bytes that include the GUID to identify the context. On the client, the application is handed a handle to a dictionary storing the context handle GUIDs. On the server, the pointer to the context is retrieved from a similar dictionary based on the GUID. No pointer is sent across, and this process works completely transparently across the 32-bit/64-bit boundary.
For example:
typedef [context_handle] void * MyContext;
void OpenContext( <some other args>, [out] MyContext * pHandle );
void UseContext( [in] MyContext Handle, <some other args> );
void CloseContext( [in,out] MyContext * pHandle ); // null when [out]
One important part of context handle semantics is that a context handle is related to the server that generated it. In particular, when there is no explicit binding handle argument of a call, the context handle argument directs the binding, meaning that the call is directed to the server that originated the handle. If an explicit binding handle is used, the binding implied by the context handle must agree with the one indicated by the binding handle, and must resolve to the same server. If there is a disagreement—for example, if different servers are implied by context handle(s) and the binding handle—the call fails by definition.
In addition, there is a group of attributes related to context handles that can force a better granularity of the context checking or some particular type of a context handle; this is outside of the scope of this discussion. For example, a programmer can specify a [strict_context_handle] type, where the context handle is valid only when used with the same server and interface that created it.
These handles are RPC handles indicating the server to connect to. They are indicated by the application configuration file (ACF) attributes, such as [implicit_handle], [explicit_handle], and [auto_handle]. These handles are internal RPC handles that are never represented as arguments on the wire. They are manipulated by means of RPC runtime application programming interfaces (APIs), and not by remote calls. Typically, a handle must be well-defined before a remote call can be executed and doesn't change during the call. Other internal RPC handles of a similar nature are pickling handles and async handles.
A generic handle is a special type of binding handle that is a cross between a binding handle and a generic data type remotable by means of RPC. When this type of handle is used in a call, special binding semantics are implied (on the client side only) by the stub before and after the proper remote call. A typical declaration may look like this:
typedef [handle] wchar_t *STRING_HANDLE;
It's used in a call like this:
DWORD
RpcOpenPrinter(
[in, string, unique] STRING_HANDLE pPrinterName,
[out] PRINTER_HANDLE *pHandle,
[in, string, unique] wchar_t * pDatatype,
[in] LPDEVMODE_CONTAINER pDevMode,
[in] DWORD AccessRequired
);
The special binding semantics consist of the client stub calling a STRING_HANDLE_bind() routine to establish a binding before the stub marshals arguments, then calling a STRING_HANDLE_unbind() routine after the stub unmarshals the reply. The application writer provides the routines. The bind and unbind calls are performed each time the client application executes a call using a STRING_HANDLE argument as a binding argument.
When data is marshaled for the call, the generic handle argument is represented by the data it has been specified with. Therefore, it is undistinguishable on the wire from the same data being sent in the usual way. As far as the data itself is concerned, it is treated like other data arguments for the purposes of marshaling and unmarshaling. In the above example, a Unicode character string would be marshaled as usual for a STRING_HANDLE argument.
Each of these is a polymorphic integral type that has a different size on 32-bit and 64-bit Windows NT® platforms to guarantee that a pointer value can be stored in it. The data types are defined in the public file, basetsd.h. Each has a size of 4 bytes on 32-bit systems and 8 bytes on 64-bit systems. On 32-bit platforms, INT_PTR maps to int and LONG_PTR maps to long. On 64-bit platforms, both the types map to __int64. Types such as these cannot be supported in a satisfactory way in RPC. In fact, if not for the porting, a type like this would never be supported and should be seen only as a means of facilitating the 32-bit-to-64-bit porting effort. The reason these types are supported by MIDL is to address a situation where a pointer needs to be kept in an integral field for some reason. Typically, a remote application programming interface (API) would send a handle across the wire disguised as a DWORD.
Please note however, that sending a pointer or handle this way could never really work with RPC. That is, either the argument didn't mean a pointer, it went unused, or the pointer or handle value was sent out only to be sent back and used at the server. In other words, this was another case of cheating on RPC. As described in this document, one way to avoid cheating is to use a context handle or an IUnknown interface pointer.
MIDL supports the 32-bit/64-bit polymorphism by means of the __int3264 intrinsic integral type described later in this document. However, this type should not be introduced to a remoted interface unless it is unavoidable because of porting. New interfaces should never use it. You should make up your mind as to the nature of data manipulated remotely and specify either an __int32 (a long) or an __int64 (a hyper) as appropriate. If the field or argument is really designed to hold a pointer, an appropriate pointer or context handle type should be used.
For the porting, you should not use the new intrinsic. Instead, you should use the types defined in public\sdk\inc\basetsd.h directly: INT_PTR, LONG_PTR, DWORD_PTR, and so forth, can be used as appropriate. The file contains all the mappings needed for MIDL and C/C++ compilation of the types.
The polymorphic types should not be used automatically, just because they are available. For clarity of design and implementation, it is worth reworking the application so that you use clean types and thereby avoid side effects, such as the performance penalty inherent to polymorphic types.
For example, in RPC's own code, there was a case where a BufferLength field was assigned a pointer because the marshaler was using a sizing routine (which increments a value of the BufferLength field by an object size according to the object type) to increment the pointer by the size of the pointee. At first, it seemed that this field needed to be converted from unsigned long to UINT_PTR. Some alignment issues had to be taken care of, but eliminating the usage was fairly simple; therefore, modifying the application removed the need to use the polymorphism at all.
In summary, at the core of each RPC porting problem is always an application not sticking to the RPC specification and cheating on the stubs and marshaler in some way.
Problems may typically arise when:
Typically, this is a design problem that should have been solved by using context handles in the first place.
For example:
CONFIGRET
RegisterNotification(
[in] handle_t hBinding,
[in] ULONG hRecipient,
[in,size_is(ulSize)] LPBYTE NotificationFilter,
[in] ULONG ulSize,
[in] DWORD Flags,
[out] PULONG Context,
[in] ULONG hProcess
);
CONFIGRET
UnregisterNotification(
[in] handle_t hBinding,
[in] ULONG Context
);
This is an excerpt from a standard (raw) RPC interface, as opposed to an object or DCOM interface.
Apparently ULONG sent as a context is intended to be a pointer. Hence, that would not work when sending back and forth from a 32-bit to a 64-bit environment. Also note, that changing the type of the Context argument from ULONG to __int64 or similar will break the wire compatibility, and that using __int3264, as described later, would keep wire compatibility intact but would not work (the cheating is not good enough).
From what we know, the ULONG manipulated here is really a server context handle—the handle is obtained in the Register call and released in the Unregister call above. We are not interested in details of the handle usage. However, notice that this design has an inherent problem in that it is not clear what happens with the server context when the client dies or goes away without unregistering first. This functionality should have been mapped to a context handle solution. In other words, the design is flawed because a shortcut is used when sending context pointers instead of a clean context handle.
When looking at a problem like this in the context of porting, it is important to consider whether wire compatibility is an issue. Your solution choice depends on whether the interface has been made public; if it has, it generally cannot be changed. On the other hand, if the interface was used only within a single machine by a component that cannot have different versions on the same machine, it typically can be changed freely.
In many situations, it is possible that backward compatibility problems can be ignored or the interface reworked so that a context handle is used. If this is possible, it is the best solution. To use context handles, the interface could be changed to take the following:
typedef [context_handle] void * NOTIFICATION_CONTEXT;
CONFIGRET
RegisterNotification(
[in] handle_t hBinding,
[in] ULONG hRecipient,
[in,size_is(ulSize)] LPBYTE NotificationFilter,
[in] ULONG ulSize,
[in] DWORD Flags,
[out] NOTIFICATION_CONTEXT * pContext,
[in] ULONG hProcess
);
CONFIGRET
UnregisterNotification(
[in] handle_t hBinding,
[in,out] NOTIFICATION_CONTEXT * pContext
);
The user needs to provide a rundown routine to clean up the server context when the connection between client and server breaks. The routine would have the following prototype:
void __RPC_USER NOTIFICATION_CONTEXT_rundown(
NOTIFICATION_CONTEXT handle);
Please see description of context handles in the context handle section for more information about the Context_rundown routine.
Typically, this is another design problem. Possibly, IUnknown * should have been used. For example, let's look at IConnectionPoint interface.
interface IConnectionPoint : IUnknown
{
< snip, snip >
HRESULT Advise(
[in] IUnknown * pUnkSink,
[out] DWORD * pdwCookie );
HRESULT Unadvise(
[in] DWORD dwCookie );
}
This is similar to the previous example in that, as far as we know, the intention here was to have the cookie be a polymorphic type that would make it possible to send DWORDs as well as pointers across the wire.
First, note that the interface is a public interface released with Windows NT 4.0; therefore, backward wire compatibility is required. For that reason, changing the interface is not an option. If the interface needed to be changed, the only solution would be to introduce another interface and phase out this one over time.
Even if the backward compatibility requirement didn't preclude changing the interface, the context handle approach could not be used here. This is because DCOM provides a context_handle-like functionality by its nature. Therefore, context handles are not needed in DCOM interfaces and MIDL would not allow using one. One way of looking at this is that server contexts correspond to objects in DCOM. So, instead of creating a context handle, you pass an interface pointer indicating the object. Instead of passing the context handle to indicate the server context, you can either call methods directly through the interface pointer or you can put that interface pointer in some other calls. Lastly, to release a server object the client can simply call the Release() method, present in every interface, through the interface pointer. This is the essence of object programming in DCOM—the contexts (objects) are indicated by interface pointers and reference counting dictates their life span.
Therefore, in DCOM, the typical solution is, in a way, symmetrical to context handles in raw RPC, or, if you prefer, context handles emulate an object behavior. For this reason, the cookie should have been defined as an IUnknown * pointer and the interface could have read:
HRESULT Advise(
[in] IUnknown * pUnkSink,
[out] IUnknown ** ppCookie );
Note that the Unadvise method isn't needed at all; to release the context it would be enough to call:
pCookie->Release();
As mentioned earlier, every interface has a Release() method by virtue of inheriting from the IUnknown interface.
Continuing the previous example, whatever can be said about errors of the original design, there may also be a legacy problem. That is, there may be a situation where a pointer was sent as a long, nothing can be done without breaking wire compatibility, and some other fix is needed. In this case, the solution would be to stick to the wrong design; that is, keep sending the long, but do an auxiliary mapping between pointers and DWORDs as necessary.
For example, the methods from the previous example could be fixed by introducing a simple RPC wrapper by means of using [call_as] attribute. (If your application uses the standard RPC interface, see the discussion of the [call_as] feature and the example of usage with a standard RPC interface).
[local]
HRESULT Advise(
[in] IUnknown * pUnkSink,
[out] DWORD_PTR * pdwCookie );
[local]
HRESULT Unadvise(
[in] DWORD_PTR dwCookie );
[call_as(Advise)]
HRESULT NewRemoteAdvise(
[in] IUnknown * pUnkSink,
[out] DWORD * pdwCookie );
[call_as(Unadvise)]
HRESULT NewRemoteUnadvise(
[in] DWORD dwCookie );
Note that the application interface has the same methods, Advise and Unadvise, except that they have to take an ULONG_PTR or DWORD_PTR instead of DWORD to be able to keep a pointer. As explained in the [call_as] section, the call still goes through Advise and Unadvise for application API compatibility, except that the remote call is now executed using the NewRemote* methods. For this particular case, the new remote methods are designed to have exactly the same wire layout. What you get from the [call_as] scheme is simply the introduction of hand-written wrappers on both sides. The server side wrapper for Advise needs to introduce mapping between the returned pointers and DWORDs by keeping an appropriate dictionary. A server wrapper for Unadvise needs to map back from a DWORD to pointer using the same dictionary. The client side wrapper does not have to do anything except pass the arguments without any changes. Therefore, the wrappers could look like this:
// server side wrappers
HRESULT
IConnectionPoint_Advise_Stub(
IConnectionPoint * This,
IUnknown * pUnkSink,
DWORD * pdwCookie )
{
LONG_PTR * UnmappedCookie = 0;
HRESULT hr =
This->lpVtbl->Advise( This, pUnkSink, & UnmappedCookie );
*pdwCookie = MapPointertoDWORD( UnmappedCookie );
return hr; // add a better error control
}
HRESULT
IConnectionPoint_Unadvise_Stub (
IconnectionPoint * This,
DWORD dwCookie )
{
LONG_PTR Cookie = MapDWORDtoPointer( dwCookie);
return This->lpVtbl->Unadvise( This, Cookie );
}
// client side wrappers
HRESULT
IconnectionPoint_Advise_Proxy(
IConnectionPoint * This,
IUnknown * pUnkSink,
DWORD_PTR * pdwCookie )
{
return IConnectionPoint_NewRemoteAdvise_Proxy( This, pUnkSink,
(DWORD *)pdwCookie );
}
HRESULT
IconnectionPoint_Unadvise_Proxy (
IConnectionPoint * This,
DWORD_PTR dwCookie )
{
return IConnectionPoint_NewRemoteUnadvise_Proxy( This, (DWORD)dwCookie );
}
Note that this is a porting solution; namely, in this solution, we always do mapping between cookies and DWORDs and we always send DWORDs out. For 64-bit-to-64-bit communications with better performance, a clean way to cheat would be to simply define another interface, one that would send an __int64 on the wire.
In this section we discuss general issues related to remoting handles as opposed to pointers. Please look at the next section to see how to deal with Win32 system handles.
The essential difference here is that while a pointer directly indicates a datum, a handle may be seen as a key to data. That is, some level of indirection is implied by the handle, some other special values of the handle can exist besides NULL, etc. If so, then the implementor of the handle package may choose to restrict the handle value range of 64b package to be the same as the range of the 32b package, most likely to be 32b key values. As 4GB range implies a lot of handles, this is most likely a perfectly appropriate solution for 64b porting. At the same time, it is typical that handles are defined as a void * type, and so following the definition their size should follow the size of the pointer to minimize impact on the related applications.
This boils down to a solution where the handle has the size of a pointer, that is 4 or 8 bytes depending on a platform, but the lower 32b are enough to identify the handle correctly by the package. There are 2 choices for the upper 32b: they either should always be set to 0, or should always be sign extended by the package. The choice is difficult as both approaches have similar advantages and disadvantages. The deciding factor is the pattern of usage by the customers of the package under discussion.
When the handle values are sign-extended, handling of all the special values, like –1, is easy. Hence, the client code requires a minimal facelift. It also agrees well with using handles in remote RPC calls via a long argument. However, when remoting the handle via a DWORD, a care has to be applied at the 64b side to sign extend the handle properly. Otherwise the handle looses sign extension and a comparison like h1 == h2 is unreliable (the handles may differ only on upper 32 bits).
When the handle values are zero extended, the special values like -1 have to be redefined as constants with appropriate casts and client code modified as appropriate. This approach agrees very nicely with RPC remoting via DWORD arguments, which is a typical hack people use. However, if RPC calls use a long to represent handle, the care has to be taken to cast the value properly. Also, if special handle values have been defined in terms of values perceived as unsigned, like 0xFFFFFFFF, the user may end up with comparing a handle with a sign extended 0xFFFFFFFF value to his surprise.
In the end it is hard to recommend clearly one of the above approaches over the other and a judgement call is needed that weighs the application code changes against the changes in the handle package itself. For application defined handles, the implementor of the interface and / or handle would have to make the call.
None of the system handles that are remoted today are supported directly by MIDL or RPC. The support is indirect. Namely, definitions for some of the handles have been placed in a public file (wtypes.idl) using [wire_marshal] or [call_as] attributes and corresponding auxiliary routines or wrappers have already been written by hand and are part of OLE32.dll code or OLEAUT32.dll.
Before we go into a detailed discussion of how individual handles are serviced, let’s mention the cases where the user does not have to do much or does nothing at all to port.
Regardless of the type of handle that is being used, in some cases the nature of the interface is such that it is guaranteed to be used in a homogenous environment. For example, the interface may be used only when calling from process to process on the same machine, with the implementation being 32b on a 32b system and 64b on a 64b system.
If this is the case and the handle was sent as a DWORD (or a ULONG), a simple porting solution is to change the remote argument type from DWORD to HANDLE_PTR. This handle type is defined in the public file basetsd.h to be unsigned long on 32b and unsigned __int64 on 64b. This solution works well and without any performance penalty as 4 bytes represent the handle on wire when calling from 32b to 32b and 8 bytes represent the handle when calling from 64b to 64b.
Note that for problems restricted to homogenous environment, a value like INVALID_HANDLE_VALUE value does not create any problems.
To facilitate the porting, a decision has been made that these system handles should stay as 32b values, sign extended to 64b on the 64b platform. That is, the individual handle types are still based on the HANDLE type, which maps to void *, and so the size of the handle is the size of the pointer, i.e. 4 bytes on 32b and 8 bytes on 64b. However, the actual value of the handle on the 64b platform, (i.e. the meaningful bits), fits within the lower 32b, while the upper bits just carry the sign.
This should make it easy to port the majority of the application code. Handling of the special values, like –1, should be fairly transparent. It also should agree nicely with all the cases where the handles had been remoted with the help of the IDL definitions from the public file wtypes.idl. However, care needs to be taken when remoting the handles was done via a DWORD, as the upper long should be properly sign extended on the 64b side. The app should use HandleToLong() and LongToHandle() macros (inline functions) to do the casting right.
So, in general we can have the following situations:
Going back to specifics of porting the legacy code, the following system handles fall into the nice 32b compatible category.
The USER handles: HWND, HMENU, HICON, HCURSOR, HDWP, HHOOK, HACCEL, HWINSTA, HDESK, HKL, HMONITOR, HWINEVENTHOOK.
The GDI handles: HBITMAP, HPALETTE, HMETAFILE, HENHMETAFILE, HMETAFILEPICT, HBRUSH, HFONT, HDC, HRGN .
HGLOBAL is different from GDI and USER handles discussed above in that it can either be a 32b handle or an actual pointer (32b or 64b depending on the platform). An HGLOBAL is created as a handle when GMEM_MOVEABLE flag is used in GlobalAlloc API, otherwise it is created as a pointer.
Accordingly, the application writer may have the following situations:
To recap, if your app sends HGLOBAL as an HGLOBAL, you don’t need to do anything as ole32.dll code marshals and unmarshals it transparently.
Also note that HLOCAL is a typedef on HGLOBAL.
When discussing support for GDI and USER handle passing available in the Windows system, the distinction between inproc and remote communications becomes important. The best support for some of the handles means sending different data depending on whether the handle is sent inproc or out of proc. In the inproc case, only a handle can be sent, as it will make sense on the other side of the RPC connection. In the remote case, (which on NT means either a different machine or a different process on the same machine), the data implied by the particular handle is sent in full, which makes it possible to create a handle at the other side that indicates the duplicated data. For some handles, sending data, or indeed using the handle at the other side of the connection, does not make sense. Nevertheless such handles may still be sent over the wire, with the idea that the only use the other side can have of it is to send it back where it makes sense. For example a server may hand out a handle that later would be used at the server when passed with a different call from a client. In such cases only a limited support exists for remoting the handles, namely data is never sent and the handle is sent as a long.
For list of handles supported in full and in a limited fashion see the sections below.
The current model for 64-Bit Windows NT makes it impossible for both 32-bit .dlls and 64-bit .dlls to be present in one process. This means that inproc calls happen only between homogenous parties. As mentioned elsewhere, RPC calls may be used in inproc case, especially for DCOM.
In the Windows 95 and Windows NT 4.0 time frame, full support was added for remoting the following system handles: HGLOBAL, HBITMAP, HPALETTE, HMETAFILE, HENHMETAFILE, and HMETAFILEPICT. This was related to an effort to directly support the OLE data type STGMEDIUM. Ever since, full support has meant that, in the inproc case, the handles have been sent as handles (that is, as long values) and in, the remote case, as data that has been used to re-create a valid handle of a given type. In Windows NT, a remote call means a call between different machines or different processes at the same machine.
The current implementation uses the [wire_marshal] attribute, with definitions in wtypes.idl, and hand-coded auxiliary routines related to [wire_marshal] being part of OLE32.dll. This solution is being changed as appropriate to support remoting between 64-bit and 32-bit environments.
For example, for HGLOBAL, the current supporting definition related to wire marshal would need to change to
typedef union _userHGLOBAL switch( long fContext ) u
{
case WDT_INPROC_CALL: LONG_PTR hInproc;
case WDT_REMOTE_CALL: FLAGGED_BYTE_BLOB * hRemote;
default: LONG_PTR hGlobal;
} userHGLOBAL;
typedef [unique] userHGLOBAL * wireHGLOBAL;
Some system handles aren't remoted at all; they have never been remoted even as DWORDs. This applies, for example, to the following handles from wtypes.idl: HMODULE, HINSTANCE, HRGN, HTASK, HKEY, HDESK, HPEN, and others. All of these are defined as void * and are never used in any remote interface.
Partial remoting of a handle means that it has been remoted only as long (or as a DWORD) in both inproc and remote cases. This applies to a handle that would make sense only in the process that generated it, and its remoting as a DWORD was no more than a fix to a legacy problem when some OLE interfaces became remoted. The following handles fall into this category: HWND, HMENU, HACCEL, HBRUSH, HFONT, HDC, and HICON.
All of these are USER handles and behave as described in the section about the GDI and USER handles.
In particular, the following definition has not changed for porting to 64b:
typedef union _RemotableHandle switch( long fContext ) u
{
case WDT_INPROC_CALL: long hInproc;
case WDT_REMOTE_CALL: long hRemote;
} RemotableHandle;Then the handles are defined in the usual manner:
typedef [unique] RemotableHandle * wireHWND;
typedef [wire_marshal( wireHWND)] void * HWND;In the future, other branches can be added fairly easily to the union to cover full support for remoting these handles if needed. Please see the section "Changing Data Types Without Changing the Interface Version" to learn more about the meaning of the lack of a default clause in a remoted union.
A standard RPC interface is identified by its GUID and the version number combination; a DCOM interface is identified by its GUID. The version consists of major and minor parts. For standard interfaces with the same GUID and different version numbers, a connection is possible only when the major version is the same and the minor version of the client is not higher than the minor version of the server.
In standard RPC, a path for upgrades and extensions is well-defined and basically requires that new methods are only added at the end of the interface and new types are used only in the new methods. If additions like these are needed, the minor version must be bumped up. Any other change requires a change in the major version because the sides would be incompatible on the wire. Adhering to these rules assures that whenever there is a connection between a new client and old server (or the reverse), both sides are fully compatible.
For a DCOM interface, the best way is to create a new interface for the expanded functionality. An equivalent of the minor version is the new interface inheriting from the old one. Changing old methods or old data types requires an entirely new DCOM interface.
As valuable as the rules described above are, they don't address some real-life situations. Note that effectively changing an interface requires that new clients don't talk to old servers. This is frequently impossible with commercial software that has been deployed in the field. In Windows NT, there have already been several public releases where interfaces have been changed without changing the GUID or version. This happened precisely because new clients had to talk to legacy servers and because the solution that a new client would support both the old and new interfaces was deemed undesirable for some reason.
The old servers and new clients can coexist smoothly because RPC always catches an attempt to call a method beyond the ones specified for an interface. A clean exception, RPC_S_PROCNUM_OUT_OF_RANGE, is raised in this case. This exception is used only in this situation. In DCOM, HRESULT_FROM_WIN32() of the same error code is returned. Therefore, the new client's code can be added to handle the exception to degrade performance gracefully or to carry out whatever workaround is appropriate when talking to an old server.
As with methods, sometimes data types need to change without overtly changing the interface GUID and/or version. This is much harder to accomplish without violating backward wire compatibility. Sometimes using some kind of argument mapping with [call_as] or [transmit_as] attributes can help. Without those attributes, about the only safe way of cheating is to use a defaultless union type. Here is a typical example of a remote API that returns a different level of information data depending on the release, and so forth (several cases have been dropped to save space):
typedef [switch_type(USER_INFORMATION_CLASS)] union
_USER_INFO_BUFFER {
[case(UserGeneralInformation)] USER_GENERAL_INFORMATION General;
[case(UserPreferencesInformation)] USER_PREFERENCES_INFORMATION Preferences;
[case(UserLogonInformation)] USER_LOGON_INFORMATION Logon;
// snip, snip
[case(UserAllInformation)] USER_ALL_INFORMATION All;
[case(UserInternal3Information)] USER_INTERNAL3_INFORMATION Internal3;
[case(UserInternal4Information)] USER_INTERNAL4_INFORMATION Internal4;
[case(UserInternal5Information)] USER_INTERNAL5_INFORMATION Internal5;
} USER_INFO_BUFFER, *PUSER_INFO_BUFFER;
This is used in a typical get info call, where the caller indicates what kind of information should be returned:
NTSTATUS
SamrQueryInformationUser(
[in] USER_HANDLE UserHandle,
[in] USER_INFORMATION_CLASS UserInformationClass,
[out, switch_is(UserInformationClass)]
USER_INFO_BUFFER *Buffer
);
The point here is that the union type above does not have the default. The RPC semantics of the situation are that, if an unknown case is encountered during unmarshaling, the NDR engine will raise the exception RPC_S_INVALID_TAG. (In DCOM this exception would come back as an error mapped by the usual HRESULT_FROM_WIN32() macro.) This is as opposed to defining a default, where the prescribed unmarshaler behavior is to treat any unknown case values as indicating the data of the type specified under the default branch. That typically closes any door on extensions because the default branch's data type is cast in stone and anything different from it on the wire would most likely cause a RPC_X_BAD_STUB_DATA exception. Again, the RPC_S_INVALID_TAG exception is specific to the described situation and enables the new client to accommodate its behavior as appropriate.
The [call_as] attribute has been created mainly to support types that cannot be used in RPC. It has been very useful when dealing with legacy applications that needed to add remoting. In this case, the legacy API interface for the application does not change. The [call_as] scheme provides a way to introduce a wrapper, allowing the use of RPC, and the remoting is done with the help of some new routines and frequently some other data types than in the original API.
However, the [call_as] attribute can be used as well when the currently remoted process needs to be modified in some way, for example, by adding argument verification or a mapping layer without really changing the arguments.
For the standard RPC interfaces, the call_as scheme works as described below. For DCOM it is very similar, except for different naming due to object nature of DCOM interfaces. See the discussion of the IconnectionPoint interface in Example 3 in the section "Examples of Problems with Cheating" for an example of call_as applied to a DCOM interface.
For a standard RPC interface, assume that the routine f1 in an interface IFace requires conversions between the user arguments and what is actually transmitted. The following example is a typical IDL setup:
[local]
long f1 ( <users parameter list> );
[call_as( f1 )]
long Remf1 ( <remotable parameter list> );
The specification indicates that instead of f1, Remf1 should be used when remoting. However, the specification would still cause the generated header file to define the interface using the definition of f1 (making the consistent old API interface for C linkage). Yet it would also provide stubs for marshaling Remf1. The prototype for Remf1 is also generated to the h file. What are not generated anymore are the stubs for f1.
The user needs to write client and server stubs for f1 that will serve as the wrappers performing the desired function: verification, conversion, and so forth. These are the f1 routine on the client side and the Rem1 routine on the server. The client app would call f1 and f1 would call Rem1 (that is the MIDL generated stub) to marshal arguments. On the server, RPC would unmarshal Remf1 arguments, then invoke the Rem1 wrapper provided by the user, which in turn would call the original f1. The user-supplied routines would be as follows:
long f1( <users args> ) // the hand-written client mapper
{
// Prepare remotable args.
Remf1( <remotable args> ); // remote call
// Prepare out user's args from [out] remotable args.
}
long <intf_name>_f1 ( <remotable args> ) // Remf1: the server mapper
{
// Prepare args for f1 server routine.
f1( <users args> ); // call original server proc
// Prepare remotable out args.
}
This scheme can be used to map arguments to completely different types when the interface is designed afresh or when backward wire compatibility is not an issue. It can also be used for backward-compatible solutions, such as the one involving mapping pointers to DWORDs mentioned elsewhere in this document.
For example, for the Notification methods mentioned in the section "Examples of Problems with Cheating," the scheme could be used as follows:
[local]
CONFIGRET
RegisterNotification(
[in] handle_t hBinding,
[out] PULONG_PTR Context,
[in] ULONG hProcess
);
[call_as(RegisterNotification)]
CONFIGRET
RegisterNotificationMapper(
[in] handle_t hBinding,
[out] PULONG Context,
[in] ULONG hProcess
);
[local]
CONFIGRET
UnregisterNotification(
[in] handle_t hBinding,
[in] ULONG_PTR Context
);
[call_as(UnregisterNotification)]
CONFIGRET
UnregisterNotificationMapper(
[in] handle_t hBinding,
[in] ULONG Context
);
The component's application writers would need to supply the following hand-written routines:
On the server side:
CONFIGRET
<intf_name>_RegisterNotification( // server mapper
/*[in]*/ handle_t hBinding,
/*[out]*/ PULONG pContext,
/*[in]*/ ULONG hProcess
)
{
ULONG_PTR APIContext = 0;
CONFIGRET Configret =
RegisterNotification( hBinding,
& APIContext,
hProcess );
if ( Configret-indicates-no-error )
MapAPIContextToOldContext( APIContext, pContext );
return Configret;
}
CONFIGRET
UnregisterNotificationMapper(
/*[in]*/ handle_t hBinding,
/*[in]*/ ULONG Context
)
{
// similar scheme
}
On the client side:
CONFIGRET
RegisterNotification(
/*[in]*/ handle_t hBinding,
/*[out]*/ PULONG_PTR pContext,
/*[in]*/ ULONG hProcess
)
{
ULONG RemoteContext = 0;
CONFIGRET Configret =
RegisterNotificationMapper( hBinding,
& RemoteContext,
hProcess );
*pContext = RemoteContext;
}
CONFIGRET
UnregisterNotification(
/*[in]*/ handle_t hBinding,
/*[in]*/ ULONG_PTR Context
)
{
// similar scheme
}The [transmit_as()] attribute and its close relative, the [represent_as()] attribute, provide a way to indicate that the transmitted type should be different from the type visible to the application. A typical definition and usage would be:
typedef [transmit_as(XMIT_TYPE)] ORIGINAL_TYPE APP_TYPE;
long FooXmit( [in] APP_TYPE * pAppObject );
The meaning of the specification is that an APP_TYPE that the application level code manipulates is associated with a different type, which is transmitted in lieu of the original. Every time an APP_TYPE is passed as an argument of a remote call, the data of XMIT_TYPE should be sent over the wire instead. The ORIGINAL_TYPE (that is, the application-visible APP_TYPE) does not have to be a valid remotable type. For this scheme to work, the user needs to supply four auxiliary routines: one for converting an object of APP_TYPE into an object of XMIT_TYPE, one for converting the other way, and two freeing routines, one for APP_TYPE objects and one for XMIT_TYPE objects. These routines would have the following fixed prototypes:
void __RPC_USER APP_TYPE_to_xmit( APP_TYPE *, XMIT_TYPE ** );
void __RPC_USER APP_TYPE_from_xmit ( XMIT_TYPE *, APP_TYPE *);
void __RPC_USER APP_TYPE_free_inst( APP_TYPE *);
void __RPC_USER APP_TYPE_free_xmit( XMIT_TYPE *);
The marshaling engine calls the routines at the appropriate times when marshaling and unmarshaling.
Note that these schemes could also be used for mapping. However, the mapping would be somewhat complicated here because the [call_as] wrappers make it easy to make the server behavior different from the client behavior. The [transmit_as] routines are shaped in terms of actions related to marshaling or unmarshaling, that is, to a receiver or sender, as opposed to client or server.
The wire_marshal() (see Endnote 2) attribute is somewhat similar to the transmit_as() attribute in that the general idea is to marshal some other type instead of the original application type. Also, user_marshal() is a type of wire_marshal(), similar to represent_as() being a type of transmit_as(). The main difference between the wire_marshal approach and transmit_as approach is that, with transmit_as, the marshaling and unmarshaling are done by RPC with the user converting or freeing the objects when necessary, while, with wire_marshal, the user is actually providing marshaling and unmarshaling code. A typical definition could look like this:
typedef [wire_marshal(XMIT_TYPE)] ORIGINAL_TYPE APP_TYPE;
long Foo( [in] APP_TYPE * pAppObject );
The user is expected to supply four auxiliary procedures, with the following prototypes:
unsigned long __RPC_USER APP_TYPE_UserSize(
unsigned long * pFlags,
unsigned long StartingSize,
APP_TYPE * pUserObject );
unsigned char * __RPC_USER APP_TYPE_UserMarshal(
unsigned long * pFlags,
unsigned char * Buffer,
APP_TYPE * pUserObject);
unsigned char * __RPC_USER APP_TYPE_UserUnmarshal(
unsigned long * pFlags,
unsigned char * Buffer,
APP_TYPE * pUserObject);
void __RPC_USER APP_TYPE_UserFree(
unsigned long * pFlags,
APP_TYPE * pUserObject);
This attribute was introduced to provide a way of transmitting nonremotable types that were either impossible or difficult to do with transmit_as, and to provide a way for improving performance. The user needs to know the arcane details of the NDR representation to use this attribute. While complicated, this approach provides good performance and a way to solve otherwise impossible problems.
However, one side effect of the specification and implementation for this feature is that, typically, an existing interface cannot be easily changed to use wire_marshal and still keep backward wire compatibility. This is, rather, a tool for providing a RPC solution to difficult legacy problems.
Starting with Windows 2000 (previously called Windows NT 5.0) Beta 3 release of the MIDL compiler, the same compiler can be used for 32b and 64b builds (i.e. 64b cross-building on 32b platform).
The compiler released with the Beta 3 had ver. 5.2.235. Windows 2000 Pro RC1 release featured MIDL ver. 5.3.268.
In 64-bit mode, the compiler supports IA64 and AXP64 platforms. A 64-bit stub is good for both platforms in a way similar to a 32-bit stub that is good for all the 32-bit platforms that have been supported over the years.
The old /env switch takes a new value now—that is, win64. There is also a new switch called /win64. The usage is fully symmetrical to /env win32 and /win32, so you can specify /env win64 or /win64 to get 64-bit stubs.
When an explicit switch is used, the compiler generates a stub for the given environment only. When the environment is not specified, the compiler generates dual stubs by default. (The old compiler default was to generate 32-bit stubs.)
A dual stub file consists of two separate parts: one for 32-bit environments and one for 64-bit environments, each guarded by an appropriate #ifdef. The proper stub files (that is, *_c.c, *_s.c, *_p.c) are generated dual. The other files, including the generated .h file, are platform-independent. We’ve found the dual stub files to be a very convenient tool for porting the existing code base and build environment.
As of summer 1999, (i.e. Windows 2000 Pro RCx releases), the 64b type libraries are supported by mapping 32b *.TLB files. When porting a project using TLBs, one should build 32b TLB files in the 32b environment for the project. Then the native 64b oleaut32.dll provides an appropriate 64b mapping for the TLB files at runtime.
-Oicf and -Os are the only relevant modes. -Oi and -Oic are not supported.
-Oicf is the main supported mode.
-Os, the auxiliary mode that is supported, is worth having as an alternative for the cases that –Oicf cannot handle.
As prescribed by the OSF/DCE standard, a pointer that can be null has a wire representation of 4 bytes. A [ref] pointer generally does not have a wire representation at all. This was motivated by performance issues for 32-bit systems where pointers were 4 bytes in memory.
For 64-bit platforms, a remotable pointer has a standard representation of 4 bytes on the wire for backward compatibility with 32-bit platforms. Pointers are fully supported at this point in a backward-compatible way. A performance optimization is planned for the future so that, if both remote parties are 64-bit, the pointer representation would be 8 bytes, thereby allowing for direct mapping between memory and marshaling buffer.
In addition, the following are tolerated as of MIDL 5.1.164, but not yet supported: __ptr32 and __ptr64. These are C qualifiers for pointers, of which only __ptr64 is currently supported by C. MIDL tolerates both (it passes them to the .h file as appropriate), but does not yet support them. In the future, the plan is to support both as appropriate. The attribute would mean that the pointer with a qualifier has the indicated size even on the other, nonnative platform.
We currently support both __int32 and __int64 in the regular compiler (that is, as of MIDL 5.1.164); __int3264 has been added to the 64-bit compiler.
__int32 and __int64 are simple integral types that are aliases for old timer types of int/long and hyper, respectively, and have the same size regardless of the environment.
__int3264 is a new integral type that has the following properties:
The wire representation is achieved by truncating the 64-bit value. When unmarshaling from 32-bit to 64-bit values, the signed value is sign extended; the unsigned value has the upper long 0 (zero).The sole motivation for introducing __int3264 is to allow for transparent support of U/INT_PTR and other polymorphic types (for example LONG_PTR, DWORD_PTR).
Note The polymorphic types like INT_PTR are supported only in regular RPC stubs and DCOM proxy files generated by MIDL. As oleaut32.dll does not support polymorphic types, they cannot be used in type libraries (i.e in *.TLB files generated by MIDL).
The basetsd.h file has the following definitions to support U/INT_PTR and U/LONG_PTR transparently:
#if ( 501 < __midl )
typedef __int3264 INT_PTR, *PINT_PTR;
typedef unsigned __int3264 UINT_PTR, *PUINT_PTR;
typedef __int3264 LONG_PTR, *PLONG_PTR;
typedef unsigned __int3264 ULONG_PTR, *PULONG_PTR;
#else // midl64
// old midl and C++ compiler
#if defined(_WIN64)
typedef __int64 INT_PTR, *PINT_PTR;
typedef unsigned __int64 UINT_PTR, *PUINT_PTR;
typedef __int64 LONG_PTR, *PLONG_PTR;
typedef unsigned __int64 ULONG_PTR, *PULONG_PTR;
#define __int3264 __int64
#else
typedef _W64 int INT_PTR, *PINT_PTR;
typedef _W64 unsigned int UINT_PTR, *PUINT_PTR;
typedef _W64 long LONG_PTR, *PLONG_PTR;
typedef _W64 unsigned long ULONG_PTR, *PULONG_PTR;
#define __int3264 __int32
#endif
#endif // midl64
In the future, a performance optimization is planned that would be symmetrical to the one planned for pointers: for 64-bit-to-64-bit remoting, the wire representation of __int3264 can be 8 bytes to enable block copying between the marshaling buffer and memory.
Note It should be emphasized that LONG_PTR by itself does not solve the cheating problem underpinning each case that requires LONG_PTR. The only thing the type does is to resolve wire incompatibility by introducing implied truncation and extension. Most likely, this is not good enough to make it fully compatible after the porting and the cheating issue has to be addressed separately. For pointer passing, some kind of mapping solution would be required.
The legacy application could not use 64-bit expressions. Integral expressions allowed in some attributes, like array attributes, such as size_is(), length_is(), and so forth, and in the switch_is() attribute, could use only expressions up to 32-bit integral size, that is, size of long. The 32-bit architecture of the late eighties influenced the OSF/DCE RPC specification so that an internal wire representation for conformant sizes and similar items are specified as longs. In addition, Microsoft's 32-bit implementation of RPC has supported sizes only up to 2 gigabytes to be consistent with limitations imposed by the Windows NT memory allocators.
Obviously, at some point support for 64-bit expressions will be needed for the 64-bit environment—for example, to support sending of conformant arrays bigger than 4 gigabytes or to make possible union switches based on 64-bit constants. However, this will require extensions to the existing OSF/DCE RPC standard and corresponding changes to MIDL and the marshaling engine. Nowhere in the system is there a situation where a sizing expression in IDL would be a 64-bit expression, or a LONG_PTR expression (that is, an expression upgraded to 64-bit on the 64-bit platform). Accordingly, for porting, the approach is for MIDL to stick to a no-64-bit expression rule. This would alert the application writer at the earliest possible moment that 64-bit sizes cannot be expressed and object to bigger-than-current limits sent on the wire.
If the application needed to use a LONG_PTR or similar for sizing after all, one solution could be to add size checking to the application prior to the call and use a call_as scheme. Another solution could be to do nothing and always truncate when the application is deemed safe. This can be accomplished with the simple cast in the .IDL file. For example:
void Foo( unsigned __int64 Size, [size_is((long)Size)] byte * pB );
This is a valid specification and calls to Foo have unambiguous semantics as long as the value of Size after cast is within the usual range of 0..2G-1. This is compatible with 32-bit environments.
To recap, future extensions are planned for sending more than 4 gigabytes of data; however, that is not a porting issue. Actually, using effective 64-bit values in these contexts implies changing wire representation and extensions to the OSF/DCE specification, and would definitely be supported at MIDL level by means of separate attribute(s) or other explicit syntax means.
Note One can always send more than 4GB of data, even on a current 32b system, by using IDL pipes. Please refer to the [pipe] IDL attribute in the documentation.
When in need of a UINT_PTR or other type defined in public\sdk\inc\basetsd.h, simply import this file to your IDL file. Don't #include the file directly in an IDL, because then the MIDL compiler would generate a .h file that has all the definitions from basetsd.h, and a likely result is a multiple definition collision at C/C++ level. You can also #include or import the public\sdk\inc\basetsd.idl file.
A handle defined as a pointer and sent by means of a DWORD typically must be serviced with a ULONG_PTR/DWORD_PTR or ULONG64 for a 64-bit platform. As explained elsewhere in this document, this type of cheating leads to inherent problems with truncation versus wire incompatibility. The following is review of design options discussed elsewhere in this document. The first solution is advised wherever possible.
Backward wire compatibility is not an issue, remoting from machine to machine happens.
Redesign the interface to use a context handle or an interface pointer as appropriate. Otherwise, define the cheating argument to be an ULONG64. This would work well both for 32-bit and 64-bit, with no performance penalty now or in the future.
Backward wire compatibility is an issue, remoting from machine to machine happens.
Use DWORD_PTR with the [call_as] scheme and supply a mapping on the 64-bit side from pointers to DWORDs. (See the section "[call_as]" and the section "Examples of Problems with Cheating.")
My handle is within a machine only, both on 32-bit and 64-bit, and it would never be sent from a 64-bit process to 32-bit process or the reverse.
You can use the first solution or you can set up the build separately for 64-bit and 32-bit, using separate 32/64-bit IDL files. In this way, you can have DWORD on 32-bit platform and ULONG64 on 64-bit platform.
You can also use DWORD_PTR, once the optimization for ULONG_PTR/DWORD_PTR to remote 8 bytes when talking between 64-bit processes and 4 bytes otherwise is supported.
What emerges from several of the examples discussed in this document is a need to have an API or a dictionary class providing mapping from pointers to DWORDs and back. When a mapped argument is sent away ([out] argument at the server side), the mapper would take an __int64 and map it to a DWORD; for the server receiving a mapped argument, a DWORD from wire would be mapped back to an __int64. The value from the wire would have to indicate a mapping already existing in the dictionary, otherwise this would be an error.
For the latest information on Windows 2000 Server, check out our World Wide Web site at www.microsoft.com/ntserver/ and the Windows NT Server Forum on the Microsoft Network (GO WORD: MSNTS).
-------------------------------------------------------------------------------------------------
© 1998 Microsoft Corporation. All rights reserved.
THIS IS PRELIMINARY DOCUMENTATION. The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.
This BETA document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENT.