January 1999

Code for this article: jan99HoC.exe (2KB)

Don Box is a co-founder of DevelopMentor, a COM think tank that educates the software industry in COM, MTS, and ATL. Don wrote Essential COM and coauthored the follow-up Effective COM (Addison-Wesley). Reach Don at http://www.develop.com/dbox.

After reading my October 1998 column on essential books for COM developers, a colleague asked me if I had given up writing technical articles for MSJ and had switched to a more soft-core focus, as would befit a developer of my advanced age (36). Not wanting to be put out to the pasture of technical management just yet, I decided to ditch the "Top Ten English Phrases You Can Put into a GUID" piece I had originally planned for this month and turned my attention to something a bit more grungy. Since I just spent four hours FTPing Windows NT® 4.0 Service Pack 4, I can think of no better topic for this month's column than the infamous type library marshaler that just got a major boost in SP4 because it now supports structures as parameters. (Note: the information given here may or may not apply to Windows 2000.—Ed.)       Most objects don't elect to implement the IMarshal interface. This is because these objects choose to let the COM runtime build a proxy/stub pair to allow references to the object to be passed across context boundaries. (Prior to Windows 2000, proxies were only needed across apartments, but alas, Windows 2000 subdivides apartments even further.) These objects are said to use standard marshaling, and are the objects I'll focus on in this column.       The COM library and wire protocol explicitly support standard marshaling for only one COM interface: IUnknown. The proxy manager and stub manager used in standard marshaling only know how to make this one interface work across contexts. Fortunately, the proxy and stub managers are extensible, which allows other interfaces to be used with standard marshaling. As shown in Figure 1, the proxy manager can load interface proxies to translate method invocations into ORPC requests. The stub manager can load interface stubs, which then turn these request messages into method invocations on the actual object, as shown in Figure 2.       All interfaces beyond IUnknown must have an interface proxy and stub registered on the system to work with standard marshaling. This is true for interfaces defined by Microsoft or by third parties. For example, if the IStorage interface is to be remotable via standard marshaling, its IID must be listed under HKEY_CLASSES_ROOT\Interface, and it must also have a ProxyStubClsid32 subkey pointing to the in-process server that implements the interface proxy and stub. Fortunately, IID_IStorage has such an entry. If you were to look up the GUID stored at IStorage's ProxyStubClsid32 entry ({00000320-0000-0000-C000-000000000046}) under HKEY_CLASSES_ROOT\ CLSID, you would find that it points to OLE32.DLL, and the friendly name of the class is either PSFactoryBuffer (under Windows 2000) or oleprx32_PSFactory (under Windows NT 4.0). This identifies the proxy/stub factory for all of the remotable interfaces that make up the COM core. MIDL and Standard Marshaling       Since the interfaces you define are not considered part of the COM core, the proxy/stub factory that's built into OLE32.DLL can't cook up interface proxies or stubs for your interfaces. Fortunately, the MIDL compiler emits the C source code that, once properly built and registered, can produce the goods to allow your interface to work with standard marshaling. Once you register the resultant DLL, the entries needed to find your proxy/stub DLL will be inserted into the registry. That's it. No code to write. Just install the DLL and your interfaces work magically, whereas before cross-context clients got E_NOINTERFACE back from QueryInterface.       To appreciate the type library marshaler, you need to first scrape through a MIDL-generated proxy/stub DLL. All proxy/stub DLLs are expected to implement a class factory that implements IPSFactoryBuffer in lieu of IClassFactory. The IDL for IPSFactoryBuffer is shown in Figure 3. To create an interface proxy, the proxy manager executes the sequence shown in Figure 4.       Note that the function CoGetPSClsid actually exists and is used by the proxy and stub managers to locate interface marshalers. This function first looks in an in-memory table to see if either the registry key has already been looked up or someone inside the process has preregistered an IID-to-PSCLSID mapping using CoRegisterPSClsid. If the IID is not found in this table, CoGetPSClsid looks in the registry for the ProxyStubClsid32 entry and caches it in the table to speed up subsequent lookups. Once the CLSID for the interface marshaler is determined, the marshaling code is loaded using normal COM techniques (CoGetClassObject) and the CreateProxy method is called. The stub manager uses a similar procedure to create an interface stub, but calls CreateStub instead of CreateProxy.       The MIDL-generated implementations of CreateProxy and CreateStub work in different ways depending on how you compiled your IDL file. If you simply chose the defaults for MIDL with no command-line switches,

midl.exe bob.idl

MIDL will emit the C source code for the interface proxies and stubs. This code resides in the _p.c file (in this case, bob_p.c) and takes the form of functions with names like IBob_MethodX_Proxy and IBob_MethodY_Stub. If you step into your proxy in a debugger, the _Proxy routines are actually stored in the vtable of the interface proxy. On the server side, if you look at the call stack while stopped inside a method call, you'll notice that your method is called directly by the corresponding _Stub method.       If you really feel like stepping through the MIDL-generated code, you will need to remove the #pragma code_seg(".orpc") directives from the _p.c file and do a debug build of your proxy/stub DLL. The reason for removing the pragma is that COM-aware debuggers don't step into code in a segment named .orpc since MIDL-generated code rarely has errors.       Figures 5 and 6 show the structure of a MIDL-generated interface proxy and stub. Note that the interface proxy's vtable is populated with the MIDL-generated _Proxy routines. Also note that the interface stub maintains a vector of function pointers to the _Stub routines. This vector is used by the standard implementation of IRpcStubBuffer::Invoke (CStdStubBuffer_Invoke from RPCRT4.DLL) to dispatch the incoming method request. While perfectly reasonable, the type of interface marshalers shown in Figures 5 and 6 have become passe in modern COM, largely because they contain compiled C code to perform the marshaling.       Most COM developers now choose to use interpretive marshalers. As of Windows NT 4.0, the MIDL compiler can emit byte code in lieu of C code by passing the /Oicf flag:

midl.exe /Oicf bob.idl

An /Oicf-based marshaler benefits from increased performance due to a considerably smaller memory footprint. Figure 7 shows the memory savings for a variety of IDL files. When building an /Oicf-based marshaler, the _p.c file contains two byte strings in lieu of executable C functions. The most important byte string is called MIDL_PROC_ FORMAT_STRING. This string contains byte code describing every method signature contained in the IDL file. Figure 8 shows the format string for the following IDL:

[ uuid(12341234-2134-2134-5235-123563234431) ] interface ISomeInterface : IUnknown { import "unknwn.idl"; struct BOB { long a; long b; }; HRESULT Eat([out, retval] long *pn); HRESULT Sleep([in] struct BOB *pBob, [out, retval] long *pn); HRESULT Drink([in] struct BOB *pBob, [out, retval] long *pn); }

Note that the representation for each method is laid out back-to-back inside the string.       The following pseudo IDL shows how each method description is laid out

struct METHOD_INFO { byte handle_type; byte old_flags; long reserved; short method_index; short stack_size; short in_param_hint; short out_param_hint; byte oi2_flags; byte cParams; [size_is(cParams)] PARAM_INFO params[]; };

with each parameter described as follows:

struct PARAM_INFO { short flags; short stack_offset; union { struct { byte fc_basetype_code; byte reserved; }; short type_offset }; };

Note that the exact format of /Oicf strings is undocumented, completely unsupported, and subject to change at any time. But it is still interesting to look at /Oicf strings because it gives you insight into how COM works.       Note that for the simple method Eat, the entire method signature can be contained inside the MIDL_PROC_FORMAT_STRING (offsets 0-27). The other two methods, Sleep and Drink (offsets 28-61 and 62-95, respectively), both of which have a pointer to a struct as a parameter, have PARAM_INFOs that provide offsets into a second string called the MIDL_TYPE_FORMAT_STRING. You can see these "pointers" into the type format string at offsets 48 and 82 in the proc format string (where the struct BOB * appears).       The MIDL_TYPE_FORMAT_STRING contains byte code descriptions of any complex (non-basetype) parameters used in the IDL. Figure 9 shows the type format string that goes with the IDL fragment for ISomeInterface shown earlier. Rather than duplicate the potentially complex description of a datatype in the proc format string, MIDL puts type definitions into the type format string once. This means that each occurrence of the type as a parameter only needs to refer to the type format string, which costs a constant two bytes in the proc format string. It is common for the type format string to contain references to itself to reduce the overall size of the format string by eliminating redundancy. Finding the Format Strings       So how do these two strings get bound to interface proxies and stubs? Figures 10 and 11 show the layout of /Oicf-based interface proxies and stubs. Note that the format strings can be found by indexing off the back of the COM-style vtable. This is fairly trivial for the interface stub, since the implementation of IRpcStubBuffer::Invoke simply needs to back up the vptr by 12 bytes, dereference the pointer found there, and 12 bytes past that you'll find the pointer to the proc format string.       The type format string can be found in a similar fashion. Inside the implementation of IRpcStubBuffer::Invoke, all you need to do is calculate the offset into the format string based on the invoked method. This is also easy, as MIDL generates a table of method index-to-offset entries. With the format strings in place, the IRpcStubBuffer::Invoke implementation jumps into the important NdrStubCall2 routine. This RPCRT4.DLL routine is the function that takes the format strings and does the interpretive unmarshal. Once the unmarshal occurs, NdrStubCall2 invokes the object's method implementation to dispatch the call. When the object returns from the method, NdrStubCall2 then uses the format strings to marshal the [out] parameters prior to returning control to the standard implementation of IRpcStubBuffer::Invoke.       The interface proxy works in a similar way. However, because a variable-length vtable must be returned to the client, MIDL generates a vtable populated with -1s rather than actual function pointers. Prior to letting the client get a hold of this vtable, the runtime populates these entries with small thunking routines named ObjectStublessClientXXX, where XXX is the method index. These thunks just shove the method index into the ecx register and jump to a common function ObjectStubless, also in RPCRT4.DLL. Here is the code for ObjectStublessClient9:

mov ecx, 0x00000009 ; unique vtable index jmp ObjectStubless ; another symbol in RPCRT4

ObjectStubless simply calls into ObjectStublessClient, passing the method index (from ecx) as a parameter. Finally, ObjectStublessClient teases out the format strings from the vtable and jumps to NdrClientCall2. Like NdrStubCall2, this RPCRT4.DLL routine performs the interpretive marshaling and unmarshaling just as if a compiled proxy and stub were in use.       While the previous discussion has glossed over many details (virtually all of which are undocumented and subject to change), the upshot is that the vtables in an /Oicf-based marshaler are annotated with format strings that completely describe the interface. The folks who work on OLE32.DLL and RPCRT4.DLL need to know every detail, but most of us don't. Just use /Oicf and watch your proxy/stub DLL size shrink dramatically. The Type Library Marshaler       So what does this have to do with the type library marshaler? Tons. Interfaces marked with either the [oleautomation] or [dual] attributes get special treatment when the underlying type library is registered. When these specially annotated interfaces are encountered by RegisterTypeLib (or LoadTypeLib in legacy mode), COM adds ProxyStubClsid32 entries for the interface with the value {00020424-0000-0000-C0000-000000000046}. This GUID corresponds to the class PSOAInterface that is registered as living in OLEAUT32.DLL, the OLE automation DLL. Because of the DLL that it lives in, this marshaler is sometimes called the [oleautomation] marshaler, although it is also called the type library marshaler or the universal marshaler. I'll refer to it as the type library marshaler, since it really has very little to do with IDispatch. (In fact, it is common to use the [oleautomation] attribute on interfaces that don't derive from IDispatch directly or indirectly.)       The class factory for the type library marshaler does something very tricky in its CreateProxy and CreateStub routines. Rather than return a statically compiled vtable (which is impossible given the fact that the requested interface didn't exist when OLEAUT32.DLL was built as part of the OS), the type library marshaler actually builds an /Oicf-style proxy and stub based on the type library for the interface. Because there is no efficient way to find the ITypeInfo for an arbitrary interface, the LIBID and version of the interface's type library must be stored under the HKCR\Interface\{XXX}\TypeLib registry key.       Once the ITypeInfo from the interface has been loaded, the type library marshaler then calls into RPCRT4.DLL to request an interface proxy or stub via the following two (undocumented and unsupported) routines:

HRESULT CreateProxyFromTypeInfo( ( [in] ITypeInfo *pTypeInfo, [in] IUnknown *pUnkOuter, [in] REFIID riid, [out] IRpcProxyBuffer **ppProxy, [out] void **ppv ); HRESULT CreateStubFromTypeInfo ( [in] ITypeInfo *pTypeInfo, [in] REFIID riid, [in, unique] IUnknown *pUnkServer, [out] IRpcStubBuffer **ppStub );

Note that these two functions have signatures that are virtually identical to the IPSFactoryBuffer methods. One notable exception is the presence of an ITypeInfo parameter, which is used by RPCRT4 to reverse-engineer the /Oicf format strings.       These two routines make the type library marshaler's job quite simple. Figure 12 shows the pseudocode for the type library marshaler's implementations of CreateProxy and CreateStub. Note that the two methods simply load the type library and pass the requested interface's ITypeInfo to the corresponding RPCRT4 routine. The two CreateXXXFromTypeInfo routines do a fair amount of caching to amortize the cost of generating vtables and format strings. The type library marshaler also performs some caching to reduce the overhead of looking up the type information at each unmarshal.       Once the unmarshal is complete, the resultant interface proxy and stub are no different than the ones generated by the MIDL compiler. The primary performance difference is that /Oicf-based proxy/stub DLLs can initialize the first interface proxy or stub faster since the /Oicf format strings are precompiled in the DLL. The type library marshaler must generate them on the fly at the first unmarshal. Choices, Choices, Choices       So given the similarities between MIDL-generated /Oicf-based marshaling DLLs and the type library marshaler, when should you choose one over the other? Figure 13 shows the advantages and disadvantages of the three types of marshalers. Non-/Oicf-based marshaling DLLs are included due to the lack of support for /Oicf under Windows NT 3.5x and non-DCOM builds of Windows® 95. Clearly, /Oicf-based proxy/stub DLLs are slightly faster at unmarshal time and have no limitations on parameter types. Type library-based marshaling has the advantage of working with both 16 and 32-bit processes.       Which is more convenient? It is actually a wash since in a distributed application the marshaler needs to be installed on both the client and server machines. Granted, your server may register its type library on the server machine, but it is often inappropriate to register the server on the client machine, which means you need to somehow get your type library registered on the client machine. Given the choice of registering a type library versus registering a proxy/stub DLL, I'd pick the latter any day since DLLs self-register and type libraries do not. Of course, for doing 16-bit work or for quick-and-dirty (non-distributed) COM servers, the type library marshaler is awfully convenient, and that's its primary reason for existence. Have a question about programming with ActiveX or COM? Send your questions via email to Don Box: dbox@develop.com or http://www.develop.com/dbox.

From the January 1999 issue of Microsoft Systems Journal.