July 1998

Use MSMQ and MTS to Simplify the Building of Transactional Applications

Download msmqmts798.exe (44 KB)

Mark Bukovec is a Program Manager for the Microsoft COM+ team. He has also worked with the ODBC, OLE DB, and MTS teams while at Microsoft. Dick Dievendorff is a Software Architect working on the Microsoft COM+ team. He also works on enterprise transaction systems and messaging systems.

Before the release of Microsoft® Message Queue Server (MSMQ), programming asynchronous distributed Windows®-based applications was not for the faint-hearted. Not only did you need in-depth knowledge of marshaling to get a client or server to properly queue, send, and receive asynchronous messages, but implementing proper failure recovery in case a message or queue went down was a time-consuming and difficult process. A server failure during the middle of an asynchronous operation could leave work in an ambiguous state. Ensuring that your asynchronous operation either committed or failed and rolled back to the last consistent state meant programming transactional support for your queue (an arduous task at best).       MSMQ, which shipped with the Windows NT® Option Pack (available at http://www.microsoft.com/Windows/downloads/ contents/Products/NT4OptPK), makes asynchronous programming possible for mere mortals. In addition to a simplified asynchronous programming model, MSMQ offers you built-in transactional support so your message queues can participate in Microsoft Transaction Server (MTS) transactions. This means that your asynchronous operations can be rolled into an existing transaction or into a new transaction you create. Not only will you be able to offer the flexibility of asynchronous messages, you can ensure that your data will not be compromised by inconsistent state. That's a pretty compelling argument for using MSMQ.       Let's explore how MSMQ simplifies building transactional applications. We'll describe how you can use MSMQ in an MTS application, and then walk you through a sample application that uses a transactional message queue. Finally, we'll talk about the evolution of MSMQ and its integration with MTS.       Before you continue (and if you have not already done so), see David Chappell's article in this issue for an overview of MSMQ. We'll assume a general understanding of the concepts of message queuing and MSMQ.        Transactional Message Queuing       Consider the following scenario. You've just built a distributed application for a pet supply company. Since most of the salespeople are on the road pitching new products, you've accommodated disconnected clients by using asynchronous message queues in your application. A salesperson can use your client app to fill out an order form and submit it for processing while unconnected to the network server. The message is stored in a queue on the client machine. The next time that salesperson connects with the company network, the Order information is uploaded to the server application for processing.       Let's say a particularly industrious salesperson has accumulated over 200 orders while crossing two states. The salesperson innocently dials into the company network and sends the messages. The server reads the Orders from the queue and updates an inventory database. Then the main server goes down while this work is happening (as it invariably does in the Se scenarios). Since the asynchronous operations were not wrapped in transactions, the salesperson has no idea which of the Orders were sent through for processing. Someone must then spend a good deal of time making sure that potential customers don't get charged for what they aren't getting shipped. If you had built your application using a transactional message queue, this scenario would have been accounted for and managed. When the company server went down and the asynchronous operation failed, the entire read message operation would have rolled back to its last consistent state. As far as the industrious salesperson is concerned, the Order has been processed. The server can retry the Operation at a later time. the Orders are safe and sound in the queue, so obviously the data was never sent to billing or shipping. The salesperson can send again at a later date, and in the duration, canvass pet stores in another two states.       Transactional message queuing is clearly important for maintaining the integrity of your data. It's also important to be able to integrate a message queue seamlessly into your distributed application. Let's take a look at how transactional message queuing combines the benefits of MTS and MSMQ.       First of all, if you aren't already using MTS (also available in the Windows NT Option Pack) to build your servers, you're doing a lot more programming than you should. See the sidebar "Scalable Servers with MTS"; for an overview of how MTS allows you to easily scale and deploy your applications. MTS also improves the fault-tolerance of your application through transactions. Transactions are powerful tools for maintaining the integrity of data. (For more information on transactions, see the sidebar "Transactions in Distributed Applications.") MSMQ and Transactions       You can break down the granularity of your MSMQ transactions by composing individual send or receive operations in a single transaction. If that smaller transaction fails, then the rollback to the last consistent state requires much less work from the server. In addition, any successful work already accomplished (through previous small transactions) is preserved. A succession of smaller transactions can be conducted asynchronously and then connected using message queues. Sending and receiving messages typically involves the following operations: Client adds a message to a queue, perhaps updating a local database. This work is the first transaction. Queue manager moves the message from the client computer to the server queue. The queue manager gets exactly one copy. Server gets the client's message from the queue, processes the request, and adds a message to a client response queue. This is the second transaction. Client retrieves its response message. This is the third transaction.       This means that each transaction can occur independently. If the client's first transaction commits successfully, then the message is stashed away in an input queue. If the server transaction fails, the transaction rolls back to its last consistent state—which means that the client's message is still in the input queue. If you had rolled the client send operation and the server processing of the message in one big transaction, then you would have lost the client's message in the rollback. Assuming that your application operates in a disconnected environment, you have to wait until your client gets back online to restart the entire process.       MSMQ can also participate in existing MTS transactions as a resource manager. Let's go over how transactions work in a distributed MTS-based application. In a distributed environment, an application partitions its work among multiple components. How is it possible to put the work of two different components into a single transaction when each component is committing its own updates to a different database? You need to have a transaction manager to coordinate all that traffic. MTS and MSMQ use the Microsoft Distributed Transaction Coordinator (MS DTC).       MS DTC oversees the two-phase commit protocol. Let's consider an example in which MS DTC is coordinating a two-phase commit with two resource managers, MSMQ and SQL Server™. In the first phase, MS DTC issues a "prepare to commit" message to both MSMQ and SQL Server. The resource managers participating in the transaction make the results of the transaction durable but do not actually commit the transaction. MS DTC then waits for MSMQ and SQL Server to report that they have prepared the transaction before continuing to the second phase of the commit protocol.       During the second phase, MS DTC tells MSMQ and SQL Server to commit their transactions. Only when both resource managers have reported that they have successfully committed their transactions can the distributed transaction be committed.       For the low-level view on how MSMQ interacts with MS DTC in a distributed transaction, read the Microsoft OLE Transactions specification. If you're programming for MSMQ, though, all you need to know is that MSMQ enables you to roll your messaging operations in a transaction as well as participate in existing transactions managed by MS DTC. MSJQueue       Now that you're familiar with the concepts behind MSMQ programming for MSMQ, let's look at actual code. We'll go over the general architecture of the MSJQueue sample first and tell you how to install and deploy the sample. Once you understand how to set up MSJQueue, we can drill down into how each object works. Finally, we'll discuss exactly how messages get passed through the queues.       The MSJQueue sample app is a Visual C++®-based application. To reduce complexity, we do not provide an IDispatch implementation of our objects. You should investigate the MSMQ ActiveX components, which provide fuller access to MSMQ functionality than the sample Queue object that we provide here. The ActiveX component also provides access to MSMQ for programmers using Visual Basic®.       The MSJQueue sample demonstrates how to build a transactional message. The client executable and the server executable each create a Queue object to run in their respective processes (see Figure 1). On the client side, the Queue object acts as a helper object for queue manipulation functions, which do things like opening queues, sending messages, and closing queues. The Queue component on the server is the "listener" that waits for messages to arrive in the queue.       In addition to the Queue object, the server executable also creates the Message Server object, which processes each message as it arrives.

Figure 1  MSJQueue Client and Server Sample

The Queue object supports a single interface called IQueue that performs most queue manipulation operations. Figure 2 describes the methods of the IQueue interface. MSJQueue Client and Server       The MSJQueue can be deployed on an MSMQ independent client. Like MSMQ servers, independent clients can create queues, modify queues, send messages, and receive messages.       Independent clients can create queues and store messages on the local computer without a live connection to the MSMQ server. MSMQ independent clients differ from servers in that they do not have the intermediate store-and-forward capability of MSMQ servers, nor do they store information from the distributed MSMQ database. MSMQ can have dependent clients as well, although dependent clients cannot function without synchronous access to an MSMQ server. Dependent clients are not useful in a disconnected environment. Note that you have to create the MSJQueue queue using the MSMQ Explorer when you set up the client application. We'll explain this in more detail later.       The MSJQueue client is an executable named Client.exe that sends data using the Queue object. The client optionally takes a single command-line argument that specifies the name of the queue. If you omit this argument, the default queue is "MSJQueue," which the application assumes exists on the same computer as the client.       The client creates the Queue object as an in-process server. The Queue object supports a single interface: IQueue. We have defined the IQueue interface as a simple abstraction over the basic queue manipulation functions, such as opening queues, sending messages, receiving messages, and closing queues (see Figure 2).       The MSJQueue server executable creates two objects: Queue and Message Server.       Queue acts as a helper object to dequeue messages. The Message Server object is an MTS component, and lives on the middle tier of our application. Message Server executes the main functionality of the server by reading the message from the queue and storing the data from the message in a database. Setting Up and Deploying MSJQueue       Now that we've gone over the components of the sample application and described how the message queue works, you should set up the MSJQueue sample so you can track the discussion of how messages are actually sent and received. You can find MSJQueue included in the code sample listed at the top of this page. You can run both the client and server on a single computer for the simplest setup. To simulate a production environment, you should set up the client application on one computer, and the message server on another machine. Setting up your client machine in a distributed environment requires Client.exe, a registered queue object, and an installation of MSMQ.       The server side of MSJQueue requires the following software: MTS, MSMQ (with access to a primary enterprise controller), and SQL Server. (Note that you should modify the MSJQueue DSN (MSJQ_DSN) to match your system administrator password for SQL Server.)       Building the MSJQueue application automatically registers the DLLs on your computer. If you deploy your server on another computer, use regsvr32.exe to register it. An MTS component must be a COM component DLL. It must export a class factory, be self-registering, and so on. We used ATL to generate this basic COM plumbing for us. Installing Message Server in a package registers the component and allows it to execute within the MTS runtime environment.       To package your components, first create an empty package. Open the MTS Explorer. Select the Packages Installed icon under My Computer and select New and then Package. Choose the Create an Empty Package option and enter MSJQ as the new package name. Keep the default setting of Interactive User and click Finish.       Next, add the DLLs to the new package. Right-click on the Components folder in the new package and select New and then Component. Select the Import component(s) that are already registered option since building the sample project registers the DLLs automatically. Select the Message Server component from the component list.       Finally, configure the package and component properties. The MSJQueue package will now be listed in your Packages Installed folder (see Figure 3). Right-click on the MSJQ package and click Properties. Click the Activation tab of the property sheets and change the setting to Library. By changing the activation setting, you are specifying that your package runs in-process with its creator, which is the server executable in this application. Next, right-click on the Message Server component in the MSJQ package. Select Properties and go to the Transaction tab. You will notice that "requires a transaction" has been set.

Figure 3 MSJQueue Package in MTS Explorer

The transactional attributes for our components determine transaction composition of our server-side message processing (see Figure 4). You can either configure the transaction attribute of the component using the MTS Explorer, or you can specify the M in the .IDL of the component by including mtxattr.h. When you run the application, Message Server becomes the "root" of the transaction. You may be wondering how we determine which component is the transaction root.

Figure 4  Message Processing in a Transaction

"Transaction required" means that when an object instance is created, it will be part of the creator's transaction. If the creator is not transactional, a new transaction is started for the created object. In our application, QueueServer is not a transactional component; it is not an MTS component at all. QueueServer is considered the base client. MTS defines base clients as the primary consumers of MTS objects, although base clients execute outside of the MTS runtime environment.

Figure 5 Creating a Queue in MSMQ Explorer

After you package your server components, you must create a queue using the MSMQ Explorer. To do so, right-click on the server computer and select New and then Queue. Enter the name of the new queue as MSJQueue and click the Transactional checkbox. Marking the queue as transactional is required for the application. You've just created a new message queue (see Figure 5). If you want to create queues programmatically, see the MSMQ SDK documentation for information on MQCreateQueue. Sending and Receiving Messages Using MSMQ Queues       Now let's go over what sending and receiving messages using the MSJQueue application entails. If you want to send messages when the client computer is not connected to the server, you will have to specify a format name for the queue. You have to provide a format name because the call to MQPathNameToFormatName must be able to access the MQIS; specifying a pathname will cause Client.exe to fail if the client cannot access the server.       The client does check to see if the queue that you entered is already a format name. A format name is a unique name generated by MSMQ that uniquely identifies the queue. Format names are either public, private, or direct. A helper function looks to see if the string is prepended with "PUBLIC=" or "PRIVATE="; if the string is prepended with either of the Se extensions, then the client has specified a format name to identify the queue. If a format name is entered, the client does not call MQPathNameToFormatName. Messages are stored on the client until a connection is reestablished with the server. The format name uses the queue's GUID to identify the queue. In the Offline scenario, you should specify PUBLIC=guid. (Don't include the standard curly braces with the GUID.) To determine the queue's GUID, right-click on the queue in the MSMQ Explorer and choose Properties. The GUID is located on the General tab: you can select and copy it from there. Finding a Queue       Given a queue name, the client calls the Open method. Open checks to see if the queue name that was passed as an input parameter is an MSMQ format name. If the client hasn't specified a format name to identify the queue, the code assumes it's been given a pathname (see Figure 6).       A pathname is of the form machine name\queue name. A dot can be substituted for the machine name if the queue resides on the local machine, such as in the default queue pathname ".\MSJQueue." Given a pathname, the Open method calls MQPathNameToFormatName to convert it to a format name. Note that MQPathNameToFormatName requires access to MQIS, which is the MSMQ information store that resides in a SQL Server database. Your deployment configuration affects how to specify the queue.       Public queues are registered in MQIS and are available from other computers. A private queue can only be accessed by the local computer, which is not as interesting for this application scenario.       Specifying a public or private queue requires a GUID, which means you can move the location of the queue to another computer and not break existing clients. MQIS will resolve the GUID to the new location.       Direct format names are similar to queue pathnames, except that they can specify the network address and network protocol of the target machine. You can also specify OS as the protocol to indicate that the computer's native protocol should be used. Like pathnames, direct format names do not require access to MQIS. Opening a Queue for Sending a Message       Once you have the format name for the queue, you can pass this into a call to MQOpenQueue. Specify MQ_SEND_ ACCESS to indicate that you are going to send messages to the queue.       The queue must be opened with either the send, peek, or receive access settings. Since the send access setting was chosen, it remains in effect for the time that queue is held open. To change the access mode to enable reading messages, you'd have to call the Queue object's Close method and then call Open, or create another instance of the Queue object and call Open. Also, when specifying send access, you must pass in MQ_DENY_NONE, which indicates that the queue remains available to others while you have it opened.       A successful call to MQOpenQueue returns a queue handle, which, like other Windows handles, is an opaque data type. You will use this queue handle to reference the queue when sending messages to it.       Our Queue object works with one queue at a time. Subsequent calls to Open will close the queue referenced by an existing queue handle prior to calling MQOpenQueue. In this example, we are working with only one queue. Transactional Message Queues       Let's back up for a moment and discuss the nature of our queue. It's a transactional message queue, which means it can only accept messages that are sent as part of a transaction.       Remember, three transactions are typically required when building an application like ours: The first transaction to guarantee enqueuing of the request The second transaction when dequeuing the request to the server The third transaction when the server sends a response to the client       The request queue on the client is handled on our behalf by MSMQ. Therefore, our transaction spans sending the message from the client to the server and implicitly includes the request queue.       Our application does not use a client-side response queue. A response queue would receive acknowledgment messages from the server indicating that the request message has been read from the queue and the transaction has been processed. Sending a Message       Once the queue has been opened, the client calls the Queue object's Send method. The Send method calls MQSendMessage, which takes a queue handle, a pointer to a MQMSGPROPS structure, and the transaction in which the message will be included (see Figure 7).       The queue handle is stored in the m_qh member of the Queue object after a successful call to Open. The other two parameters to MQSendMessage require a little more explanation.       The MQMSGPROPS structure allows you to describe the message. This structure contains one or more message properties. In this example, only PROPID_M_BODY is used, which is the body of the message. Other message properties include the authentication level, label, priority, and response queue.       Our sample's MESSAGEFORMAT structure is defined in MessageFormat.h with a signature, version, creation time, message index, and the sizes of the two strings. The strings the Mselves are appended to the end of the structure (see Figure 8).

Figure 8  An MSJQueue Message

You have to send the message as bytes because MSMQ does not interpret the body of the message. Your choices for PROPID_M_BODY_TYPE are limited to simpler types. When you read the message, you'll have parse this byte stream to get the data members of the structure.       MQSendMessage optionally takes a parameter that specifies whether or not the message is to be sent as part of a transaction. Since you are using a transactional queue, you must indicate a transaction, the value of which depends on whether you are currently part of an MTS transaction. Your Send method calls the GetObjectContext function, which is used to determine if you're running in MTS.       If GetObjectContext succeeds, you will pass in MQ_MTS_ TRANSACTION to indicate that you want to be part of the MTS transaction. This is an implementation choice that allows our application to run outside of MTS; you do not need to check for the existence of object context if you know your application will always run under MTS. Otherwise, you pass in MQ_SINGLE_MESSAGE, which specifies an MSMQ internal transaction. An MSMQ internal transaction uses MSMQ's own mechanism to guarantee message delivery. Since no other resource managers are involved in the transaction, an MSMQ internal transaction takes less time to execute.

Figure 9 MSJQueue Messages

Now, try running the client and looking at the queue in the MSMQ Explorer. You should see 10 messages in MSJQueue as in Figure 9. If you right-click on one of the messages and choose Properties, you actually view the contents of the message just as you had packaged it (see Figure 10).

Figure 10 The Contents of an MSJQueue Message

QueueServer       The QueueServer component is a "listener" that waits for messages to arrive in the queue. QueueServer loops through the wait-read cycle. If it gets back any other error, such as not being able to open the queue, then QueueServer tears itself down (see Figure 11).       You've implemented the QueueServer as a local server for simplicity, but another useful implementation would be as a Windows NT service because services should not require any user input. As a service, it could run without an interactively logged-in user, making it available whenever the computer is running.       The QueueServer creates a Queue object, which it uses to check the queue for arriving messages. QueueServer also creates the Message Server MTS component. Message Server writes the contents of the message to a database. Message Server composes the read from the message queue and the database update as a single transaction.       QueueServer calls the Queue object's Open method to open the queue. This time MQ_RECEIVE_ACCESS is used to specify that you want to open the queue for reading messages. Remember from our previous discussion that you have to indicate to MSMQ how you are going to use the queue before opening it.       You then call MQReceiveMessage to read from the queue. You will pass it a timeout value of one minute (see Figure 12).       If no messages are found in the queue before the timeout elapses, then MQReceiveMessage returns MQ_ERROR_ IO_TIMEOUT. You also specify MQ_ACTION_PEEK_CURRENT, which will allow you to read the message but not remove it from the queue. The "current" part of the setting refers to the current cursor location. A cursor is defined by a call to MQCreateCursor and allows you to scan through messages in a queue. Because you did not pass in a cursor as a parameter to MQReceiveMessage, you will see the first message in the queue.       You are only peeking at the queue to see if there's a message, which is a nontransactional operation. Once you've found a message in the queue you'll go back and read the contents, but for now, you only read the size. In the message properties parameter, you specify PROPID_M_ BODY_SIZE, which will return the size of the message in bytes. You can use this value to size your buffer when you later do your read and remove the message from the queue.       You may be wondering why we don't just read the contents of the message in one shot. The problem is that you can't wait within a transaction on an empty transactional queue. For this reason, you specify MQ_NO_ TRANSACTION. Processing the Message       Once you've found a message, QueueServer calls the DoWork method on the Message Server component to do the transactional work (see Figure 13). You wrap both, dequeuing the message and the database update in an atomic transaction. In case of failure, the transaction will be aborted, and you will go back to the previous state; the database update is rolled back and the message read is undone so that it remains on the queue. Using an MTS component allows you to automate the implementation of this transaction.       On the call to DoWork, you pass in the interface pointer passed to our Queue object. You can do this because you've created your components in the same process and implemented the M so that they aggregate the freethreaded marshaler. Components that aggregate the freethreaded marshaler can pass interface pointers to components in other apartments within the same process without marshaling. Not only is this a big performance gain, but it simplifies your code. If your components lived in separate processes, you would have to pass around information stored in your Queue object to another instance of the Queue object in the Other process.       You may be wondering why we created this additional component just to compose a transaction. Why not just hardcode the transaction scope within the QueueServer component and avoid MTS altogether? One reason is component reuse. You've coded the Queue object to be a useful helper class for basic MSMQ operations. Your Queue object detects the presence of MTS when deciding the transactional setting for queue operations. The transactional attributes of the component using the Queue object (which is, in this case, Message Server) determine the scope of the transaction. Composing transactions using MTS offers more flexibility and helps achieve truly reusable components.       Also, there's the simplicity of the MTS API. Before returning, DoWork calls SetComplete on the Object context. If an error occurs, then DoWork calls SetAbort. the Se calls let MTS know that the MessageServer component has finished its work and can either commit or abort the transaction.       Your transaction includes an update to a SQL Server database. First, you open a connection to your database. You chose to use Active Data Objects (ADO) for your database access. You then take our buffer and cast it as a MESSAGEFORMAT structure. Next, you insert the members of the structure into a query string as columns of an UPDATE query. Finally, you execute the query and close your connection. Note that by using ADO under MTS, your database connection is automatically pooled. When the next request comes into the server for the same connection, the One you just released becomes available, saving time required to create a database connection. As connections to the server increases, connection pooling ensures that our application scales by releasing resources as soon as possible for reuse. The Future       In the upcoming release of COM+, the MSMQ features will be integrated with MTS as the Queued Components service. Queued Components will allow you to use queuing in a more RPC-like fashion. In our sample we had to marshal the data ourselves by packing it into a message. This requires both client and server to know a priori what the message will look like, which lacks the "discoverability" of COM component type information. Also, it took us a good bit of code to parse the Se messages.       Queued Components will simplify matters by automatically marshaling your data in the form of an MSMQ message. To the component programmer, method calls will look more familiar. Thus you can simply access data as object properties and let Queued Components do the hard part. There will be some limitations; for example, Queued Components cannot have [out] parameters, because in a disconnected scenario, there may be no client connected when the call is serviced. Regardless, Queued Components will allow you to reduce code in your sample application by setting component attributes in an administrative utility and letting COM+ do the rest. From the July 1998 issue of Microsoft Systems Journal.