Putting Microsoft Message Queue to Work, MIND August 1998

This article may contain URLs that were valid when originally published, but now link to sites or pages that no longer exist. To maintain the flow of the article, we've left these URLs in the text, but disabled the links.

This article assumes you're familiar with C/C++, COM, and Visual Basic.
Download the code (2KB)

Putting Microsoft Message Queue to Work
Chris Dellinger

MSMQ is a cool new product, but where should you use it? We'll take you through some real-world projects that really take off with MSMQ support.

All right, you know what MSMQ is, but perhaps you are having a hard time finding practical applications for it in your enterprise development. (If you don't know what MSMQ is, see "Microsoft® Message Queue is A Fast, Efficient Choice for Your Distributed Application," by David Chappell, in the July 1998 issue of Microsoft Systems Journal.) Or maybe you're thinking, "It sounds great, but can't I already do the same thing with existing tools?" To answer these and other questions, let's take a look at two real-world applications I have been working on that show where MSMQ really excels.
      Any new technology must offer a compelling reason to use it. For me, the technology should meet one of four criteria: previously unavailable functionality, better performance, less code to write, or downright coolness. MSMQ is one of those rare technologies that meets all four.
      Among my favorite features of MSMQ is its offline processing support, commonly called store and forward. With store and forward, an application writes information to disk when it's disconnected from the network, and later, when network connectivity is restored, routes this information to the appropriate location. If you've done any serious enterprise development, you've had to write this type of code at one time or another, especially when dealing with communication between two systems. Thankfully, those days can now be put behind you.
      With the introduction of MSMQ, developers now have a far more robust and better-performing store-and-forward mechanism than they could reasonably write on their own. And MSMQ is supposed to be the messaging standard for Windows NT for years to come (no more reinvention of the wheel). Best of all, as you'll soon see, MSMQ provides these store-and-forward features with minimal coding. Already, MSMQ has met my requirements of previously unavailable functionality, code reduction, and coolness.
      Another great feature of MSMQ (one that helps meet my criteria of improved performance) is its ability to send messages asynchronously. There are quite a few areas in today's distributed systems where performance can be improved through asynchronous messaging.

Store and Forward
      The advent of the laptop computer revolutionized corporate sales. Countless sales computer systems have been created over the last decade, not without a fair share of pain and heartache. One of the biggest problems for a laptop sales system is the fact that the inventory, accounts receivable, and shipping systems are usually very large systems residing at corporate headquarters. Sooner or later, all the new sales information entered into the laptop by the salesperson needs to be entered into the system back at HQ.
      When designing an application for a laptop system, developers usually have two options: running the application remotely to the corporate network via a RAS or Internet connection or writing a store-and-forward mechanism into the application. Both methods work fairly effectively, but each has its shortcomings. In the first scenario, phone line connections cannot be guaranteed, and even when a connection is attained, the line throughput can be terribly slow. The major drawback of the second alternative is that implementing a store-and-forward mechanism from scratch requires a substantial amount of coding and testing.
      What if you want high performance and have a short development schedule? MSMQ comes to the rescue. It can easily fix your problems as long as you follow several basic rules. First, an MSMQ-based application is not automatically store and forward-capable. Because of the nature of applications requiring store and forward, it is quite possible that your application will be offline and unable to connect to the network and the Message Queue Information Store (MQIS). In an offline situation, public queues cannot be opened to retrieve or peek at messages, so any code that normally accesses public queues must have adequate error handling. You must also ensure your application avoids any API or ActiveX® calls that directly access the MQIS (see Figure 1). These calls would force an attempt at network activity, causing an error or a timeout.
      Another thing to be aware of is that messages sent in express mode are only held in RAM and will be lost when the computer is turned off. For this reason, I almost always send messages in recoverable mode, which guarantees that they are written to disk and successfully saved in case of a power failure. To send messages in recoverable mode, the delivery property of the message must be set to MQMSG_ DELIVERY_RECOVERABLE.
      When opening a queue offline, you can use its private, public, or direct format name. After opening the queue, all messages sent to it are stored locally by the client computer's queue manager. Upon reconnecting the client computer to the network, these messages will be passed on to the destination queue. To use a public format name, the identifier of the queue must be known by the caller; accessing private queues requires the identifier of the client computer and the queue's name.
      Direct format names will send the message directly to the queue as soon as network connectivity is restored, bypassing the MQIS. This requires that you know the exact network address. Direct format names are extremely helpful when you are trying to access a queue that is not in your enterprise, or when you want to ensure that MSMQ sends the message to the queue in one quick step. In all my sample code, I used direct format names for performance. The downside to this strategy is that if for some reason a queue has to be moved to another machine, the new queue format name must be communicated to all the client applications that want to send messages to the queue.

A Real-World Example
      MSMQ's store-and-forward capabilities have helped me solve one of the biggest problems I routinely encounter with large-scale enterprise development: universal error handling. Debugging and tracking down problems can become quite complicated in today's distributed systems. Narrowing down the origin of an error can be quite time-consuming because of the numerous components of a system distributed throughout a company (multiple client applications, ActiveX business servers, and database servers). Almost every client-server system I have ever worked on has had some way to log error messages—either a file or a database—making systems easier to test, debug, and support.
      In situations where messages are written to individual log files, literally hundreds, if not thousands, of log files can be littered throughout the corporate network, making it very difficult for system administrators to monitor the system for problems as they occur. Instead, administrators are forced to monitor only key logs, waiting for users to call them with problems before they'll investigate other logs. In many mission-critical applications, this solution is totally unacceptable.
      One way around this problem is to write all error messages to a common location like a SQL Server™ database, so administrators only have to monitor one log for the whole system. Unfortunately, this solution is not foolproof; it fails when the database server is down, the user is working in an offline mode, or the network is malfunctioning.
      Using MSMQ, I was able to develop an error handler that managed my most complicated needs by routing all error messages through MSMQ. If the application experiencing the error is connected to the network, the message will be sent directly to the message queue; if the application is offline, the message will be automatically written to the local message queue and later routed to the final destination when connectivity is restored. In both scenarios, an ActiveX server will read the messages from the error queue and write the error information to a SQL Server database that can be queried by system administrators (see Figure 2).

Figure 2: Routing Errors through MSMQ

      I was confronted with the dilemma of determining the best way to package this logic. One possibility was to have all components invoke MSMQ directly and send the messages to the error-handling queue. Another solution was to create an ActiveX-based server that contained the error-handling logic, which could then be invoked by the application experiencing the error. Even though the MSMQ ActiveX wrapper can be easily invoked from anywhere I am currently writing code (in Visual Basic®, stored procedures, ASP scripts, and so on), I decided to write my own ActiveX error-handling object. In the long run, I believe it reduced the amount of code all the developers on my team had to write and ensured one consistent way of communicating with the error queue. Once implemented, this ActiveX server would be deployed to run locally on each of the machines that could possibly access it. The properties and methods for this ActiveX server are outlined in Figure 3.
      In addition to sending error messages to the queue, the error object was also designed to receive messages from administrators. During development and testing, it is quite common to log informational messages concerning the state of an object in addition to error messages. This gives developers of distributed systems an easy way to monitor the flow of the system and catch programming errors. Once the system goes live, administrators usually do not want to be inundated with non-error messages. For this reason, the error object was designed to read the proper level of message logging from an actual MSMQ queue.
      Whenever the error object is instantiated, it attempts to read the current logging level from a public queue. If the error object is unable to access the queue, it assumes that it is working offline and uses the logging level that was last read from the queue. Developers who utilize the error object are forced to specify the error level when they log the messages. The error object filters out the messages based on the messaging level specified by the public queue. In situations where the system is up and running and an unexpected error keeps occurring, administrators can simply send a message to all the components within the system to increase the level of messaging to help them resolve the problem. The code for instantiating the object can be found in Figure 4, and the code for logging error messages can be seen in Figure 5.

Asynchronous Communication
      Another great reason to use MSMQ is to increase the speed of your applications with asynchronous communication. A message sent through MSMQ is always sent asynchronously; in contrast, calling a method of an ActiveX server is asynchronous operation.
      In many situations, you can get better performance by offloading resource-intensive business logic from the client onto more powerful machines. This n-tier, or distributed, approach can also improve performance in circumstances where the client machine is in a remote location and numerous database calls are required by the business logic. In this situation, it is much more efficient to have the business logic executing locally to the database than it is to have numerous database calls executing remotely across a WAN.
      Unfortunately, even the best-designed ActiveX-based servers can at times take large amounts of time to finish processing, which in turn can paralyze client applications. Reasons for this can range from the execution of very complex business logic to the process of communicating with several systems across a slow network. This delay can be extremely frustrating for users who want to kick a process off without locking up their computer waiting for the ActiveX server to finish.
      Until the advent of MSMQ, there were very few solutions to this problem. One of the easiest ways to improve system performance is to buy faster machines and network connections; however, this approach still has its limits. Other solutions include using threads in the client application, coding callback routines into creating a rudimentary messaging application that writes information to disk for later access and processing by a server running in the background. Again, all of these solutions are quite code-intensive.
      By using MSMQ's asynchronous capabilities, developers are able to create fast solutions with very little coding for situations where utmost performance is demanded. As stated earlier, ActiveX servers do not return control to the calling application until the server finishes processing, tying up the client application. With MSMQ, the client application will send all the information that normally goes to the server (through properties and parameters) to a queue instead. As soon as this information is successfully sent to the queue, control is returned to the client application, freeing it to proceed with other tasks.
      Meanwhile, a server process running in the background periodically reads the message queue to see if any new messages have been sent. Once a new message has been received, another ActiveX server is called to help process the information. This is the same server that would have been called directly by the client application prior to the introduction of MSMQ. This server processes in the background, minimizing its effects on the client application's performance. In situations where the client application needs to know the outcome of the server's processing, the server can write the outcome information to another message queue. The client application can then periodically, or upon demand, check this queue for a return message. This solution is similar to a callback server, except that far less coding is required and a consistent method can be employed for multiple ActiveX servers.
      This type of solution works equally well in both online and offline situations as long as the client application follows the store-and-forward rules outlined previously. For example, I am currently working on an application for a manufacturing company. The application collects new order information, performs some proprietary business logic (credit checks, cost calculation, and so on), and then writes the data to a series of database tables.
      There are several flavors of client applications for this system: a laptop client for users who are located in the field disconnected from the corporate network, a Windows 95-based version for users who are processing sales orders over the phone while connected to the corporate network, and finally an Internet browser application that allows users from all over the world to order the company's products online. In all three cases, basic order information is collected and must be saved to the database. The customer will be contacted directly by a customer service representative later if for any reason the order application is turned down or additional information is required.
      The same business logic is required to process the information for all three clients. In each implementation, this information needs to be collected as quickly as possible to avoid tying up the client computer. This is especially important in the online and phone user situations, where delaying the customer for a prolonged period may result in lost business. For these two reasons, it was an easy decision to break the similar business logic into an ActiveX server, which I will refer to as the Orders server.
      MSMQ is needed to handle the needs of the system's laptop users, who are never directly connected to the corporate network while on-site with a customer. Before MSMQ, I'd have had to write some mechanism to take data entered by the user and cache it to disk. I'd then have to write another application that would be instantiated the next time the laptop computer was attached to the corporate network. This second application would be responsible for reading the data that had previously been cached to disk and sending it on to the Orders server. Needless to say, this type of solution would be quite code-intensive, with extensive amounts of time being spent in development and testing.
      With MSMQ, I was able to cache the data to disk and have it automatically routed to the Orders server the next time the laptop was connected to the corporate WAN, usually through a RAS dial-up connection. The code for this type of solution was very easy to write, being very similar to the code outlined previously for the error handler.

Figure 6: Original Order Processing System

Figure 6: Original Order Processing System

      Once the data is successfully delivered to the NewOrders message queue on an MSMQ server, another ActiveX server, FindOrders, processes each order, or message, by instantiating the Orders server and passing it all the necessary information (see Figure 6). The FindOrders server is constantly running, processing new messages from the queue as soon as they appear.
      With this solution, all three client applications could now access the Orders server easily, and the offline users were able to store new order information locally, routing it for processing the next time the computer was online. However, after some prototyping it was determined that the system was still running too slowly for the two online scenarios. During peak business hours, the business logic that was occurring in Orders and its underlying servers was taking up to three minutes to process.
      Upon further investigation, several performance bottlenecks were discovered within the Orders server. During processing, it interacts with several systems (some mainframe, some Windows NT-based) and databases. Even with hardware upgrades and performance tuning, the Orders server would probably never be able to process a new order in under a minute, which was totally unacceptable from the viewpoint of the client.
      In comparison, the laptop portion of the system was operating quite efficiently, owing to the fact that sending information to an MSMQ queue was almost instantaneous. For speed reasons alone, it was an easy decision to send all new orders, regardless of origin, to the NewOrders queue instead of directly accessing the Orders server. Multiple FindOrders and Orders servers were instantiated on several machines to alleviate the additional load caused by routing all orders through the NewOrders queue. My customer was happy because all the orders were being processed quickly, usually within an hour or two, and the customer was only being detained for the time needed to collect all the vital information. This new configuration can be seen in Figure 7.

Figure 7: Streamlined Order Processing System

Figure 7: Streamlined Order Processing System

Packaging Information
      I had several choices when sending order information to the message queue. MSMQ messages can be of any simple data type, such as strings, integers, and dates. The message can also contain persistent ActiveX objects such as Microsoft Excel spreadsheets or Word documents as long as the object supports both the IDispatch and IPersist ActiveX interfaces. The options I considered were sending one message to the queue with all the data fields contained in a string and delimited by a series of characters, sending an individual message for every piece of data, or packaging all the data elements in an ActiveX object.
      After some deliberation, I decided to use ADO recordsets as the mechanism for packaging orders. It was a fairly easy decision since ADO recordsets support both IDispatch and IPersist, which I was already using in the system. This approach is very flexible: I can send one order per message, or I can package multiple orders into one recordset, resulting in only one message being sent to the queue.
      In my sample, the orders recordset is read from a message queue that the client application reads the first time it is started up. For this reason, the laptop systems must be run at least once on the corporate network before they are able to start processing orders offline. Thanks to ADO, this recordset can be easily cached to a file on disk. Later, quite possibly even after rebooting the machine, this recordset's schema and data can be reloaded into a recordset. The code for caching this recordset to disk is as simple as:

rsOrder.Save App.Path & "\order.rs"

Later on, this recordset can be recreated in memory through the following line of code:

rsOrder.Open App.Path & "\order.rs"

      Whenever a new order is placed, it is added as a new record in the recordset. Depending how the system is configured, more than one order can be added to a recordset before sending it as a message to the queue. The code for adding a new order to the recordset is shown in Figure 8; the code for sending the recordset to the appropriate queue can be seen in Figure 9.
      It's worth taking a look at the logic for the FindOrders ActiveX server. FindOrders processes all client orders by first reading the ADO recordset from the New Orders queue. It then passes this recordset as a parameter to the Orders server. The Orders server performs some fairly complex business logic (this is the code that can take several minutes to run). After the business logic finishes processing, the new order information is inserted into a SQL Server database table.

Transactions
      FindOrders can be viewed as the lifeblood of my client's company since it processes all orders from the three systems. One hundred percent reliability is required since misplaced or double orders result in lost business. To meet this standard, I use transactions to ensure that every order is processed only once. Retrieving the message from the queue and writing the order to the database are contained in one transaction. By placing this logic within a transaction, the customer is guaranteed that a new record is inserted in the database for every order retrieved from the queue. In situations where the message is successfully retrieved from the queue, but for some reason the database insert fails, the message remains in the queue.
      Developers have several options when implementing transactional processing with MSMQ. The three most common transaction processing options are MSMQ internal transactions, Microsoft Distributed Transaction Coordinator (DTC) external transactions, and Microsoft Transaction Server (MTS) transactions. MSMQ internal transactions provide the best performance for scenarios that only involve the sending and receiving of MSMQ messages. In fact, MSMQ automatically utilizes an internal transaction whenever a single message send operation is performed. Unfortunately, MSMQ internal transactions cannot be passed to another resource manager.
      Internal transactions are as easy to program as:

Dim xdisp as New MSMQTransactionDispenser Dim xact as MSMQTransaction Dim msg as New MSMQMessage Dim qSend as MSMQQueue ... Set xact = xdisp.BeginTransaction ... msg.Send qSend, xact xact.Commit

Basically the developer must instantiate new MSMQTransactionDispenser and MSMQTransaction objects. Next, the BeginTransaction method of the MSMQTransactionDispenser object must be called. This method returns a new MSMQTransaction object, which can be used whenever any messages are sent or retrieved. Finally, the transaction can either be committed or aborted by calling the Commit and Abort methods of the MSMQTransaction- object.
      In transactions with more than one resource manager, developers will normally use external transactions or MTS transactions, both of which use DTC. DTC external transactions are coded very much like internal transactions:

Dim xdisp as New MSMQCoordinatedTransactionDispenser Dim xact as MSMQTransaction Dim msg as New MSMQMessage Dim qSend as MSMQQueue ... Set xact = xdisp.BeginTransaction ... msg.Send qSend, xact xact.Commit

The major difference is that external transactions are necessary when a transaction includes actions beyond simply sending or retrieving MSMQ messages. In these situations, the application must ask DTC for a transaction object and explicitly reference that object each time it sends a message, retrieves a message, or executes an action upon another resource manager. The only difference in the code is that the MSMQ Transaction Dispenser object is replaced with the MSMQ Coordinated Transaction Dispenser object.
      Of all the transactional processing methods I've used, MTS is my favorite because, in my opinion, it is the easiest to code. I like to follow a distributed approach, with most of my business logic residing in ActiveX servers. By running within the MTS environment, MSMQ uses the current MTS transaction if one is available. In this situation, MTS uses the services of DTC for transaction coordination. The developer writes the MTS server code in the exact same way they would when dealing with database transactions. The code for the FindOrders server can be seen in Figure 10.

Conclusion
      MSMQ has helped me solve several problems that are quite common in client-server and Web development. It provides functionality that was previously unavailable and helps boost system performance in certain situations. Note that I said "certain situations"; MSMQ is by no means the answer for every situation. Sometimes applications require a response from an ActiveX server as quickly as possible. Even though MSMQ could easily send a return message to the client, the most efficient method would be to call the ActiveX server directly and wait for the response. Good design—not necessarily the latest tools and technologies—makes systems efficient.
      Although my examples have been simplified for illustrative purposes, they should still show what's necessary to implement MSMQ in the real world. As you are designing tomorrow's client-server and Internet systems, I urge you to seriously consider MSMQ. It can dramatically reduce the amount of code your development team has to produce, while adding new functionality and possibly increasing performance.

From the August 1998 issue of Microsoft Interactive Developer.