July 1998

Microsoft Message Queue Is a Fast, Efficient Choice for Your Distributed Application

David Chappell

How do you get the various parts of your distributed app to communicate with each other? One option is message queuing middleware like the Microsoft Message Queue (MSMQ), which lets applications communicate reliably on unreliable networks.

This article assumes you’re familiar with COM

David Chappell (david@chappellassoc.com) is principal of Chappell and Associates, an education and consulting firm in Minneapolis, MN. He is the author of Understanding ActiveX and OLE, published by Microsoft Press.

The fundamental problem in building a distributed application is providing some way for the various parts of the app to talk to each other. One ever-more-popular approach to this is to rely on a standard Web browser client such as Microsoft® Internet Explorer, and use HTTP to move information between it and a server. Another option, one that's much older, is to use a remote procedure call (RPC) protocol such as Microsoft RPC or its object-oriented doppelganger Distributed COM (DCOM). But developers in the Microsoft environment now have another option as well. Today, distributed applications can be built that communicate via messages sent using Microsoft Message Queue Server (MSMQ). MSMQ is now included with Windows NT® Server 4.0. You can also download it as part of the Windows NT 4.0 Option Pack. The URL for the MSMQ Web site is http://www.microsoft.com/msmq.

Figure 1  Message Queuing

The idea of message queuing is tremendously simple. As shown in Figure 1, an application builds a message and sends it to a queue. Some other part of the application (or some other application) can then read the message from the queue. If needed, this reader can respond, again by sending a message into some queue that's later read by the original sender or even by somebody else. The process is straightforward and easy to understand. Of course, in real apps you'd make lots of features available, so products that support message queuing commonly offer much more than just simple sending and receiving. MSMQ is no exception, providing a broad range of features for message-oriented applications. When to Use MSMQ       So when should you use message queuing rather than a Web-based app or RPC? It's usually not too hard to figure out if message queuing is the right choice for your distributed application. If you want to use a standard client and the information you're communicating can be reasonably exchanged via HTTP, build a Web app—it's the simplest choice. If you're willing to write a custom client, something that's often required, the choice is between RPC and message queuing. A jihad has raged between these two camps for several years, but by now it seems that both sides have accepted the need for the other. Here are some simple rules to help you decide when to use each one.       If the sender must wait for a response from the receiver before proceeding, you may as well use RPC (since a call blocks until a result is returned). RPC is typically simpler to use than message queuing, so if it fits the bill, use it. If the sender need not wait for a result, but can usefully do something else in the meantime, message queuing is probably a better choice.       If the sending and receiving applications may run at different times, use message queuing—RPC assumes that both client and server are available at the same time.       If a sender isn't sending to a specific receiver, but instead to any one of a group of receivers, use message queuing. Once a message has been placed in a queue, it is potentially available to any application that can read from that queue. With RPC, by contrast, a client typically makes a call to a specific server.       If requests need to be logged and possibly reprocessed to recover from failures, use message queuing. While it's certainly possible to build logging into an RPC-based app, mechanisms to do this are an intrinsic part of MSMQ.       If your clients just call some server, get a response from that server, and go on, RPC is fine. But if you need more complex interactions between the parts of a distributed application, use message passing. With MSMQ, A can send a message to B, who sends one to C and D, who both send response messages back to A. This kind of flexibility isn't possible with RPC.       It's tempting to think of performance as a reason to choose one over the other. RPC somehow seems like it should be significantly faster than message queuing since requests and responses don't have to go through a queue. In reality, though, the performance difference between MSMQ and RPC is insignificant. Both are really fast, so don't fall into the trap of viewing messaging as inherently slow.       One last point: don't confuse message queuing with email. The two can seem very similar (and to add to the confusion, both are often called just "messaging"), but they have quite different purposes. The simplest way to think about it is to view email protocols as a way to send messages between people, while products like MSMQ are designed to send messages between applications. There's no real technical reason why, say, some future version of Microsoft Exchange couldn't be built on MSMQ, but standard protocols for email, such as SMTP, are already in place. As a result, the historically rooted distinction between technologies for person-to-person messaging and those for application-to-application messaging shows no signs of going away.

Figure 2  Queue Managers Communicating

  MSMQ: The Big Picture

One of the nice things about message queuing is that, conceptually at least, it's so straightforward. As shown in Figure 2, there are three major components of MSMQ. An API that applications can use to send and receive messages. MSMQ actually provides two different APIs, as I'll describe later, and more are on the way. Messages that are created, then sent and received by applications. Queues, managed by a queue manager, from which those messages are sent and into which they are received. Queue managers may communicate with one another to send messages from one queue to another.       From these basic components, MSMQ defines three different kinds of applications (see Figure 3). The most complete is an MSMQ server, which contains queues and a queue manager, support for the MSMQ API (and thus for applications that send and receive messages), software to route messages between queues, and more. At least one MSMQ server must be available in every MSMQ installation.

Figure 3  Types of MSMQ Applications

MSMQ also defines two types of clients. An independent client supports the MSMQ API and also has a queue manager with its own queues. Applications running on an independent client are free to send messages at will since they can be stored in the client's queues. If the client is currently connected to a network, those messages can be forwarded to an MSMQ server immediately, where they are available to other applications. If the client isn't connected to a network (maybe the client is running on your laptop and you're trying to get some work done while crammed into a coach seat on a long flight), messages sent by the application are stored in the client's queues. When an independent client reconnects to a network, its queue manager will automatically detect this and forward those messages onto an MSMQ server.       MSMQ also allows dependent clients. Like independent clients, dependent clients can only support applications that send and receive messages—they can't act as message routers. But dependent clients are even more limited than independent clients. Since they support only the MSMQ API, an application running on a dependent client must have online access to an MSMQ server. In fact, MSMQ servers implement a proxy function solely to support these needy systems. Dependent clients are intended for systems with permanent connections to a network, such as a desktop machine connected to a LAN. And because they have no queues of their own, managing an MSMQ environment full of dependent clients can be simpler than managing one full of independent clients. MSMQ APIs       MSMQ provides two different APIs. For developers working in C++, there's an API defined as a set of C function calls. For those working in Visual Basic®, Java, or C++, there's also an API defined as a set of COM objects. It's possible to build, send, and receive messages with either one, but the C API offers access to a few more services than the COM API.       The MSMQ C API is not especially simple, but understanding the basics is not too hard. Here are some of the functions in this API. MQCreateQueue creates a new queue in either an MSMQ server or an independent client. MQDeleteQueue destroys a queue. MQOpenQueue opens an existing queue. A specific queue can be located using a SQL Server™-based directory service (which will be replaced by Active Directory once Windows NT 5.0 is available). MQCloseQueue closes a queue. MQSendMessage sends a message to a specified queue. Each message has a large number of properties that can be set by the sender, the most important of which are described in the next section. MQReceiveMessage receives a message from a specified queue. Messages can be read either synchronously, with the receiving application blocking until the message arrives, or asynchronously, with the application receiving a notification that a message has arrived via a callback function or in some other way. An application can retrieve a message, which removes it from its queue, or peek at it, which lets the application examine the message's properties without removing it from the queue. An application can also choose which properties of a message it wishes to receive—it's not obligated to read the entire message. And finally, by creating and using a cursor, applications can examine and receive messages that aren't currently at the head of the queue.       To invoke a function in the C API, you must first populate some number of structures with appropriate values, then pass those structures as parameters into the desired function. This is a classic approach to building a messaging API, so it's not surprising that MSMQ offers this option. But MSMQ also provides a more modern style of interface using COM objects. You can access the MSMQ services by setting properties and invoking methods on these objects.       To create a queue, for instance, you'll use an MSMQ-Queue-Info object. The process is simple: just assign the queue's name to the MSMQQueueInfo object's PathName property, then invoke this object's Create method. To open this newly created queue, the app invokes MSMQ-QueueInfo::Open. This call returns an MSMQQueue object, which you can use to reference the open queue. To get an MSMQQueue object that refers to an existing queue instead of creating a new one, an application can use the LookupQueue method of the MSMQQuery object. This method has parameters that allow the application to search for queues that meet specific criteria.       To build and send a message, you create an MSMQ-Message object, set its properties, then invoke its Send method. Which queue the message gets sent to is controlled with an MSMQQueue object passed as a parameter on this call. To receive a message, an application invokes the appropriate method on the MSMQQueue object representing the queue from which the message should be read. As described earlier, there are many options for how a message can be received—applications aren't forced to blindly wait for the next message in the queue.       Knowing a little about the MSMQ APIs is useful, but what's really interesting is what those APIs let applications do. Perhaps the clearest way to see the capabilities of this technology is to look at some of the properties that can be set on each message and each queue. MSMQ Message Properties       The most important property in a message is its Body. The Body contains the data being sent in the message, and it can be up to 4MB in size. The data in the message body can be just an untyped string of bytes, or it can carry an indication of the types of data being sent.       The Delivery property of a message determines how that message will be handled and stored by queues. The two possible values for this property are Express and Recoverable. If a message is marked for Express delivery, all queues that handle this message will store it in memory only—the message won't be written to disk. Messages marked with Express delivery will get to their destination queue very quickly, but they pay a price for this speed. If an Express delivery message is currently resident in some queue and the machine that holds that queue crashes, the message will be lost.       Messages marked for Recoverable delivery, by contrast, are always written to disk by the queues that handle them. If a machine holding a queue crashes, the Recoverable messages in that queue will not be lost since they've been written to nonvolatile storage. MSMQ offers a choice, and you must weigh the trade-off between more reliable delivery and faster communication.       Some messages are time-critical. If a message has not been received within a certain number of seconds, for example, it may be appropriate to throw that message away. MSMQ defines a pair of message properties for use in this situation. The value in a message's Time To Be Received property specifies the total number of seconds this message will exist after it has been sent. If the message is not received by some application within this time period, MSMQ will discard it. Similarly, a message's Time To Reach Queue property specifies the total number of seconds a message has to reach its destination queue. If it doesn't make it in time—for example, if the only path to that queue is down—MSMQ will discard it.       All this talk of discarding messages raises another important question: how does a sender know what happens to its messages? Which ones are received correctly? Which ones are discarded? How these questions are answered depends on the value of the Acknowledge property for a particular message. This property has five possible values.       Full Reach Queue causes MSMQ to automatically send an acknowledgment message indicating that the message has reached its destination queue or that it will never reach it (perhaps because the value in the message's Time To Reach Queue property has expired).       Full Receive causes MSMQ to send an acknowledgment message indicating that the message has been received or that it will never be received, perhaps because the message's Time To Be Received property has expired.       Nack Reach Queue causes MSMQ to send an acknowledgment message indicating that a message can't reach its destination queue. Again, this could be because the message's Time To Reach Queue timer has expired or for some other reason. Only a negative acknowledgment is sent, so to the sender no news is good news. If no acknowledgment message comes back, the sender can assume the message has reached its destination queue.       Nack Receive causes MSMQ to send an acknowledgment message indicating that a message can't be received. This can occur, for example, when its Time To Be Received timer has expired. Again, only a negative acknowledgment is sent.       The last possible value is None. MSMQ sends no acknowledgment messages if the sender does not explicitly set the Acknowledge property. None is the default.       One of the nice things about message queuing is that it's easy to keep a record of what messages have been sent. If your application is for, say, a stock exchange with a legal requirement for effective auditing procedures, this can come in very handy. An application can set each message's Journal property to determine what, if any, records are kept at the sending system about this message. If this property's value is set to Journal, a copy of this message will be placed in a special queue on the sender called, not too surprisingly, the Journal queue. This queue can be examined by other applications interested in knowing what messages have been sent.       And what happens to messages that, for whatever reason, can't be received? Messages whose Journal property is set to Dead Letter wind up in the Dead Letter queue, another special queue created by MSMQ. Both properties can be set on the same message, indicating that it should be logged when sent and forwarded to the Dead Letter queue if it can't be received. Finally, if no value is explicitly set for a message's Journal property, the default is to do nothing—the message will not be recorded in the Journal queue when it's sent, and it won't be forwarded to the Dead Letter queue if it can't be received.       In a typical application, some messages are more important than others. For example, messages that submit trades in a brokerage application will have priority over those that just inquire about stock prices, and higher dollar trades will have higher priority than the lower dollar trades. MSMQ recognizes this fact, and so each message has a Priority property that applications can set. Defined as an integer value between 0 and 7, messages with lower numbers have higher priority. This priority is taken into account when MSMQ makes routing decisions and when messages are inserted into queues. Messages with higher priorities (those with a lower value for their Priority property) are inserted toward the front of the queue rather than strictly in order of their arrival.       When an application sends a message to a queue, it may well expect some kind of response. With RPC, it's obvious where to respond—the RPC runtime takes care of this little detail. With message queuing, however, it may not be obvious how to get a response back to the sender of a message. After all, one of the benefits of messaging is the ability to send a message without knowing precisely who's going to receive it.       Short of hardwiring it into the application, how can a sender let the receiver of a message know how to respond? The answer is to use another property of an MSMQ message. By placing the name of a queue in a message's Response Queue property, the sender can inform the receiver where it would like a response to this message sent. Setting this property isn't required since some applications choose to use only a predefined set of queues. But it does provide a convenient and standard way to let a message's receiver know where a response should be sent. Note that setting this property in no way obligates the receiver to send a response. Whether a response must be sent is part of the application's semantics, not those of MSMQ.       A related question is: how can a sender associate responses with the message that engendered that response? Suppose, for example, that an application sends 10 messages and gets a response message for each one. There's no guarantee that the 10 responses will arrive in the same order as the requests were sent.       To allow a sender to figure out which response message goes with a particular request message, MSMQ automatically generates a unique Message ID for each message that's sent. Both the sender and receiver of a message can read this property. The receiver can copy this Message ID into the Correlation ID property of any message it sends in response. By remembering the Message ID of the message it originally sent and matching it with the Correlation ID of a received message, the sender can figure out which responses match which sent messages. In fact, MSMQ does this when it sends an acknowledgment (when required by a message's Acknowledge property).       MSMQ defines many more properties for messages. Messages can be encrypted, digitally signed, marked with an application-specific label, and more. While the basic idea of message queuing is simple, the kinds of things applications want to do with messages are often not so simple. Accordingly, MSMQ provides a powerful set of services for those applications that need it. Message queuing is inherently more complex for developers than RPC, but there are plenty of cases where the extra effort is worthwhile. MSMQ Queue Properties       Queues have properties too, which can be set by applications and people with the right permissions. The most important property of a queue, and the only one that's absolutely required when creating a queue, is its Pathname. A queue's Pathname is just a character string (such as "machineX/myqueue") identifying the machine that queue is on and giving a name for the queue. Once this property has been set for a queue, it can't be changed. There are several other important properties for queues. I'll discuss these next.       The Quota property specifies the maximum size in bytes that the queue can hold. If the total size of all messages in the queue reaches this limit, further attempts to send messages to this queue will fail. Depending on the value of the rejected message's Acknowledge property, this may also generate an acknowledgment message.       The Journal property controls whether messages removed from the queue will be copied to a Journal queue. Don't confuse the recording of messages removed from a queue with the journal option described earlier. Message-based journaling keeps a record on the sending machine of messages sent by applications on that machine, and it's controllable on a per-message basis. Queue-based journaling, on the other hand, records all messages removed from a queue, and is turned on or off for the queue as a whole.       The Base Priority is used to make decisions when routing among queues. Messages sent to a higher-priority queue will be routed more expeditiously than those sent to queues with lower priorities.       Type is a GUID that specifies a type of service. When an application uses the MSMQ queue location services, it can specify that it only wants to find queues with a certain value for their Type property. This can be a very useful way to locate one of a set of queues that all provide the same kind of service, such as a print queue.       Queues have other interesting properties, too. One queue property in particular is so important that it deserves its own section: the Transaction property. MSMQ and Transactions       A transaction is just a group of two or more events that either succeed or fail as a unit. The most common example of events that are grouped into a transaction is changes to one or more databases. If the changes are all made to a single database management system such as SQL Server, the DBMS itself can ensure that the entire transaction either succeeds or fails. If the changes are made to two or more databases, however, some kind of external transaction coordinator is typically used to ensure that all the databases behave correctly. The Distributed Transaction Coordinator (DTC), a standard part of Windows NT, provides this kind of service.       It's possible to write transaction-oriented applications that use DTC directly, but it's not easy. The COM-based Microsoft Transaction Server (MTS) makes life more pleasant for application developers. MTS relies on DTC, but it provides a much simpler programming interface. MTS also provides other services that are nice to have when you're trying to write scalable server applications.       What does all of this have to do with MSMQ? Although the most common use of transactions today is with -databases, transaction processing people use the generic term "resource manager" to refer to anything that manages changes that are part of a transaction. And database management systems aren't the only things that can act as resource managers. MSMQ, too, can be a resource manager.       To get a sense of what this means and why it's so useful, imagine a Web-based application that accepts orders from customers on the Internet. Suppose the part of the application that interacts with the client is written using Internet Information Server (IIS) and Active Server Pages, as shown in Figure 4. In this application, each order requires making changes to a database containing inventory information about the goods being ordered. Once a customer submits an order, that order is sent off to another machine via MSMQ to be filled. It's entirely reasonable to demand that either both of these things happen—the database is changed and the order is sent off for fulfillment—or neither one happens. While it would be possible for the developer to write code to guarantee this, it's much easier to offload the burden. Why not group both events into a transaction, then let MTS do the hard work?

Figure 4 Using MSMQ with MTS

To allow a queue to participate in a transaction, its Transaction property must be set when the queue is created. Once this is done, the transactional queue will be capable of acting as a resource manager under the control of the DTC (and indirectly, of an MTS application).       In the example shown in Figure 4, the MTS application is invoked from the ASP pages to build and submit the order. In the process of doing this, that application makes changes to the database and sends a message to a transactional queue. If all goes well, the MTS application commits the transaction, causing two things to occur: the database management system makes the changes to the database permanent, and MSMQ actually sends the message.       Because the queue was marked as transactional, any message sent to it is held until the transaction that message is part of completes. If the transaction is committed, the message is actually sent. If the transaction is rolled back, however, the message is removed from the queue-—it never gets sent. Both changing the database and sending the fulfillment message either happen or don't happen, which is exactly as it should be for this application. Although it's not shown here, the act of removing a message can also be part of a transaction—if the transaction commits, the message is permanently removed from the queue. If it aborts, however, the message is returned to the queue, waiting to be received again.       MSMQ certainly doesn't require you to use transactions when sending or receiving messages. You may find, however, that many, even most, of your applications are sending messages to transactional queues. Here's a big reason why: for nontransactional queues, MSMQ does not guarantee "exactly-once" semantics for messages. In other words, a message sent to a nontransactional queue may be delivered more than once. If the message requests, say, a deduction from your bank account, this is not an appealing prospect.       For transactional queues, however, MSMQ does guarantee exactly-once semantics. One transmission is guaranteed to result in one message appearing in the destination queue. Also, all messages sent to transactional queues have their Delivery attribute set to Recoverable automatically, ensuring the message won't be lost if a server fails along the way. If your application's messages must be sent only once—a common requirement—use transactional queues. Wrap-up       MSMQ is a standard part of Windows NT Server today, just like MTS and IIS. MSMQ clients are available from a company called Level 8 for a range of other operating systems, including various flavors of Unix, OpenVMS, MVS, and the AS/400. The codename for MSMQ during development was Falcon, so Level 8 calls its MSMQ product line Falcon-MQ.       In the RPC versus messaging wars, I spent several years as a diehard RPC bigot. I was wrong. Message queuing is a really good idea, and there are plenty of problems for which it's just the right solution. While it's hard to argue that MSMQ is especially innovative, Microsoft was able to do a whiteboard design of the product that allowed them to build in the best features of existing products. The result is a state-of-the-art message queuing service that is sure to be used in lots of distributed applications. From the July 1998 issue of Microsoft Systems Journal.