Microsoft Corporation
February 23, 1999
Summary: This article presents several useful tips and strategies for dealing with various aspects of MSMQ application development. The following issues are addressed:
A: You may use the MSMQ COM objects from C++ like any other COM objects. You would access them with standard COM APIs, like CoCreateInstance and the like—there is a sample in the MSMQ SDK subdirectory (by default in %SystemRoot%\Program Files\msmq\sdk\samples) that indicates how to do this: see the mqtestoa subdirectory.However, it is much easier to use the COM components in Microsoft® Visual C++® version 5.0 with the #import directive. This option allows you to consume a DLL that contains a type library and creates a set of "smart pointer" wrappers that hide all of the gory COM reference counting and memory management details for COM objects. In fact, you'll see that your MSMQ programs in Visual C++ 5.0 become virtually identical to your Microsoft Visual Basic® programs in terms of number of lines written. Likewise, there is good support for handling BSTRs and variants.
Here's a code fragment that indicates how you might open a queue and send a message to it in Visual C++:
#include <windows.h>
#include <stdio.h>
#import "mqoa.dll" no_namespace
void main()
{
//
// create queue
// open queue
// send message
//
OleInitialize(NULL); // have to init OLE
//
// declare some variables
//
IMSMQQueueInfoPtr qinfo("MSMQ.MSMQQueueInfo");
IMSMQQueuePtr qSend;
IMSMQMessagePtr m("MSMQ.MSMQMessage");}
qinfo->PathName = ".\\q99";
try {
qinfo->Create();
}
catch (_com_error comerr) {
// assume that only error generated is that queue exists already,
// ignore and fall through...
};
qSend = qinfo->Open(MQ_SEND_ACCESS, MQ_DENY_NONE);
m->Body = "body";
m->Send(qSend);
qSend->Close();
}
For the sake of comparison, the equivalent Visual Basic code would be:
Sub Main
Dim qinfo As New MSMQQueueInfo
Dim qSend As MSMQQueue
Dim m As New MSMQMessage
qinfo.PathName = ".\q99"
On Error Resume Next
qinfo.Create
On Error GoTo 0
Set qSend -= qinfo.Open(MQ_SEND_ACCESS, MQ_DENY_NONE)
m.Body = "body"
m.Send qSend
qSend.Close
End Sub
Click here to download the sample associated with this "MSMQ and Completion Ports" section.
A: The most efficient way in MSMQ to asynchronously receive messages is with the completion port mechanism. This mechanism is scalable in the number of queues and messages by adding more processors/threads. Likewise, generic completion port handlers can be provided to handle other Microsoft Windows NT® resources as well as queues.
Using completion ports is somewhat more complicated than the other two asynchronous techniques supported by MSMQ: namely, callback functions and standard Windows events. However, it is not all that hard and, in fact, both the MSMQ COM components and the MQ API can be used together to construct a fairly straightforward and extensible solution. In the sample that is presented here the COM components are used for queue creation, open, and message send, whereas the MQ API is used to implement the actual completion port-based asynchronous receive.
The following steps explain the sample and demonstrate how completion port handling might work in general in MSMQ:
Note that the Windows NT scheduler will select the "best" available completion port thread that is synchronously waiting for a completion notification.
A: Some fundamental aspects of MSMQ queues help in understanding how MSMQ cursors behave. The most basic point is that MSMQ queues are priority queues. A queue is ordered and has a well-defined "head": namely, the queue location of the current first message. This simply means that higher priority messages appear before lower priority messages. In addition, within each consecutive subset of messages of the same priority, messages are sorted chronologically (by arrival time).
Cursors are used by MSMQ applications to iterate in a controlled manner through an MSMQ queue. In general, an application can create any number of cursors that reference the same queue. Of course, these cursors can interfere with one another to an extent. For instance, let's consider what happens when you open a queue and subsequently create two cursors, C1 and C2, that reference that queue. Initially both cursors reference the first message. You now remove the message referenced by C1 and then attempt to obtain the message referenced by C2. This will result as expected in an error since C2 now references what amounts to an empty location.
Additionally, before a cursor can be advanced to reference the next location it must be initialized by peeking at the current location. Otherwise, a self-explanatory error will be produced.
The MSMQ COM object model supports a limited subset of MSMQ cursor functionality that is nonetheless very useful. The rest of this discussion will focus on the COM objects.
In the COM model, unlike the MSMQ C API, an explicit cursor object is not exposed. However, when a queue is opened (an MSMQQueue instance is obtained that references an open queue) a single cursor is implicitly created. This cursor is not exposed to the programmer, but it is used implicitly by the MSMQQueue.PeekCurrent, MSMQQueue.PeekNext, and MSMQQueue.ReceiveCurrent methods.
While there is only a single implied cursor per MSMQQueue instance, you can of course create multiple MSMQQueue instances that reference the same underlying MSMQQueue. In this way, you now have the functionality of multiple cursors per queue.
A typical loop for enumerating all the messages in a queue would look something like this in Visual Basic:
Dim m As MSMQMessage
Dim q As MSMQQueue
Set m = q.PeekCurrent
Do Until m Is Nothing
Set m = q.PeekNext
Loop
Pretty straightforward.
Things get a bit more complicated when cursors are used in conjunction with asynchronous messaging. Say you'd like to implement an asynchronous version of the preceding message enumeration. Let's take a look at a code fragment in Visual Basic and analyze its various elements.
This sample simply asynchronously peeks at or receives the next message in the queue. Use AsyncPeek to asynchronously peek at each message as it arrives, and AsyncReceive to asynchronously receive.
Dim qinfo As MSMQQueueInfo
Dim qRec as MSMQQueue
Dim WithEvents qeventPeek As MSMQEvent
Dim WithEvents qeventReceive As MSMQEvent
Sub AsyncReceive
Set qRec = qinfo.Open(MQ_PEEK_ACCESS, MQ_DENY_NONE)
Set qeventReceive = New MSMQEvent
qRec.EnableNotification qevent, Cursor:=MQMSG_CURRENT
End Sub
Sub AsyncPeek
Set qRec = qinfo.Open(MQ_PEEK_ACCESS, MQ_DENY_NONE)
Set qeventPeek = New MSMQEvent
qRec.EnableNotification qevent, Cursor:=MQMSG_CURRENT
End Sub
Private Sub qeventPeek_Arrived(ByVal Queue As Object, ByVal Cursor As Long)
Dim m As MSMQMessage
Set m = Queue.PeekCurrent
MsgBox m.Label
qRec.EnableNotification qevent, Cursor:=MQMSG_NEXT
End Sub
Private Sub qeventReceive_Arrived(ByVal Queue As Object, ByVal Cursor As Long)
Dim m As MSMQMessage
Set m = Queue.ReceiveCurrent
MsgBox m.Label
qRec.EnableNotification qevent, Cursor:=MQMSG_CURRENT
End Sub
You will note that there are three cursor-related elements in the canonic structure of this sample:
So we see that there are 2 * 3 * 2 = 12 different combinations—not all of which necessarily make sense. Try out some of the other variants and see what effect they have on your enumeration.
Here are a couple of samples that show something important about how cursors work with priority queues—there's nothing Visual Basic-centric per se about them. The same samples could be written in C.
The main point that is demonstrated in these samples is that a cursor will be repositioned by MSMQ to reference a newly arrived message if that message arrives while your application is actually waiting for a message.
Dim qinfo As New MSMQQueueInfo
Dim qSend As MSMQQueue
Dim qReceive As MSMQQueue
Sub TestCursor()
qinfo.PathName = ".\q" & Now
qinfo.Create
Set qSend = qinfo.Open(MQ_SEND_ACCESS, 0)
Dim m As New MSMQMessage
For i = 1 To 10
m.Body = i
m.Priority = Int(Rnd * 3) + 1
m.Send qSend
Next
'
' peek to end of queue
'
Dim m2 As MSMQMessage
Set qReceive = qinfo.Open(MQ_RECEIVE_ACCESS, 0)
Set m2 = qReceive.PeekCurrent(ReceiveTimeout:=10)
While Not m2 Is Nothing
Set m2 = qReceive.PeekNext(ReceiveTimeout:=10)
Wend
'
' send a hipri msg
'
m.Body = "hi pri"
m.Priority = 4
m.Send qSend ' will be appended to front of queue
m.Body = "lo pri" ' send a lopri msg
m.Priority = 0
m.Send qSend ' will be appended to tail of queue
'
' returns the lo-pri message: need PeekCurrent,
' not PeekNext since previous PeekNext failed.
'
Set m2 = qReceive.PeekCurrent(ReceiveTimeout:=10)
msgbox m2.body
End Sub
Note this sample returns the lo-pri message at the tail of the queue because the cursor is still positioned at the end of queue when the Peek call is made.
Contrast this with the next example:
Dim qinfo As New MSMQQueueInfo
Dim qSend As MSMQQueue
Dim qReceive As MSMQQueue
Dim WithEvents qevent As MSMQEvent
Private Sub TestCursor2()
qinfo.PathName = ".\q" & Now
qinfo.Create
Set qSend = qinfo.Open(MQ_SEND_ACCESS, 0)
Dim m As New MSMQMessage
For i = 1 To 10
m.Body = i
m.Priority = Int(Rnd * 3) + 1
m.Send qSend
Next
'
' peek to end of queue
'
Dim m2 As MSMQMessage
Set qReceive = qinfo.Open(MQ_RECEIVE_ACCESS, 0)
Set m2 = qReceive.PeekCurrent(ReceiveTimeout:=10)
While Not m2 Is Nothing
Set m2 = qReceive.PeekNext(ReceiveTimeout:=10)
Wend
'
' setup async handler
'
Set qevent = New MSMQEvent
qReceive.EnableNotification qevent, MQMSG_CURRENT, 10000
'
' send a hipri msg
'
m.Body = "hi pri"
m.Priority = 4
m.Send qSend ' will be appended to front of queue
'
' send a lopri msg
'
m.Body = "lo pri"
m.Priority = 0
m.Send qSend ' will be appended to tail of queue
End Sub
Private Sub qevent_Arrived(ByVal Queue As Object, ByVal Cursor As Long)
Dim qReceive As MSMQQueue
'
' get next msg
'
Set qReceive = Queue
Dim m2 As MSMQMessage
Set m2 = qReceive.PeekCurrent(ReceiveTimeout:=10) ' returns hi-pri msg
msgbox m2.Body
End Sub
In this case, the high-priority message from the head of the queue is returned. Why? Because it arrived at the head of the queue while the cursor was positioned at the queue tail still waiting for a message to show up there. So, effectively the cursor was repositioned to the newly arrived message.
This is a nice feature—it basically allows you to write cursor-based asynchronous message handlers that never miss newly arrived messages.
A: First, an observation: managing transactional messaging is not trivial. On the other hand, the benefits of transaction-based messaging are significant so it's worth your while to invest the effort to understand the issues and the relevant solutions.
So, MSMQ provides support for sending messages in a transaction. What does this mean? That these messages will either be sent together, in the order they were sent, or not at all. In addition, consecutive transactions initiated from the same machine to the same queue will arrive in the order they were committed relative to each other. Moreover, each message, if it arrives, will arrive exactly once and MSMQ guarantees that it will arrive or that you will be notified of its disposition. This last point is the crux of the rest of this discussion. How do you recover from catastrophic failures?
The MSMQ APIs relevant to transactions are MQSendMessage and MQReceiveMessage (or the COM methods MSMQMessage.Send and MSMQQueue.Receive). When a message is sent in a transaction to a queue and the transaction is subsequently committed, this simply means that the local queue manager has accepted the message for future sending. MSMQ guarantees that the message will be sent but, at this point, the message has not even left the sending machine. It certainly has not yet been received, nor reached its intended destination.
There are many system or application-specific reasons that your message might not be delivered. System failures can be due to security problems or queue quota limitations: these can be addressed to some extent by the MSMQ infrastructure. Other failures can be logical problems that cause the receiving application not to be able to successfully process the message correctly.
In MSMQ, a logical transaction consists of two separate physical transactions: sending and receiving transactions. A single distributed transaction is not sensible in the possibly disconnected dispersed enterprise. It is not desirable to create a single transaction that encompasses both the sender and receiver since this can lead to long delays in application availability. Why? Because a transaction must complete before its resources are freed up for other clients to access.
Thus, typically, in the distributed world, these sending and receiving transactions are performed by two separate processes running over the network. Their host machines might very well be disconnected (which is the whole point of using messaging) and quite often, the sending and receiving applications' lifetimes won't overlap.
MSMQ provides mechanisms for applications to implement reliable messaging: namely, a per-machine transactional dead letter queue, an application-specific administrative queue and finally an application-specific response queue. The dead letter queue is used by MSMQ to record messages that were not processed correctly. MSMQ uses the admin queue for explicit positive and negative notifications as to the outcome of specific messages. Finally, the response queue can be used by sending/receiving applications to establish a private protocol. It turns out that all three of these constructs are needed for a robust system.
The fundamental problem that needs to be solved satisfactorily is how the sending application can categorically know that its messages were consumed correctly or otherwise at the destination.
The following guidelines should be used when writing a transactional MSMQ program:
MSMQ guarantees that if a sent message times out (either TTBR or TTRQ), it will be moved to the sending machine's transactional dead letter queue (XDLQ). So, in principle, by monitoring that queue you can retrieve problematic messages and decide programmatically or manually how to compensate for/recover from the failure. Messages in the XDLQ will contain an indication of why they are there, in the message class property—for example, "time to be received expired." It is crucial to understand at this point that the sending transaction itself succeeded but something further downstream failed: for example, a network failure or a logical problem in the receiving application. What could be simpler?
Well, there are a couple of flies in the ointment that complicate matters a bit—forcing you to consider the administrative and/or response queues that were just mentioned. Consider the case in which you extract a message from the XDLQ—can you assume that it really wasn't processed appropriately at the other end? Unfortunately, as we shall see, not always.
If the class is anything except time-out, you know definitively that the message was not processed at its destination. This is good (well, at least, simple!).
However, if the class indicates time-out, it just means that the sender has yet to receive an ack/nack from the other side. This might be due to a real problem with message itself or due to an ack/nack delivery issue. To resolve this, you should additionally inspect the message's admin queue. Why? Because of the following network scenario:
This creates an "in-doubt" state—the sending application can't be sure that its message really wasn't appropriately processed. Note that this limbo only occurs for messages whose class indicates a time-out failure—all other classes require no further confirmation.
So the interesting cases happen when a message appears in the XDLQ because its time-out (TTBR, TTRQ) expired. In these cases, in order to programmatically reduce the number of in-doubts, you may choose to inspect the message's administrative queue (again, this can be learnt programmatically by obtaining the message's AdminQueue property). If you find a matching ack/nack there, this message is no longer in limbo. Otherwise, you still can make no assumptions about your message's fate. You might need to invoke some private protocol at this point to resolve these remaining problems: for example, programmatically initiate a transactional response-queue-based conversation with the receiver or, gulp, use the phone!
However, you should only inspect the admin queue after some amount of additional time. How long should you wait? Long enough to be fairly certain that since the message time-out expired, the network was up at some point long enough to allow any ack/nacks to flow back from the receiving machine to the sender's machine.
In order to extract the appropriate message from the admin queue, you simply compare the MsgId of the message in the XDLQ with the CorrelationId of messages in the admin queue. Now you must take a look at the admin message's class property: it will indicate either a positive or negative ack. If the ack is positive, you know that effectively you have a "false alarm": the message in the XDLQ was placed there "prematurely" by the sending MSMQ service and you can safely ignore it. Otherwise, if the ack is negative, this is a true failure and your application can now take the appropriate corrective compensating action.
This seems quite complicated—let's see if we can simplify the above. It turns out that the only case in which the sender's XDLQ is not sufficient is when the message was moved to the XDLQ before its positive ack was received by the sender from the destination. In all other cases, the information in the XDLQ is accurate and reliable. In short, the mechanism indicated by the previous section is just a technique for filtering out this particular case.
In some cases, it might be unnecessary based on your application requirements to even programmatically worry about in-doubt scenarios: Sometimes you can't use a reasonably finite time-out—since you don't in general know when your messages will be processed. In this case, just use the admin queue. You don't need to bother with the XDLQ at all. Note that the default MSMQ enterprise-wide time-out is 90 days.
If you can set a reasonable time-out (but there are important cases where the preceding "in-doubt" scenario might occur) use the XDLQ/admin queue/response queue solution just described.
Should you use transactional messaging? By all means. Bear in mind the guidelines: monitor the sender's XDLQ and consider using admin and response queues. Use a transactional response queue for private application-specific recovery in the case of failures.
MSMQ1.0 doesn't support transactional read from non-local (remote) queues. This means that reliable, exactly once reception isn't available from remote queues. Note that transactional send to remote queues is supported.
There is a reasonably good solution in MSMQ that largely addresses the remote transactional read problem in the multi-client scenario. Consider the following scenario: multiple senders push messages to some central location, which are intended for processing by multiple clients. The application requires both scalability in terms of senders and processors (clients) and transactional semantics for reliability.
The following algorithm indicates how to obtain transactional remote read semantics while only performing remote transactional send and local transactional reads. This solution is also scalable in the number of clients.
Our configuration consists of external senders, a server, and multiple external clients.
In order to emulate remote transactional read, the following process is implemented for each client C(i):
The Server transactionally processes its IncomingClientRequests queue thus:
The client transactionally locally receives message M2 from its ServerResponses transactional queue. This, of course, can be done with standard MSMQ blocking or non-blocking mechanisms.
There are actually advantages to this scheme:
There is no issue with message loss with respect to the client. Here's why:
The server will need to write "recovery" code to process messages discovered in its "xact dead letter" queue (due to client TTBR failure).
The following is sample code that indicates how to implement this on the server. For simplicity there is not an async handler but simply a dispatch loop that processes incoming client requests on the server:
Dim qinfoIncoming As New MSMQQueueInfo
Dim qinfoOutgoing As New MSMQQueueInfo
Dim xdisper As New MSMQTransactionDispenser
Dim qIncoming As MSMQQueue
Dim qOutgoing As MSMQQueue
Sub Initialize()
'
' create server queues and xact dispenser
'
On Error Resume Next
qinfoIncoming.PathName = ".\incoming"
qinfoIncoming.Label = "IncomingClientRequests"
qinfoIncoming.Create IsTransactional:=True
qinfoOutgoing.PathName = ".\outgoing"
qinfoOutgoing.Label = "OutgoingMessages"
qinfoOutgoing.Create IsTransactional:=True
On Error GoTo 0
'
' open incoming request queue for receive
' open outgoing message queue for receive
'
Set qOutgoing = qinfoOutgoing.Open(MQ_RECEIVE_ACCESS, 0)
Set qIncoming = qinfoIncoming.Open(MQ_RECEIVE_ACCESS, 0)
Set xdisper = New MSMQTransactionDispenser
End Sub
Private Sub ProcessIncoming()
'
' obtain an internal xact
'
Dim xact As MSMQTransaction
Set xact = xdisper.BeginTransaction
'
' get next client message from incoming queue
'
Dim msgIncoming As MSMQMessage
Set msgIncoming = qIncoming.Receive(Transaction:=xact, ReceiveTimeout:=100)
If Not msgIncoming Is Nothing Then
'
' obtain response queue from incoming client request
'
Dim qinfoResponse As MSMQQueueInfo
Set qinfoResponse = msgIncoming.ResponseQueueInfo
If Not qinfoResponse Is Nothing Then
'
' open response queue for send
'
Dim qResponse As MSMQQueue
Set qResponse = qinfoResponse.Open(MQ_SEND_ACCESS, 0)
'
' get next message from outgoing queue to forward to client
'
Dim msgOutgoing As MSMQMessage
Set msgOutgoing = qOutgoing.Receive(Transaction:=xact, ReceiveTimeout:=100)
If Not msgOutgoing Is Nothing Then
'
' forward outgoing message to client
'
msgOutgoing.Send qResponse, xact
'
' now we can commit. In all other cases, the xact
' will abort and things will revert: in particular, if there's
' a client request for a message but the outgoing queue is empty
' this request will be re-processed.
'
xact.Commit
Exit Sub
End If
End If
End If
xact.Abort
End Sub
Sub ProcessMessages()
Do
ProcessIncoming
DoEvents ' yield so that someone else can get something done
Loop Until False
End Sub