Two MSMQ Scenarios

R.L. Parker

In this article, R.L. discusses two scenarios that have presented difficult problems for VB programmers in the past. Both scenarios involve physically separate applications that must collaborate to get some work done, even in the absence of a persistent and reliable physical connection. R.L. shows how Microsoft Message Queue (MSMQ) provides the infrastructure that makes solving these kinds of problems possible and goes on to suggest a higher-level abstraction of the problem that unifies the two scenarios -- an abstraction he calls "asynchronous remote method invocation."

Suppose we have two or more separate computer applications that need to collaborate to get a job done. Furthermore, suppose that the physical connection between the two systems is intermittent or unreliable. Since real-time communication between the systems isn't feasible, we need an alternative communication infrastructure. This alternative must detect when a physical connection becomes available and guarantee data transmission between the two systems despite any problems that occur in the physical or lower-level software layers. If it's not possible to complete the transmission, our infrastructure needs to re-try the transmission the next time the physical connection becomes available. If this sounds hard, you're right. It is. And it's been done before -- lots of times. People have been writing message queuing engines ever since mainframes. Fortunately for us, Microsoft provides a ready-to-use engine that works under Windows NT server. Its name is Microsoft Message Queue (MSMQ).

The "lob" scenario
There are a couple of variations on the problem. In the first variation, an application is divided into three parts. The first part does sensing, sampling, or some other data collection function. I'll refer to the first part as the "field" part. The second part of the application processes the data that was collected by the field part. I'll refer to the second part as the "headquarters" part. Finally, the third part is the infrastructure part responsible for transporting the data from the field part to the headquarters part.

I call this the "lob" scenario, because the field part just "lobs" the data over to the headquarters part. The field part doesn't care what happens to the data after it lobs it. It has no further responsibility in terms of data or application integrity. We'll see later how this contrasts with the second scenario.

Let's rephrase the lob scenario in n-tier terms: The field part of the application wants to invoke a method on a headquarters business object. Let's refer to this as a "remote method invocation." The logical design for such a solution is shown in Figure 1.

If we have a reliable, high-speed connection, we could set up the physical design identically to the logical design: We could develop a business object that the field application could invoke remotely via, say, DCOM. That is, the third part of our application would be implemented by synchronous remote method invocations on the headquarters business object.

However, if we only have an intermittent connection, this approach won't work. MSMQ (see the sidebar, "Quick Intro to MSMQ") provides the infrastructure that solves our problem. We can use MSMQ to implement the third part of our application. This refinement is shown in Figure 2.

Notice that we've added a business object that runs locally for the field application to use. The local business object exports the same interface as the "real" business object that runs at headquarters. Using this technique, we can insulate the application from the fact that sometimes the remote object is available and sometimes it isn't. Another way to characterize this solution is by saying that the local business object acts as a "proxy" for the "real" business object. Our field application can always invoke methods on this "proxy" object, without regard to connectivity to the outside world, as depicted in Figure 3.

The following numbered steps match the numbers in Figure 3:

1. The application calls a method on the local business object (let's name the method "AddItem"). In the lob scenario, the local business object's "AddItem" method is implemented by sending a message (via MSMQ) that will ultimately result in a call to the "AddItem" method on the headquarters business object. Note that the field system is set up as an MSMQ "independent client." As we'll see, this is transparent to the local business object (the MSMQ client). It enables the store-and-forward functionality that the local business object will rely upon to transport its method invocations across to the real business object at headquarters.

2. On the headquarters side, we need a "monitor" object. Its job is to read the messages as they're delivered to the queue. As we'll see, the monitor can be event-driven.

3. When the monitor detects that a message has arrived in the queue, it decodes the message and then calls the appropriate method on the headquarters business object.

4. The headquarters business object stores the data (or takes whatever other appropriate action on the request) if possible.

Some examples
Let's say your client phones and says something like, "We need a system that will let some remote weather stations collect meteorological data and then periodically upload the data to the main office for analysis." Or, "We'd like to lower the cost of monitoring the levels in the bulk tanks at all of our gas stations. Right now, we have men drive around once a day to each one to take the level, but we'd like to install a sensor that would take readings and then send them into the office more frequently instead." Now we know we have a tool in our toolbox that makes solving these problems relatively easy.

The "RSVP" scenario
In the second scenario, the field part of the application is a peer with the headquarters part. The two parts share responsibility for the data. This might be categorized as a truly "distributed" application where the headquarters part perhaps provides a coordination function, but otherwise is just another member of the team.

In this scenario, the field part wants to send a request to a business object at headquarters, but it must get a positive response from headquarters before relinquishing responsibility for the data. On the other hand, if the field part gets a negative response from headquarters, it means that headquarters declined to take responsibility for the data. The field must maintain responsibility for the object to preserve the overall integrity of the application. The logical design for such a system is shown in Figure 4.

As you can see, the "request" part of the scenario is a "lob." But now the field system needs to add some mechanism to "remember" requests that have been made and to determine what to do if a negative response is received from headquarters. The details are filled in with a physical design, as shown in Figure 5.

The "response" part of the RSVP scenario is symmetrical to the request part. The headquarters monitor formulates a response message based on the outcome of the call that it makes to the headquarters business object. If the call succeeds, the monitor generates a positive response. If, on the other hand, the call fails (for instance, the headquarters business object raises an error), the monitor generates a negative response.

The following numbered steps match the numbers in Figure 5:

1. As in the lob scenario, the field application calls "AddItem" on the local business object interface.

2. In the RSVP scenario, the field part's local business object takes responsibility for the data by retaining it locally. Note that in this scenario, the local business object interface is a superset of the headquarters business object interface; the local business object interface has a couple of additional methods on it that support the "transfer" functionality.

3. At some point, the application (or possibly the business object itself) decides that it wants to "transfer" responsibility for some data to headquarters, so it invokes the "TransferItem" method on the field-side business object.

4. The headquarters monitor object reads a message from the request queue.

5. The headquarters monitor object invokes the "AddItem" method on the headquarters business object's interface.

6. The headquarters business object either assumes responsibility for the data (by successfully executing the method) or declines to accept responsibility for the data (by raising an error).

7. The headquarters monitor object formulates the appropriate message and sends it to the response queue.

8. The field-side monitor object reads the response message.

9. The field-side monitor invokes the "UpdateTransferStatus" method on the field business object with a parameter value that indicates success or failure of the "remote method invocation."

10. The field-side business object updates its durable state with the results of the request.

An example
I recently worked on a project for an insurance agency. Policy information was created in field offices. Some processing could be done in the field, too, but in some cases, workload could be transferred to a headquarters office if the field office got too busy. The headquarters could either accept responsibility for the processing, decline responsibility, or transfer responsibility to a field office that wasn't busy. A perfect application for "remote method invocations" via MSMQ!

The sample application
The accompanying sample application (available in the accompanying Download file) is an implementation of the physical design depicted in Figure 5. On the field side, we have a simple application that lets us test the implementation. The application has three command buttons:


We start by adding a couple of items to the field-side data store.

Figure 6 shows the application immediately after the Add to Local button was pushed. The grid at the bottom of the form shows a view of the information in the data store.

In Figure 7, we've requested a transfer of item #1. Notice that the business object has set the "Transfer Requested" value to true, but headquarters hasn't acknowledged the request yet ("Transfer Confirmed" = false).

Now, in Figure 8, we've requested transfer of both items.

Figure 9 shows the GUI for the headquarters monitor shortly after a physical connection was established with the field. Headquarters will decline to accept responsibility for item #1 (id = 58). Why? In this case, it's because the headquarters business object was coded to reject any request whose id was evenly divisible. In real life, it could be because of a variety of reasons, such as a business rule violation, a lack of storage space, an error that happened during processing, and so forth.

Now, back on the field side, we can look at the GUI for the field-side monitor (see Figure 10). It tells us that the response messages have been received from headquarters, and, presumably, the business object has been called, once for each received message, to update the transfer status for each of the items.

Finally, we look at our application again and see that item #2 was successfully transferred, but item #1 is our responsibility again (see Figure 11).

The code
Surprisingly, there's not really a lot of code to look at -- MSMQ is pretty straightforward to program. Most of the MSMQ-specific work is encapsulated in two helper classes, one for incoming and one for outgoing queue messages.

The field business object
Let's start on the field side. If we want to transfer an item, the TransferItem method on the field-side business object is invoked. In the implementation of TransferItem, we create an instance of the helper class, clsQueueOut:

 Dim clsQueueOut As clsQueueOut
 Set clsQueueOut = New clsQueueOut


Then we set a few properties on the clsQueueOut instance:

 clsQueueOut.AppSpecific = lID
 clsQueueOut.QueueName = mcsRequestQueue
 clsQueueOut.AppendOperationName "TRANSFER"
 clsQueueOut.AppendArg sValue


We tell the clsQueueOut instance to send the message, and finally, we deallocate the clsQueueOut instance:

 clsQueueOut.SendMessage
 Set clsQueueOut = Nothing


clsQueueOut
Most of the interesting MSMQ stuff in clsQueueOut is in the SendMessage method. MSMQ provides an automation interface to the VB programmer that includes a pretty simple object model. As you can see, we're using MSMQQueue and MSMQMessage objects.

 Dim qSend As MSMQQueue
 Dim qMsg As MSMQMessage


First, we open the MSMQQueue (represented by the qSend instance):

 Set qSend = OpenQueue(msQueueName, MQ_SEND_ACCESS)


Then we create an instance of an MSMQMessage and set some of its properties: 

 Set qMsg = New MSMQMessage
 With qMsg
    .Label = mcsAppName & " " & Date & " " & Time
    .Body = MakeBody()
    .Delivery = MQMSG_DELIVERY_RECOVERABLE
    .AppSpecific = mlAppSpecific


Finally, we send the message to the queue:

    .Send qSend
 End With


and clean up:

    qSend.Close
    ClearArgs
 End If


There are a few things to note:


The headquarters monitor
The headquarters monitor is the application that's really at the core of the RSVP scenario. It's responsible for five things:

1. monitoring the request queue for messages from the field;

2. unpacking the message;

3. invoking the appropriate method on the business object;

4. displaying a visual indicator of activity; and

5. invoking the appropriate field-side business object method.

As we can see from the following declarations, which are in frmMain, the headquarters monitor uses both types of helper classes as well as our "real" business object:

 Private moHQBusiness As HQBusiness.clsHQBusiness
 Private WithEvents moQin As clsQueueIn
 Private moQout As clsQueueOut


One important thing to notice is the use of the WithEvents keyword for the clsQueuIn instance. Visual Basic automatically adds an empty event handler (a private sub) to our code for each event that the clsQueueIn instance might raise. In this case, there only two: Arrived and ArrivedError. In this sample, we're only interested in the Arrived event. This event will notify our form when a message has been put in the queue to which the clsQueueIn instance is "attached". The event mechanism helps us with responsibility #1. Here's the code in our Arrived event handler:

 Private Sub moQin_Arrived(ByVal queue As Object,_
    ByVal Cursor As Long)
 On Error GoTo EH
    Const csProcName As String = "moQin_Arrived"
    Dim qMsg As MSMQMessage


First, we fetch the message out of the queue:

    Set qMsg = queue.Receive


Then, we process it. ProcessMessage is a private method that unpacks the method name and arguments that we stuffed into the MSMQMessage body property back on the field side (responsibility #2), then invokes the appropriate method on our business object (responsibility #3).

     ProcessMessage qMsg


Next, we display the results on the GUI (responsibility #4):

    List1.AddItem Date & " " & Time & ": Transferred _
       " & CStr(qMsg.AppSpecific)


And, finally, we send a message back to the field side (responsibility #5). Remember that the field side is waiting for either a positive or a negative acknowledgement so that it can decide whether it's still responsible for the data. In this case, we've successfully invoked the business-side business object (no error was raised), so we know that the business-side object has assumed responsibility for the data and we can send back a positive acknowledgement:

    SendResponseMessage "CONFIRM", qMsg.AppSpecific   
    Set qMsg = Nothing    
    Exit Sub    
 EH: 'your error handling code goes here


On the other hand, if an error was raised while executing moQin_Arrived, we know that our headquarters business object has not taken responsibility for the data. We must send a negative acknowledgement back to the field:

    'send decline message back to field
    SendResponseMessage "DECLINE", qMsg.AppSpecific    
    'display result on GUI
    List1.AddItem Date & " " & Time & _
       ": ERROR while processing " & CStr(qMsg.AppSpecific)
    Set qMsg = Nothing    
 End Sub


Conclusion
Two physically separate applications might need to collaborate to provide some functionality. The collaboration might be in one direction only (the "lob" form) or bi-directional (the "RSVP" form). This article has illustrated how to implement this collaboration via "remote method invocations" between custom business objects. MSMQ is used as the transport infrastructure for the remote method invocations.

Download MSMQCODE.exe

R.L. Parker is a Microsoft Certified Solution Developer (MCSD) and Master Technical Lead at DB Basics, Inc. in Raleigh, N.C, who specializes in mentoring and custom development of mission-critical database applications including data-driven Web sites. rlp@dbbasics.com.


Sidebar: Quick Intro to MSMQ
Microsoft Message Queue (MSMQ) is a technology that enables loosely coupled inter-application communication. The applications involved are loosely coupled in time (the message sender and the message receiver[s] don't have to be online at the same time). Another way to say this is that the MSMQ protocol is asynchronous and connectionless. The applications are also loosely coupled in identity (the message sender doesn't have to identify the intended recipient of the message, nor does the message receiver necessarily know or care about the sender of the message). The semantics of inter-application communication via MSMQ is more like event semantics than procedure-call semantics.

While getting data from application A to application B is the core of what MSMQ does, it supports many other features. A full enumeration and discussion of them would require a book -- several of which exist. That said, here's a list of some of the more important features:


Interoperability with other message queue systems, such as IBM MQSeries, is possible via MSMQ "connector servers" such as Level 8 Systems' FalconMQ Bridge (www.level8.com) -- part of which Microsoft has licensed to include in future versions of MSMQ. (The fact that a queue is in a "foreign" system is transparent to an application on the MSMQ side.) Microsoft distributes MSMQ free as part of the Windows NT 4.0 Option Pack.

Jargon


Administration
MSMQ is administered through its own Explorer interface (see Figure 1a). Servers and queues are listed in a tree view.

By right-clicking on a server or queue in the tree view, the object's properties can be examined and/or set (see Figure 2a).

Programming
MSMQ presents a rich object model to the VB programmer. Here's a partial list of the ActiveX exposed by MSMQ:


Some of its methods include Create, Delete, and Update.

References
Msmqadm.hlp is the administrator's manual delivered with MSMQ. Msmqsdk.hlp is the programmer's manual delivered with the MSMQ SDK. I've found both of these manuals to be very useful.

Summary
MSMQ makes a lot of functionality available to the VB programmer. Though it isn't the answer to every problem, it's a good tool to have in your toolbox.