Erik Gunvaldson
Microsoft Corporation
January 2000
Summary: This article demonstrates how Queued Components (QC), a key feature of COM+ and based on Microsoft Message Queuing Services (MSMQ), transparently provides asynchronous and guaranteed delivery assurances. (12 printed pages)
Introduction
What Is Hanson Brothers?
How to Place an Order
Asynchronous Order Workflow with QC
Conclusion
In this article we will demonstrate how Queued Components (QC), a key feature of COM+ and based on Message Queuing Services (MSMQ), transparently provides asynchronous and guaranteed delivery assurances. We show this using a sample application called Hanson Brothers (HB).
In particular, we will step through an HB trade order. The order will be considered from a customer's perspective, followed by an in-depth review of its back-end workflow. There, we will focus on a single HB component and explore its activities and transaction boundaries. We will conclude by looking at how the order is filled.
Before starting, let's first introduce Hanson Brothers and define a few of its COM+ applications.
Hanson Brothers (HB) is a simplified retail brokerage three-tier COM+ sample application shipping in the Microsoft® Windows® 2000 platform SDK. HB supports Web- and Microsoft Visual Basic®-based clients. Its purpose is to demonstrate a range of COM+ services in a real-world asynchronous distributed scenario. For this article, we will concentrate on the Queued Component elements of HB.
The HB design is fictitious; it is not intended to model any existing trading or brokerage system (which are often highly customized). In addition, HB does not attempt to depict "best practices" in all cases. HB is designed so that users can easily explore and experience alternative paths within a fairly complete end-to-end distributed application.
There are four COM+ applications that define the HB scenario.
HBInstitution customers are identified by social security number and have one or more brokerage accounts. Each account maintains balances and a history of trade transactions. In addition, each account maintains a list of stock owned or held, called holdings. Also provided is a pending order list. Pending orders are simply historical orders that are confirmed, but waiting to be filled. The account details, holdings, and histories are available to the customer through the UI.
Customers, considered base clients of Hanson Brothers, may place BUY or SELL orders with HBInstitution. Orders are placed either as "Market" orders—that is, buy or sell now; or "Limit" orders—that is, wait until the desired buy or sell price is met. To place an order, the customer clicks an intent-to-execute button labeled "Verify." If enough funds exist to buy the stock at the current market price or enough stocks exist in holdings to sell, the customer is allowed to "Confirm" or place the order.
The customer can track the status of the order by periodically checking history. For example, immediately after successful execution, history will show the order's status as Institution-Confirmed (I_CONFIRM). Shortly after, when HBExchange receives the order, the order's status will be changed to Market-Confirmed (M_CONFIRM). Some time later, depending on the order type and price, HBExchange will fill the order and the order status will be set to Settled (SETTLED).
Settlement is defined as Trade + zero business days. Therefore, the terms order fill and order settlement have the same meaning for HB.
The last state an order transitions to, before it can be considered owned or held, is Reconciled (RECONCILED). This status is obtained after HBInstitution reconciles all settled orders with their respective accounts. This is typically a batch operation executed at the end of each trading day.
After the customer places the order, a number of independently running "logical" threads or activities are used to complete the order request. Each of the following activities may execute in parallel with other customer order activities.
When these activities complete the order is considered complete.
For purposes of this article, an order is considered complete when the orders settles, not when it is later reconciled.
The robustness of the HB system is determined by the transactional properties of the server objects and the invocation type of the server object executing within these activities. Figure 1 walks through a typical order and identifies these activities, their tasks, and their transactional characteristics.
Figure 1. Asychronous order workflow
Step One: The customer, Jon, places an order to "Buy at the market" one share of Walleye Pond Bakery (WEYE) to HBInstitution. The order is synchronously submitted to HBInstitution.Iorder.
If Request("submit") = "Execute" then
rc = objOrder.Execute(ssn, AcctNum, symbol, shares, price, TradeAction, _
TradeType, iOrderNum, true, False, 0, 0, sMsg)
Step Two: HBInstitution.IOrder.Execute receives and verifies Jon's order. Jon's authorization to perform the trade is declaratively and programmatically verified. For example, method level security ensures Jon is a member of the Trader, Rep, or Customer role. Security context is used to ensure Jon is not, for instance, using a trading logon to place orders over the Web.
Jon's account is also checked to ensure enough funds exist to buy the one share of WEYE. The current WEYE quote is used to estimate costs.
Step Three: The Institutional DB is updated. Jon's account cash debit is updated in the account table and the order is inserted into the holding table through a stored procedure. This procedure returns a unique Order ID (iOrderNum), which is used in the confirmation message (sMsg) and for subsequent processing. The initial status of Jon's order is Institution Confirmed (I_CONFIRM).
Step Four: The order is queued to the exchange. HBInstitution.IOrder.Exexcute creates a queued HBExchange object and invokes a method to pass Jon's order information asynchronously.
Set oExchangeOrder = CreateObject("queue:/new:HBExchange.IExchangeOrder")
When HBExchange.IExchangeOrder is created with the queue moniker, HBInstitution.IOrder receives not a direct object reference, but a reference to a call recorder. As methods are invoked, the calls are recorded on the client side. When the client deactivates the component, QC uses MSMQ to transport the recorded calls to the server.
The QC recorder initializes itself from information supplied in the queue moniker and information in the COM+ catalog. The QC recorder normally will not issue MSMQ API calls until the recorder's last reference is released, and then only if methods have been called.
The server listens for messages and activates a Player component that creates the actual IExchange object and plays the recorded calls into it. From the client's view (HBInstitution), the only difference between using queued versus a non-queued component is the way in which the component gets activated.
Call oExchangeOrder.EnterOrder(sSSN, sAcctNum, sSymbol, iAction, iQuantity,
cLimitPrice, iTradeType, lOrderId, iInstitutionId, bQueued, bTrackFlow)
The bQueued parameter directs processing to be handled asynchronously, via QC (the default), or synchronously. The bTrackFlow parameter determines whether order tracking will be turned on or not. The iInstitutionID parameter informs HBNotify which institution should receive incoming HBExchange order notifications.
HB could have implemented a response object by passing a queued callback notification reference rather than an integer. By passing a response object, the identification of the HBNotify object moves from the hands of HBExchange to HBInstitution. This could, for instance, allow notifications for mutual fund-managed orders to be handled or directed differently than customer orders.
During marshaling of the recorder (when you pass the recorder's reference as an interface pointer), the QC recorder marshals the queue format name without reference to the queue manager. If the response object recorder is created, marshaled, and released with no method calls, there is no reference to MSMQ or to the network. However, should override parameters be supplied in the queue moniker to alter the queue name, these parameters will be validated during recorder creation and the queue will be opened at that time.
Step Five: IOrder.Execute returns the following message to Jon:
"Order Entered Successfully. Your Status (Institutional) Number is 946"
This is Jon's assurance his order was properly received. Should Jon review his order history, he will note that order 946 has a status of I_CONFIRM'ed.
To ensure all-or-nothing order handling, these tasks are placed under transactional control. Selecting the "Requires New Transaction" property in the Visual Basic IOrder.cls, or through the Component Manager, defines this method as the root order transaction. Should IOrder fail, all work will be undone or rolled back, including the queued invocation, and Jon will receive an appropriate error message.
Step Six: The Player creates the server-side component and plays HBExchange.IExchangeOrder.EnterOrder.
Step Seven: The order is inserted into the exchange's pendingTrades table.
On Error GoTo ErrorHandler
…
cmd.CommandText = "sp_HBInsertPendingTrades"
cmd.CommandType = adCmdStoredProc
cmd.Parameters.Append cmd.CreateParameter("ssn", adVarChar, adParamInput, 10, sSSN)
…
rs.Open cmd
lConfirmNumber = CLng(rs("confirmationNumber")) ' Unique exchange number
rs.ActiveConnection = Nothing
…
Step Eight: A "market" confirmation message (M_CONFIRM) is sent to HBInstitution using the queued component HBNotify. This notification is the HBExchange assurance to HBInstitution that Jon's order was reliably stored.
Set oNotify = CreateObject("queue:/new:HBNotify.INotify")
Call oNotify.ProcessConfirmation(lOrderId, iInstitutionId, lConfirmNumber, bTrackFlow)
When INotify plays back a market (or settled) confirmation invocation, it uses the institution ID generated by HBInstitution.IOrder to call the correct HBInstitution server component. The IOrderID parameter is used to update Jon's order record in the holding table. By queuing these calls, INotify could become a lightweight mediator, possibly implementing a simple form of custom load balancing based on order ID.
Step Nine: After recording the confirmation invocation, EnterOrder concludes by reading the current quote and submitting the order to the IMarketFloor component, where it is checked for settlement.
Set oQuote = New HBExchange.IQuote
rc = oQuote.RetrieveEx(strSymbol, dQuoteTime, cBidPrice, …)
Set oMarketFloor = New HBExchange.IMarketFloor ' Default, used to illustrate
The Visual Basic implementation of New depends on where the object is being created. If created in the same project, COM activation is not used. If created in another project, CoCreateInstance is used.
Call oMarketFloor.GetMatchingTrades(lOrderId, strSymbol, cBidPrice, …)
If Not GetObjectContext Is Nothing Then GetObjectContext.SetComplete
Exit Sub
ErrorHandler:
…
If Not GetObjectContext Is Nothing Then GetObjectContext.SetAbort
LogError "IExchangeOrder.EnterOrder."
End Sub
Activating IMarketFloor with the New keyword will satisfy an HB requirement that market (and limit) orders be immediately considered for settlement. But is a synchronous implementation the right way to go?
One way to answer this is by looking at the tasks of EnterOrder and regarding them in terms of critical path and time dependencies. If no dependencies exist, it's likely this synchronous set of tasks, can be separated into two independent activities.
Determining if IMarketFloor is in the critical path of IExchangeOrder can depend on whether IExchangeOrder should wait on IMarketFloor and whether a failure from IMarketFloor will have an effect on the processing of IExchangeOrder. If not, a "Fire-and-Forget" relationship with IMarketFloor exists and IMarketFloor can be considered outside IExchangeOrder's critical path.
HB uses different rules for order acceptance and (market) order fulfillment response times. For example, an HBExchange quality of service agreement with HBInstitution may state a response time of two seconds for order acceptance (to the INotify queue) while leaving the (market) order fill time entirely dependent on market conditions. This suggests a time-independent relationship between the two components.
Because IMarketFloor does not share dependencies with IExchangeOrder, two separate activities may be used. This is easily accomplished by changing:
Set oMarketFloor = New HBExchange.IMarketFloor
To:
Set oMarketFloor = CreateObject("queue:/new:HBExchange.IMarketFloor")
And remaking HBExchange.dll with binary compatibility and setting the Queued Property in the COM+ Application IMarketFloor interface.
Notes:
Breaking IMarketFloor into a separate asynchronous activity also helps size the exchange, regardless of whether IMarketFloor employs transactions. For instance, a sudden increase in market activity will increase the load on IExchangeOrder without immediately affecting the settlement process. Over time, this load will shift to IMarketFloor. Additionally, the load on IMarketFloor may increase, independent of new orders. This may occur, for instance, when a large number of stop-loss limit orders fill during a market crash. Each component is therefore able to process to their maximum capabilities, independent of one another.
Now that the EnterOrder processing is broken up into two activities, how should it be transacted and how should it respond to failures?
The transactional behavior of IExchangeOrder depends on its level of transaction support, and whether exceptions are caught and handled by IExchangeOrder or by the player. The following table explores these transactional considerations using an un-transacted IMarketFloor.
| IexchangeOrder transaction |
Handled SetAbort | Unhandled SetAbort |
| None | Order may be Lost* | Order may be Entered Multiple Times |
| Supports | Order Retried | Order Retried |
| Requires New | Order may be Lost* | Order Retried |
* The Player does not receive a failed HRESULT or exception, nor does it participate in an IExchangeOrder abort. Consequently, the order message is successfully dequeued. EnterOrder must properly handle the exception, or the order may be lost.
Should the un-transacted IExchangeOrder handler choose not to raise an exception, it does so realizing the order will never be replayed. How the handler responds depends on the nature of the error. For a DB error, the handler could either initiate a compensating transaction or resolve the DB problem itself. A compensating transaction would delete the holding record, requiring the institution to inform the customer his originally confirmed order is no longer valid. Because losing an order would be quite damaging to the business, the handler is better advised to resubmit the order to an available DB server. Should an error occur while queuing the notification, the handler may choose to ignore the problem and post a warning message. The problem would not be considered immediately serious because IMarketFloor will still find the order in the pendingOrders table.
Not raising an exception while under a new transaction brings about the same situation. Although IExchangeOrders work can now be rollback as a whole, there is no way to replay the dequeued order message.
Raising an error from a non-transacted IExchangeOrder produces unwanted duplicity. The order is inserted into pendingTrades table, confirmed, and may be settled multiple times. Duplicate orders occur when the component fails and work is not rolled back, the player receives a failed HRESULT or catches an exception, and the order message is subsequently replayed. For example, when a market order message is initially played, EnterOrder produces 2 notifications (one M_CONFIRM, one SETTLED—by IMarketFloor). When the player catches an EnterOrder exception, the order message is failed and QC moves the order to the first private retry queue (hbexchange_0). From this queue, three replay efforts result in eight queued notifications. If the failure persists, the order message is moved to the next retry queue and is attempted three more times. This results in 14 queued messages. By the fourth time (hbexchange_4) 26 notifications will be produced—that is, (4 retry queues * 3 tries/each queue * 2 notifications) + 2. At this point, some 30 minutes later, 13 settled orders would be inserted into the PendingTable, instead of the expected one.
Setting IExchangeOrder to support transactions allows QC to participate in the abort and retry the order. Should IExchangeOrder fail in the middle of this transaction, the message is not lost on the hbexchange queue. The message, along with any EnterOrder work, is rolled back or undone. If the problem persists too long, the message may find its way to a final resting queue, where manual intervention may still be required.
In particular, when the final retry on the last retry queue (hbexchange_4) fails, Queued Components run time retrieves the target component from the message and checks for an exception class defined in HBExchange.IOrderExchange. If defined, the run time instantiates the exception class, queries IplaybackControl, and calls IPlaybackControl::FinalServerRetry. Afterward, the run time plays the EnterOrder method from the message to the exception class whose implementation may involve submitting the order to a different DB server. If the exception class is not defined, or its activation fails, the message moves to the final resting queue (hbexchange_deadqueue), where messages will stay until they are manually moved.
Raising an exception while under a new transaction initiates the same retries. The abort rolls back all work performed on the method and the exception raised by the handler instructs the Player to requeue.
To explore the transactional characteristics of QC in greater detail, consider QC from a client-side and server-side perspective.
From the client side (HBInstitution.IOrder), the recorder assumes the transactional characteristics of the server-side object (HBExchange.IExchangeOrder) that it is proxying for. The recorder is a transacted component if the client-side catalog representation of the server component is transacted. Conversely, the recorder is not transacted if the server-side component does not use or require transactions. For example, the client-side representation will put the recorder in a new transaction if the server-side component "Requires New Transaction." The MSMQ MQSend operation is included in the recorder's scope if there is a transaction at the recorder and if the MSMQ queue is transacted. If the MSMQ queue is not transacted, the message will be sent and the client-side transaction may subsequently abort. If the queue is transacted and the recorder is not (as HB is currently implemented), an MQSendMessage with a transaction value of "MQ_SINGLE_MESSAGE" is used, which tells the queue manager to accept this message for a transacted queue but does not involve the MQSendMessage operation in the current (DTC) transaction. QC matches the parameters on the MQSendMessage or MQReceiveMessage API to the combination of queue attributes and whether you are in a transaction. (See MSDN documentation regarding the MQ_NO_TRANSACTION, MQ_MTS_TRANSACTION, and MQ_SINGLE_MESSAGE the pTransaction parameter.) QC also ensures the ITransaction pointer of the current DTC transaction is used when appropriate. The recorder gives the composite message to MSMQ when it sees deactivation (when the last interface reference is released). The transaction boundary of the client side includes all the MSMQ sends generated by recorder components, plus any resource manager updates (generally SQL Server™ database updates) made by the component on the client side. This scope does not include anything at the server side. The transaction completes at the client with the message deposited in the client-side representation of the MSMQ queue. It will not matter if the server is connected or disconnected.
After the client-side transaction has committed, MSMQ moves the message from the client to the server where the queue is located. If the queue is transacted, MSMQ does this as a separate transaction, within MSMQ. This transaction assures that the message is delivered just once, and is deleted at the client and installed at the server atomically. DTC, QC, and your application are not involved in this transaction; it is internal to MSMQ.
After MSMQ brings the message to the server, it awakens the QC listener. The QC listener creates a transacted component called an "Integrator." Under the scope of the Integrator's transacted context, an MQReceiveMessage operation is issued. If the queue is transacted, the MSMQ dequeue joins the server-side transaction. After successful dequeue, the Integrator creates a Player (an internal component that doesn't affect the transaction), which then creates the server-side component. If that component is marked "Supports" or "Requires" transaction, it and the resources it touches join the transaction issued under the MSMQ dequeue. If that component is marked "Requires New Transaction," another transaction is created that can have an independent outcome. Typically, server-side components would be marked "Supports" or "Requires."
If the MSMQ queue is created without the transaction attribute (something the developer can do manually before creating the QC Application), the integrator, a transacted component, still opens and reads the queue, but the queue does not participate in transactions. Thus, a message can be dequeued, start the application, and lose power, and upon restart the input message disappears, having never been successfully processed. For this reason, the default queues created by QC are transactional.
The top constraints levied on this component was that it have high performance and that orders be immediately stored for market floor consideration. HB cannot wait on failure conditions (such as a crashed DB server) that may or may not correct in a reasonable amount of time, thereby delaying a market orders acceptance. The designs had to ensure that HB can proactively and immediately resolve the failure.
Less of a concern was the unlikely failure of a notification. Should a persistent client-side MSMQ failure occur, operations could intervene and manually correct the situation. Because the customer still has an institutional confirmation and the ability to fill his order is not compromised (since the order has been successfully entered into PendingTrades), having the (fixed) notification take a few hours to reach HBInstitution was considered reasonable.
In short, IExchangeOrder was not transacted in order to increase performance and reduce the risk of having any orders delayed due to a long-lived failure.
Note This implementation reflects the specifics of the HB scenario rather than those of a general rule. In Addition, HB does not currently employ clustering or implement DB fail over functionality.
Raising exceptions was also not considered due to the duplicity problem that may occur following notification failures.
Lets move beyond IExchangeOrder and return to Jon's order.
Step Ten: If the HBNotify component is running, the QC listener pulls Jon's market confirmation message off the hbnotify public queue and instructs the Integrator/Player to create and play ProcessConfirmation. This method creates a non-queued instance of HBInstitution.IHoldingUpdate and invokes ConfirmOrder.
Step Eleven: ConfirmOrder updates Jon's holding record (the one created in step three) with the status of market confirmed. At this point, should Jon choose to refresh his client, he will see that his order status has become M_CONFIRM.
In the last steps, IMarketFloor will fill (settle) Jon's order.
After the order has been market confirmed, IExchangeOrder.EnterOrder() calls IMarketFloor.GetMatchingTrades() in an attempt to fill the order (see step nine). This method is responsible for matching market or limit orders sitting in the pendingTrades DB table with current market conditions (that is, a particular stock quote, generated from an HB application called HBFeed.exe). For example, Jon's market order would be immediately filled, regardless of the current price of WEYE. However, if Jon's order were a request to buy one share at the Limit price of $53, Jon's order would not settle until a WEYE price of $53 or below was reached.
Step Twelve: The method GetMatchingTrades fills the order by first executing a multistep stored procedure, the first action of which is to update the status of all matching orders from PENDING to UPDATING. The stored procedure's second step returns as a record set those orders that where changed in the first step.
Step Thirteen: GetMatchingTrades then iterates through this record set and sends settlement notifications for each order by invoking the ProcessSettlement method on the queued component HBNotify. After each notification has been invoked (recorded), a concatenated SQL string is used to later update the orders status from UPDATING to SETTLED.
After all notifications are sent, the concatenated SQL string is executed and all orders are updated to their final settled status. Should an error occur during this process, the process can continue onto the next notifications (resume next), knowing the order has already been marked UPDATED, but not SETTLED. Because IMarketFloor handles all error and is not transacted, all recorded notifications will be sent regardless of component failures.
IMarketFloor is not transacted because this update-read-update work may be fertile grounds for a Conversion Deadlock. These types of deadlocks occur when two concurrent transactions are holding a share lock (the read) that neither can convert to an exclusive lock (required for the update). When this deadlock occurs, SQL will automatically abort one of its connections and an error 1205 may be returned. The typical prevention measure, using the UPDLOCK hint in the stored procedures SELECT, will prove ineffective because the read records will exist beyond the batch operation.
IMarketFloor had to implement a reasonably reliable design without using transactions. HB could not simply set the matching order to SETTLED, because a failure while iterating may leave the records in a questionable state (that is, can the order be considered settled if the notifications can't be sent? If the answer is no, the order must be set to a non-SETTLED status if it encounters an error during notification). To avoid marking a record that cannot be sent as settled, GetMatchingTrades concatenates an SQL string as notifications are fired off. (A recordset batch update implementation is provided as an alternative.)
Note The problems encountered are an artifact of the market's design for doing broad updates. A more realistic implementation would have HBExchange spin off separate instances of IMarketFloor for each stock. Each instance would handle pending orders from separate BUY and SELL queues.
Step Fourteen: Lastly, ProcessConfirmation is played and the order in the holding DB table is updated with a SETTLED status.
Many real-life applications combine synchronous COM invocations where appropriate and asynchronous/deferred activities using QC, LCE, or MSMQ where possible. For the asynchronous activities that require reliable message delivery, the decision to use QC or MSMQ directly may depend on how much performance is required, how much control over the delivery of the message is required, how much flexibility is needed as far as the replies, and how skilled the developer is.
For those who want the simplicity of implementing asynchronous solutions using the COM programming model, without worrying about the complexities of the underlying plumbing, Queued Components is well worth considering.
Many thanks to Dick Dievendorff for answering my many questions.