Simple Object Access Protocol (SOAP)

September 10, 1999

By David Chappell

Summary: The Simple Object Access Protocol (SOAP) is a way to use the existing Internet infrastructure to enable applications to communicate directly with each other without being unintentionally blocked by firewalls.

Introduction

In just the last five years, accessing information via the Internet has become an essential requirement of the modern world. By far the most common protocol used to accomplish this today is the Hypertext Transfer Protocol (HTTP). Supported by every browser and Web server, it's an effective technology for transferring text, graphics, and other information.

But HTTP by itself isn't enough. An increasingly important part of using the Internet is the ability to interact with remote applications. This kind of interaction is commonly modeled as a remote procedure call (RPC), with the client passing in parameters and then getting some kind of result in return. While HTTP is quite flexible, it was not designed as a protocol for invoking remote procedures. Instead, various distributed object protocols exist that can be used to communicate with remote applications. Among them are the protocol used by Microsoft's Distributed Component Object Model (DCOM), the Internet Inter-ORB Protocol (IIOP) defined by the Object Management Group, and others. All of these protocols provide a similar service, allowing a client to issue an RPC to a server application and then receive a response.

Using distributed object protocols on an intranet works quite well. Using these same protocols on the public Internet is more problematic, however. Any server connected to the Internet can potentially be accessed by any Internet user, which raises some obvious security concerns. To address these concerns, most organizations insert a firewall between their publicly accessible Web servers and the masses who can access those servers. Firewalls, provided by products such as the Microsoft® Proxy Server, can block incoming traffic based on various criteria and thereby increase an organization's confidence in the security of its systems.

While they're essential to secure use of the Internet, firewalls make the efficient use of distributed object protocols very challenging. To address this problem, Microsoft has created the Simple Object Access Protocol (SOAP). SOAP is a distributed object protocol that, like DCOM and other protocols in this category, allows RPCs between clients and servers. Unlike existing protocols, however, SOAP is explicitly intended to be usable with firewalls in place. Equally important, SOAP is not designed to be used with only one component technology. Up to now, a great deal of effort has been spent debating which distributed object protocol, and thus which component technology, should be used. However, unlike the distributed object protocols in use today, each of which is tied to a specific technology, SOAP does not assume a particular approach to creating components—this single protocol can be used with any of them.

This document explains how SOAP addresses the problem of working with Internet firewalls, and then describes the protocol itself. The goal is to make clear why SOAP is useful and to provide information about how it works.

The Trouble with Firewalls

Understanding why firewalls cause problems for distributed object protocols requires understanding how a firewall is able to distinguish one protocol from another. In the TCP/IP architecture, each widely used protocol is assigned its own port number and each request made using this protocol carries that number. HTTP, for example, is assigned port 80, while the File Transfer Protocol (FTP) relies on port 21. Most firewalls allow blocking a specific protocol by rejecting all traffic sent on the port used by that protocol. In general, firewalls are configured to allow traffic on port 80—if they weren't, HTTP requests from browsers couldn't get through. But many firewalls block most other ports, assuming that it's better just to block every port number that the firewall isn't specifically set to allow rather than allow other protocols into an intranet when each one could be a potential security threat.

This is exactly where the problem arises with distributed object protocols. Unlike HTTP, FTP, and other widely used protocols, distributed object protocols don't generally have a single well-known port number assigned to them. Instead, these protocols typically use dynamically assigned ports, with port numbers chosen arbitrarily as needed. If no firewall intervenes in the communication between client and server, this approach works well. Insert a firewall, however, and communication stops—the firewall blocks all traffic using this protocol because it's not configured to pass requests on arbitrary port numbers. The figure below illustrates this situation.

Various approaches exist to address this issue. Some firewalls, for example, can be configured to pass traffic with port numbers in a specified range. If the distributed object protocol in use can be configured to assign ports only in this range, client requests can successfully get through. Security-minded network administrators tend to frown on opening up an arbitrary set of ports, however, which makes this a less-than-ideal solution. Another option specifically for some users is COM Internet Services, which makes the initial request on port 80 and then sends ordinary DCOM packets over TCP. While useful in many cases, this technology works only with Microsoft Internet Information Server and DCOM—it's not a complete solution. A more general approach is needed.

Since nearly all firewalls pass traffic on port 80, all that's required to get those firewalls to pass a distributed object protocol is for that protocol to use port 80. This isn't quite as simple as it sounds, however, since this port is already assigned to HTTP. The solution adopted by SOAP is to piggyback a distributed object protocol on top of HTTP. The HTTP protocol is quite simple: It consists mainly of a few basic verbs—GET, PUT, POST, and others—sent from a browser to a server. Each verb can be followed by other information, all of which is typically sent as simple character strings. SOAP doesn't change any of this, nor does it require adding any verbs to HTTP itself. Instead, SOAP uses the Extensible Markup Language (XML) to define the format of request and response messages and then allows the use of the normal HTTP POST command to send this information. All SOAP traffic goes through port 80, which means that SOAP can be used on the Internet with any Web server—firewalls are no longer a problem. The figure below shows how this looks.

One of the primary design goals for SOAP is to ensure that it can be used effectively on top of the Internet's existing infrastructure—that is, HTTP, firewalls, proxies, and all the rest. For example, SOAP can use the Secure Sockets Layer (SSL) protocol for security, take advantage of HTTP's connection management facilities, and more. SOAP makes communicating with distributed object applications on the Internet as easy as it is to use browsers to access Web pages.

A SOAP Example

The easiest way to understand SOAP is to walk through a simple example. Suppose that Qwickbooks.com, a fictitious Internet bookstore, allows its customers to place orders via SOAP. Suppose too that the method used to actually submit those orders looks, in some generic syntax, as follows:

boolean PlaceOrder([in] Title string,

[in] Author string,

[out] DaysToDelivery integer);

When the client invokes this method, it passes in the title and author of the desired book, both represented as character strings. When the call completes, it returns a Boolean value indicating whether the order was successfully placed, along with an integer indicating the estimated number of days until the book is delivered.

Now suppose that a client wishes to order the book Happy All The Time by Laurie Colwin. When the method above is invoked, the information in the request must be sent to the server in some format using some protocol. If SOAP is used, that protocol is HTTP and the request's format might look as follows:

POST /BookServer HTTP/1.1

Host: www.qwickbooks.com

Content-Type: text/xml-SOAP

Content-Length: 160

MessageType: Call

<Title>Happy All The Time</Title>

<Author>Laurie Colwin</Author>

</PlaceOrder>

</SerializedStream>

As the preceding example shows, a SOAP request can be sent using the standard HTTP POST verb. As the HTTP headers indicate, this request will be sent to the application BookServer located at www.qwickbooks.com. The message's content type is "text/xml-SOAP", a new content type defined for SOAP, and its length is 160 bytes. The next header, MessageType, indicates that this message contains a call—that is, a SOAP request. Following this is the body of the request, expressed in an XML-defined format.

The body is an instance of the XML element <SerializedStream>. Within this element is another element whose name is identical to the name of the method being invoked. Since that method is named PlaceOrder, this is the <PlaceOrder> element in the example above. Within this element, in turn, are elements representing the parameters being passed with the request, which are the method's [in] parameters. In this example, the name of each of these elements matches the name of the parameter and each contains an appropriate value: "Happy All The Time" in the <Title> element, and "Laurie Colwin" in the <Author> element. This simple set of XML-defined information contains all that's needed to convey this call request to a server.

When this message has been received and processed by the Qwickbooks.com server, the server can return a response such as the following:

HTTP/1.1 200 OK

Connection: close

Content-Type: text/xml-SOAP

Content-Length: 177

MessageType: CallResponse

<__return>1</__return>

</PlaceOrderResponse>

</SerializedStream>

This is a standard HTTP response to a successful POST request, containing a response code of 200. The Connection header indicates that the underlying TCP connection should be closed, and as before, the Content-Type is "text/xml-SOAP". Following the Content-Length header is the MessageType header, just as in the request, although now its value is CallResponse, indicating that the body contains the result of a call. And once again, that body is defined using XML.

As before, a <SerializedStream> element contains the response's information. The element directly inside <SerializedStream> must have the same name as the method that was invoked, but with the word "Response" appended to it. For the response to the PlaceOrder method, then, that element is <PlaceOrderResponse>. Within this element are two more elements containing the information returned from this method call. The first, <__return>, contains the method's return value. In this example, the order was successful, so the element contains the Boolean value TRUE, represented as 1. The second element, <DaysToDelivery>, contains the value of this method's out parameter, the integer value of 7. And although none are shown in this example, SOAP also defines various error results, allowing a client to determine with some degree of clarity exactly what happened if a call fails.

Because it defines standard XML-based conventions for representing requests, responses, and the information contained in each, SOAP is both simple to understand and powerful to use. And because its messages can be carried on standard HTTP requests, SOAP gets around the problems with firewalls that bedevil other distributed object protocols. Firewalls can still be configured to block SOAP requests if desired, as they can for any other protocol. But allowing SOAP calls through does not require opening up a range of ports, as is typically the case with other distributed object protocols.

A More Detailed Look

The preceding example, while entirely valid, is relatively simple. There's more to SOAP than shown so far, and the following section describes a few other features of the protocol.

Defining Types

Requests and responses contain data representing a method's parameters and results. To pass this information, SOAP must define a way to represent various data types. Toward this end, SOAP defines two categories of types: compound types, each of which has distinct, named parts; and simple types, which do not have named parts. Arrays and structures are compound types, for example, while integers and strings are simple types. For simple types, SOAP uses the types defined in the "Specific Datatypes" section of the XML-Data Specfication (http://www.w3.org/TR/1998/NOTE-XML-data). For compound types, SOAP defines its own XML-based representations.

The types defined for use with SOAP include the following:

Integer and floating-point types
Unicode strings
Byte strings
Boolean
Various date and time types
Variants and unions (values that can take on any one of a defined set of types)

Note An instance of a Variant type must contain an indication of which of the types allowed in that Variant is currently represented.

Structures and records
Arrays

Note SOAP allows transmitting an entire array, only parts of an array (sometimes called a varying array), or just specific elements in an array.

Passing Implicit Parameters

As already shown, SOAP defines a way to pass the parameters and results of method calls using ordinary HTTP messages. Passing the parameters that are explicitly defined in a method is essential, of course, but there are situations in which it's nice to have a way to pass implicitly defined parameters as well. For example, suppose the PlaceOrder method shown earlier was invoked along with one or more other methods. Suppose, too, that all of those methods needed to belong to a single transaction. For this to work, all requests would need to carry some kind of transaction identifier, indicating that they were all part of the same transaction. Yet requiring the programmer to represent this information as an explicit parameter on the method is impractical—typically, the transaction identifier is hidden from developers. What's needed is some mechanism that allows the transaction infrastructure software to silently pass this value. In other words, a way to pass implicit parameters is required.

To allow this, SOAP defines payload headers. Sent within the <SerializedStream> element, payload headers contain elements representing implicit parameters. If a transaction identifier were sent with the PlaceOrder method shown earlier, for instance, the HTTP POST message might look like as follows:

POST /BookServer HTTP/1.1

Host: www.qwickbooks.com

Content-Type: text/xml-SOAP

Content-Length: 244

MessageType: Call

</headers>

<Title>Happy All The Time</Title>

<Author>Laurie Colwin</Author>

</PlaceOrder>

</SerializedStream>

The <TransactionID> element here carries the value 328. This example also uses mustUnderstand, an optional attribute on any payload header. If this attribute has the value 1, the receiver must understand what this element means and how to use it to correctly process this call. If mustUnderstand has the value 0 or if it is omitted, the receiver can ignore this element and still correctly handle the call.

POST and M-POST

As already shown, a SOAP request can use HTTP's POST verb. In fact, however, the protocol requires that the first request to a server is made using M-POST. M-POST is a new HTTP verb defined using the HTTP Extension Framework (

http://www.w3.org/Protocols/HTTP/ietf-http-ext

). If a request made using M-POST fails, the client can try again using a standard POST request. (In this case, future requests can also use POST because the server obviously doesn't support M-POST.) M-POST allows sending HTTP headers that can't be sent via the standard POST verb, providing more flexibility for SOAP users. Firewalls can even force the use of M-POST if desired, by simply refusing all HTTP POSTs with a content type of "text/xml-SOAP".

Conclusions

There is no shortage of distributed object protocols in the world. Yet none of them can be used unchanged with the infrastructure in place on today's Internet. By providing a simple, flexible mechanism to send requests and responses, one that rides on the ubitquitous HTTP, SOAP allows invoking remote methods without changing what's currently in place. Given the importance of accessing applications across the Internet, this is no small thing. And because it's not tied to any single object model, this new technology can potentially be used in many different scenarios. For all of these reasons, SOAP is a useful addition to today's protocol portfolio.