by Mark Berman, Glenn Scott, Caesar Samsi, Mark Kapczynski, Lori Kingery and Mukesh Agarwal
Other Contributors: Matthias Leibmann, Per Vonge Nielsen, Michael Ohata, Marco Argenti (iFront), Richard Warren (AppNet), Cecile Major (Proxicom), Saqqara Systems Inc., Walker Interactive Systems, Inc.
This document discusses the “things to consider,” from an architectural and technology perspective, when designing commerce solutions built into the digital nervous systems of enterprises conducting business electronically both with consumers and with trading partners. The intent of this document is to provide a pragmatic approach to help understand the pieces of technology and organizational aspects that need to be put together to successfully evaluate and build commerce solutions, and extend them over time.
To successfully deliver on the above objective, this document has been divided into the following four parts:
Part I—Provides an overview of the Commerce space, the building-blocks of commerce solutions addressing business problems, and a map of enabling technologies to accomplish sound commerce solutions.
Part II—Covers an example of building a Corporate Purchasing solution using a framework called the Corporate Purchasing Solutions Framework, and thereby illustrating the architectural considerations and questions a team must ask when addressing the requirements of their Corporate Purchasing environment.
Part III—Discusses the vital questions that one must ask when designing the architecture of Direct Selling and Marketing commerce solutions to meet their business needs.
Part IV—Outlines technical recommendations with a special emphasis on scalability and architectural improvements to enhance commerce solutions.
The latter half of the 1990’s has seen an explosion in the use of the Internet/Intranet and its accessibility to individuals, corporations, and educational institutions. This revolution has dramatically changed the way organizations conduct business with its consumers and with each other. The geographic boundaries that offer limited access to goods and services are crumbling, and companies of all sizes are busy building commerce solutions and adapting to new ways of doing business. The Internet/Intranet, with inherent features like easy access, real-time information, and low cost, is a natural driver for commerce solutions. Further, companies enticed with the promise of the following competitive advantages (but not limited to) are undertaking electronic commerce projects:
Electronic commerce (we will refer to electronic commerce as commerce solutions) is often misunderstood to be limited to buying and selling of goods and services over the Internet. Actually, commerce solutions are a lot more than just the handling of business transactions and fund transfers over the Internet. They define new forms of doing business. In addition to providing buying and selling services, commerce solutions can provide a complete system of services built into an organization’s digital nervous system so it supports the sales processes and provides total account management.
The services that help build the foundations of successful commerce solutions are:
The following diagram illustrates the interaction between these services within the context of commerce solutions.
Some common features of a commerce solution utilizing one or more of the above services include:
Commerce is a leading scenario in Microsoft’s Digital Nervous System (DNS) vision. Microsoft’s commerce strategy delivers a comprehensive commerce platform that allows businesses to build solutions that cost-effectively engage and transact with customers and partners online.
Microsoft’s commerce platform foundation consists of technologies and products that enable and support electronic commerce. Businesses can build low-cost, high value commerce solutions that scale with the needs of their business and integrate with their existing systems and data. The main products/technologies supporting Microsoft’s commerce strategy within the context of the services defined above include:
The following diagram illustrates how the above-mentioned Microsoft products and technologies work together to help businesses design and implement flexible, scalable commerce systems quickly and with reduced risk. One additional benefit of this flexible, scalable architecture is that numerous third party technologies can be integrated into the system to enhance and meet the needs of specific commerce solutions. Further, technologies like Microsoft SNA Server provide a way to embrace Internet/Intranet technologies while preserving investments in existing mainframe-based systems.
The following diagram shows a development model for Windows DNA.
Commerce solutions map very well into the Windows DNA model. Like the architecture of most robust, scalable solutions, the design of a typical commerce solution includes a layered architecture comprising of the Presentation, Logic/Business Rules, and Data tiers supported on a platform of system services and powerful development tools which provide programmability across the platform. In the following paragraphs, we discuss how some technologies within the Windows DNA tiers map into implementations within the commerce solutions.
Presentation: Makes the application easy to use and provides an intuitive way for users to interact.
Win32 API—Microsoft Management Console Snap-ins, Pipeline Editor, Analysis Report Writer.
Components—Microsoft Wallet, List Control object, and so on.
HTML/DHTML/Scripting—Storefront, Store Management, Server Administration, and so on.
Logic/Business Rules: Components to create and enforce the required customer, product and business rules.
Web—ASP display logic integrated with business and data functionality exposed through COM objects.
Transaction—Commerce pipelines and COM components enforcing transactional integrity through Microsoft Transaction Server (MTS).
Queuing—Interchange pipeline and COM components enforcing asynchronous transactional communications between distributed programs.
Security—User management through Site Server AUO (Active User Object), application security using certificates, IIS, and so forth.
Pipelines—Platform for modeling business logic and data interchange.
Data: Real-time (or close to real time) data exchange among various systems responsible for making data flow.
Heterogeneous data store—User profiles, product information, order status, inventory, and so on, stored on various types of data stores, such as RDBMS, file systems, Exchange public folders, Web servers, FTP servers, News servers, and more.
ADO—Simplified programmatic access to data.
OLE DB—Provides a common API for accessing all types of structured data.
XML—Standard-based data representation and data translation used in the Commerce Interchange Pipeline for information interchange.
Cost-effective, scalable commerce solutions can be built using the model and platform mentioned above. Microsoft Site Server 3.0 Commerce Edition, also known as Commerce Server, is the comprehensive server for conducting business online. It is closely integrated with Microsoft Internet Information Server (IIS) version 4.0 and Microsoft Windows NT® Server version 4.0 to provide an Internet Commerce platform with the business intelligence to do, among other things, the following:
Presentation:
Logic/Business Rules:
Data:
Site Server 3.0 Commerce Edition—Commerce Server—is designed to enable businesses to author new commerce sites, add commerce capabilities to existing Web sites, or extend Intranet sites to select business partner sites on an Internet Commerce platform which is both secure and reliable.
The three main goals/features of Commerce Server that combine to address the needs of commerce sites or applications are:
Electronic commerce solutions involve doing business online. Businesses want to harness the power of digital information to understand the needs and preferences of customers and trading partners and then deliver the products, services, and information to them as quickly and with as little human interaction as possible. Companies, enticed by the promise of delivering targeted, personalized, automated goods and services are engaging in various commerce applications such as online stores, online advertising, exchanging order, inventory and billing information with partners, online banking, and corporate purchasing applications that extend their enterprise to key trading partners.
The following diagram displays multiple commerce communication and solution possibilities between Business-to-Consumers (B2C) and Business-to-Businesses (B2B). For example the arrow marked ‘A’ displays an instance of a B2C solution where a consumer can place an order for a product with a distributor and/or retailer. This commerce solution can be further extended to a B2B solution as indicated by the arrows marked ‘B’. Here trading information and communication between partners (in this case the partners being distributor/retailer, some corporation, and suppliers) is being exchanged electronically using agreed upon protocols implemented through specific commerce solutions put in place to facilitate this interchange.
Based on the these two broad categories (B2C and B2B) Microsoft has identified three main scenarios or service areas where companies conduct business online today:
Selling over the Web allows businesses of all sizes to reach consumers all over the world. This requires that businesses provide consumers with an engaging shopping experience, run price promotions, cross-sell items and accept typical consumer payment methods such as credit cards. Further, when an order is placed through an organization’s Web site, consumers must be able to track it. Commerce solutions that sell to consumers must be able to map to existing business processes, and integrate with multiple databases and/or accounting systems.
Supply Chain integration, also known as Value Chain integration, uses the low cost of the Internet to highlight a tighter integration across suppliers, manufacturers, and distributors. Many of the fundamentals of building a site, extensible order processing and integration with other systems remain the same as that in the Direct Marketing & Selling scenario. But in the Supply Chain scenario, new requirements arise, including authenticated log-in, generating custom catalogs for key customers and pricing and payment based on custom agreements. Suppliers need to be able to provide their business partners with secure access to their existing Web site or be able to “push” catalogs into another business’s system, with the ability to maintain these product catalogs when pricing and/or inventory changes.
Businesses want to leverage the Intranet and the Internet to make existing business processes more efficient. At the heart of this business model are commerce solutions that facilitate the process of purchasing low-cost, high volume goods for maintenance, repair, and operations (MRO) of a business. Labor and paper-intensive operations are converted into self-service applications where purchase approvals and business policies are enforced through automated business rules. Approved purchase orders then need to be sent to suppliers. Corporate Procurement commerce solutions allow for transactions to be made with partnering businesses, suppliers, and distributors, regardless of the data format, and data is communicated—whether it be over the Internet, an EDI VAN (Value-Added Networks), e-mail, or simply fax.
The following sections discuss the “building blocks” and the Microsoft technologies that support the Direct Marketing & Selling, Supply Chain Integration, and Corporate Procurement scenarios. It is interesting to note that the building blocks provided by the Microsoft commerce platform are similar for these solutions. However, the emphasis and methods on how these commerce solutions Engage, Transact, and Analyze to accomplish the desired functionality is varied. In the remaining portion of Part 1 of this paper the three leading scenarios of Direct Marketing & Selling, Supply Chain Integration, and Corporate Procurement are discussed.
This scenario covers the retail solution where an organization increases its customer reach by creating a virtual store so consumers can obtain products online. Further, this virtual store should be able to advertise other products and be able to promote sales through discounts, cross-selling, and up-selling.
The following diagram displays the architectural framework of a Direct Marketing & Selling Solution.
Building Blocks for a Direct Marketing & Selling Solution
Microsoft Technologies supporting Scenario 1
Active Server Pages (ASP) running on Microsoft Internet Information Server (IIS) installed on machines running Microsoft Windows NT Server, build and serve up HTML pages over the HTTP protocol. Visual Interdev™ and/or Microsoft FrontPage® can be used to create these Active Server Pages. Site Server 3.0 Commerce Edition comes with five sample stores four of which demonstrate different kinds of Direct Selling solutions. The Store Builder Wizard can be used to modify these stores without having to manually re-develop the store from the bottom-up.
Personalization and Membership are two features of Microsoft Site Server 3.0.
Membership provides a storage directory that is a central repository of user data, personalization profiles, and address book information. This Membership Directory is stored in a Microsoft SQL Server database and accessed using LDAP version 3.0, which is an industry standard for storing and retrieving data from a directory. The Membership Directory Manager is a tool that is used to manage the information contained in the Membership Directory.
Personalization allows sites to offer personalized content automatically to different users, matching stored information about available content to user needs and preferences. There are three sides to personalization: User Profile data, Content Sources, and Rules.
User Profiles are kept in the Membership Directory and are used to customize Web content for users logging onto the Web server.
Content Sources is any location where tagged content is available. Personalization can draw from many types of content including files, databases, newsgroups, Microsoft Exchange public folders, and so on. The content is tagged using the Tag Tool so it can be identified easily and matched with a user’s preferences, or via some content management method when content authors publish content to the site.
Personalization Rules are the link between the delivery of content and the user. Personalization rules evaluate a statement about the user and, based on the result, a specific action is performed. The statement can be based on data found in the user’s profile stored in the Membership Directory and/or based on actions performed by the user. The Rule Builder tool is used to create these conditional rules.
Customers collect products that they want to acquire in a virtual ‘shopping basket’ called the OrderForm object (part of Microsoft Site Server 3.0 Commerce Edition). The OrderForm object provides for the in-memory storage of customer and purchase information. The OrderForm object stores the items that a customer has chosen to purchase, and also stores receipt information that reflects a given customer’s purchase history.
Windows NT accounts and Access Control Lists (ACLs) are used to assign different levels of permissions to address the needs of various groups (for example, customers, managers and administrators). Typically, most of the servers sit behind a Firewall to limit access to sensitive information and may have a secure mechanism for the transferring of sensitive information.
Microsoft IIS can impersonate the browser user and can provide authentication using one of the following methods:
Anonymous Access—typically used for the anonymous browsing of information.
Basic Authentication—clear text authentication that can be used in conjunction with SSL (Secure Sockets Layer) for security.
HTML Forms—similar to Basic Authentication except state is automatically maintained via a cookie.
Cookie Authentication—if a user has been authenticated before and received a cookie, they may be able to log on again by presenting the cookie.
DPA (Distributed Password Authentication)—similar to Windows NT Challenge/Response, the password is never sent over the wire. This is a proprietary solution.
Client Certificate Authentication—users could be supplied client certificates for added security.
Also, Microsoft IIS adds a layer of security on top of the NTFS file security system. This can help determine exactly what permissions are granted to users by IIS, depending upon whether they are accessing a particular file, directory, or virtual directory using the Web service or through a network file service.
Several strategies (or combinations of them) can be used to protect sensitive customer information like credit card information, passwords, and so forth. Secure Sockets Layer (SSL) can be used to encrypt data between the transport and the socket layer of the TCP/IP protocol stack between client and server. All HTML and ASP files on the IIS Web server that receive, display, or process information sensitive to the customer must be secured by SSL. During the SSL handshake, a unique encrypted session key is generated by the browser for bulk encryption of information, and then transferred to the IIS Web server via a public key operation so that information can be transferred securely.
Microsoft Wallet provides security to consumers by encrypting the payment data stored on the user’s workstation. Credit card information is encrypted and stored on the local machine that has Microsoft Wallet installed and can only be accessed using a password. Further, data stored in the Wallet is never transmitted without the customer’s approval. Secure Electronic Transaction (SET) is an industry standard that can be used to provide a secure transport mechanism between the customer, merchant and bank, without the merchant receiving sensitive information, such as the customer’s credit card information. Third-party payment providers would need to provide custom solutions to support this as far as server side and bank gateway integration are concerned. The Microsoft Wallet, provided with Internet Explorer 5.0, is SET 1.0 certified and can encrypt transactions and communicate with SET compliant financial institutions.
The Order Processing Pipeline (OPP) is a framework included with Site Server 3.0 Commerce Edition. This framework has an extensible architecture that can be customized so that orders are processed according to the specific business rules of an organization. Typically, three types of pipelines are used for B2C commerce solutions: Product, Plan and Purchase. Each pipeline consists of a sequence of stages that mirror the steps involved in a customer’s normal shopping experience. Each stage consists of zero or more components, which are executed in a sequence performing operations on the OrderForm object. In addition, the pipeline architecture supports transacted pipelines for guaranteed delivery of information via Microsoft Transaction Server (MTS) and the MtsTxPipeline object.
Business rules are typically implemented in re-usable COM components written in any COM compliant language, such as Visual Basic®, Visual C++® and/or Visual J++®. These components can be plugged into the various stages of the OPP. Business logic can also be implemented using VBScript and the Scriptor tool included with Site Server Commerce Edition. This provides a developer with the ability to wrap script as a COM component without having to understand how to develop using COM.
Advertising Server (Ad Server) is a service included with Site Server 3.0 Commerce Edition that manages the scheduling and delivery of online advertising. The two main elements of Ad Server are Ad Manager and the AdServer Component.
Ad Manager is an ASP-based application for managing data about advertisers, campaigns, ads, schedules, and so on.
AdServer Component is a COM object that runs on the IIS Web server processing requests for advertisements and determining the most appropriate advertisement to deliver to the consumer’s browser based on personalized attributes defined by business rules.
Ad Server also provides the ability for a site to allow external campaign managers to log-on securely and manipulate their campaigns. Further, Ad Server can audit the number of ‘click-throughs’ a campaign may generate for revenue assessment.
The Buy-now Wizard included with Ad Server is a powerful online direct marketing tool that can help present product information, order forms, or capture customer profile information within any online context, such as an online ad banner. It allows these functions without requiring the customer to leave the current site and visit that particular product’s site. This enables a site to support Direct Marketing without having to worry about losing visitors to advertisers.
The Predictor object in Site Server 3.0 Commerce Edition uses previous shopper trends and transaction information to recommend products of interest to other customers. The Predictor object is a COM object that makes its suggestion by correlating the items that the customer has ordered with a database of items that other customers with similar profiles have previously ordered. This feature, commonly known as “collaborative filtering,” makes it possible to create communities of shoppers sharing similar preferences, who can be reached by targeted product offerings, thereby creating opportunities for additional distribution.
Commerce solutions built using Site Server 3.0 Commerce Edition are basically a collection of ASP pages and COM objects whose development is greatly simplified by the tools, servers, and wizards provided by Site Server 3.0 Commerce Edition. The Commerce Server architecture revolves around data that is obtained dynamically. Data is stored in ODBC-compatible databases whose tables are designed to reflect the business requirements of the site. This allows for flexible schemas and for solutions to leverage investments in pre-existing database(s). Typically Microsoft SQL Server™ is used as the database to store the data; however, other data stores like Oracle, DB2, and so on may also be used to store information.
Microsoft ActiveX Data Objects (ADO) is the preferred connection object that uses ODBC and/or OLE DB to connect and retrieve data from the data stores via ASP. The use of Microsoft Transaction Server (MTS) to manage connection pooling between the database and the ASP pages greatly simplifies the programmatic access to database objects, and also maximizes the efficiency of the site by compressing data traveling across servers.
Site Server 3.0 Commerce Edition is tightly integrated with IIS and the Microsoft Management Console (MMC) for flexible site management and administration. The tasks of site administration and server administration have been separated into two areas based upon typical job functions.
Server Administrator is a server management tool that has access to all commerce sites running on the system, and can be used to perform server maintenance tasks such as the creation and deletion of sites, opening and closing of virtual stores, and other server administration tasks.
Site Administrator is a site management tool that typically oversees the management of a single commerce site. This tool provides Web-based administration for tasks such as adding and removing products, modifying the site’s department structure, running sales and promotions, reviewing site traffic, and customer orders.
The Promotion Manager included with Site Server 3.0 Commerce Edition is a tool that helps with the running of price promotions and cross/up selling easily. It allows for updating and managing of promotions dynamically, and for the creation of complex rules that determine how the promotions are applied (e.g. buy X get Y at Z% off).
The Site Analysis tool provided as part of Site Server 3.0 Commerce Edition allows for importing of data from IIS and Site Server log files to a data warehouse, and to run a series of reports about site usage, sales, and customer statistics. The Site Analysis tool can be used to automate the measurement of traffic to the site, both overall and by page/section, to obtain information on the most frequently purchased items, customer’s geographical distribution, distribution of purchases over time, and so forth.
All these reports are of extreme value when running an online store, since they provide the business intelligence to measure the efficiency in running the site, placing products, running promotions, targeting customers, and so on.
This scenario covers the solution where a business buyer purchases goods from a supplier. Often this is simply described as “supplier-side purchasing” by distributors, manufacturers, retail merchants, transport providers, distributors, and other business buyers. The full power of electronic commerce is harnessed by creating seamless chains of materials and services which work together to supply the needs of the consumer. This helps participants plan more effectively and adapt to changing business conditions rapidly. The diagram below illustrates the chain that links various possible entities in a Supply Chain solution.
For an example of how a Supply Chain Integration may work, let’s use an order for a new PC. The process has the following steps:
The goal of a Supply Chain electronic commerce solution is to deliver a dynamic data stream that connects trading partners from around the world with real-time data. To reach this goal, all participants in a Supply Chain solution must adopt an integrated set of data standards so that data flow through them is smooth and seamless.
Building Blocks for a Supply Chain Integration Solution
The core commerce platform for a Supply Chain Integration solution is very similar to the Direct Selling & Marketing scenario. In a Supply Chain Integration solution, the business is selling to a business rather than selling to a consumer. Most of the building blocks for this solution are very similar to a direct marketing and selling solution except that the majority of the communication between businesses is automated:
Few additional things that need to be put in place to accomplish this scenario are:
Microsoft Technologies supporting Scenario 2
SSL and password authentication using either basic or html forms log in can be used to provide secure log in and password authentication, provided a process of trust between the applications of the trading partners has been established. At the protocol level this could be via VPN (using PPTP [Point to Point Tunneling Protocol]), while at the application level the trust process could be achieved via S/MIME or through the Commerce Interchange Pipeline (CIP) using digital certificates.
This is the key building block of all B2B solutions. The Commerce Interchange Pipeline (CIP) framework, included with Site Server 3.0 Commerce Edition, provides for application-to-application interchange in multiple data formats and transport protocols to support B2B. Trading partners with high levels of automation and reliability often exchange information using electronic data interchange (EDI) which traditionally uses leased lines or value added networks (VANs). The CIP can be used to provide integration of EDI for handling business information over the Internet. CIP is a flexible workflow system that is similar in concept to the order processing pipeline (OPP), but is designed to simplify the integration of business communication among Internet-based partners. CIP enables businesses of all sizes to exchange information, such as purchase orders, receipts, shipping notices, invoices, billing records, product availability, pricing look-ups, and more, in various formats over multiple transports. Typically business information is exchanged in pure XML, XML wrapped in Binary objects over SMTP, FTP, HTTP, DCOM, MSMQ and/or EDI VANs (see diagram below).
Transactional integrity across components embedded in the CIP can be provided by MTS. The CIP fully supports transactions provided that all components, which are plugged into the pipeline, are transaction enabled.
Digital Certificates are the primary tools for securing the transfer of information in the CIP and managing trust relationships between trading partners. Trading partners can establish authenticity by verifying the partner’s digital certificates, which can be generated by using the Microsoft’s Certificate Server. A free downloadable add-on to the CIP, called the CIP Manager, can be used to easily manage the creation of certificates, trust relationships between trading partners, and data translation either via XML or EDI (provided an EDI translator is running).
The diagram below illustrates how the CIP, running under Microsoft Windows NT, can communicate either with a CIP on another computer running Windows NT, or with an EDI translator or XML parser running under any operating system. Application A and Application B exchange information through two Commerce Interchange pipelines. One of these pipelines, the Transmit pipeline (sending application side—Application A), packages business data into an object, maps, digitally signs, encrypts and transports the object, and audits the transmission of the object. The other pipeline, a Receive pipeline (receiving application side—Application B), receives, reads and performs the reverse process as it receives, decrypts, verifies the digital signature, maps, and audits the object.
For such a solution to be effective there will have to be some integration with existing systems. In most cases, organizations already have some ERP/LOB system that provides financials accounting or inventory of goods or both.
The CIP can be used to enable certain business processes so that automation can be achieved. In the Supply Chain example at the beginning of the section, for a dealer to interrogate a manufacturers catalog or send a request for goods, there has to be some integration with the existing systems. The dealer may not have it’s own inventory management system with certain available parts and instead choose to leverage off a manufacturer's catalog. The dealer may do a look-up into the supplier's catalog to check for product availability by developing a stage in the CIP. If certain components are not available, the dealer may generate a request for these components from the manufacturer, which in turn may trigger a request to the supplier for the specific parts.
For such a solution to be effective, the manufacturer has to either maintain it’s own catalog of available products or the dealer has to have it’s own copy of the manufacturer's catalog so as to be able to question product availability, pricing, and so on. The supplier would also have to house some cataloging facility so that suppliers can also check for product availability, and so forth. The dealer and supplier could easily develop a stage(s) in the CIP that would look up the relevant information in the respective catalog. XML or EDI could be used for the translation of information back and forth between the respective parties.
As opposed to a Business to Consumer scenario, where goods are typically priced at a common level for all customers, and price promotions are applied in special cases to vary the base price, in a B2B/Supply chain scenario prices are the result of often complex calculations.
Prices typically vary on a per-account basis, and are determined by factors such as:
Custom COM components or scriptors in the Product Pipeline (invoked every time a price needs to be displayed or calculated) or in the final stages of the OPP can be developed to calculate custom pricing based on information stored in the trading partner’s profile and the contents of the Order Form.
This scenario is another example of a business-to-business commerce solution. Typically this scenario involves an Intranet environment where employees of a particular organization (Requisitioners) can order MRO goods, office supplies, and general services from various vendors and suppliers in different countries. The goal of Corporate Procurement solutions is to automate the task of processing and fulfilling orders on goods that are low cost but high-volume.
In this scenario, typically users fill out requisitions using an interface such as a browser to use self-service Web applications to place orders for goods. These orders can then be electronically routed to approving managers. Company officials are able to enforce purchase approval policies through automated business rules. Once purchase orders are approved, they are routed electronically to the business partners (typically suppliers) and committed for fulfillment. Products ordered subsequently will be logged to accounts payable applications and delivered to the Requistioner in accordance with the terms in place between the buyer’s organization and the supplier. This process is illustrated in the diagram below:
Successful Corporate Procurement systems offer the ability to place orders with multiple vendors and have a flexible order processing system. This means that they must be able to communicate purchase orders, status reports, and delivery notifications via the means most suitable for each of the business partner.
The three most common ways that typical Corporate Procurement systems exchange and transact order information are: e-mail messages, flat-file order output, and custom applications built in the Commerce Interchange Pipeline (CIP). Some of the key attributes of a Corporate Procurement commerce solution are:
Building Blocks for a Corporate Procurement solution
The commerce platform for a Corporate Procurement solution is very similar to that of the Supply Chain solution. Although the back end purchasing process may be automated as in a Supply Chain scenario, the actual buying experience will be a manual process (as in the Direct Selling solution) in which individuals purchase goods as required. The technical considerations/implementation and the ‘building blocks’ for both these scenarios are almost identical. However, following are a few business and design considerations that are different between these two solutions:
Microsoft Technologies supporting Scenario 3
During the design of the application, there may be a requirement to standardize on a browser type and version so that the richest interface can be achieved without having to develop/design for the lowest common denominator. Internet Explorer provides a rich development environment in which developers can take advantage of client-side scripting so that users experience a lightweight, feature rich environment. Internet Explorer supports VBScript, Java Script, and DHTML.
There may be a requirement for an application where users can physically update/modify information in an existing e-mail system, such as Microsoft Exchange Server or an HR database. An ActiveX control could be built to take advantage of systems where personal information may reside and this information could at some stage be synchronized with other systems within the environment. As an example, if the organization were running a SAP HR system and Microsoft Exchange Server, a connector is supplied by SAP so that information can be updated/modified in one place and synchronized between multiple systems. An ActiveX control may be developed so as to empower the user to manage and update individual personal information, like telephone details, shipping details, and so on.
By standardizing on a browser type and version, certain security functionality can be leveraged within an organization. Windows NT Challenge/Response provides the ability for users to be authenticated seamlessly without the need to physically log-onto a resource by supplying a username and password. Existing security infrastructure and policies can be leveraged by using a browser, such as Microsoft Internet Explorer, which fully supports this kind of transparent authentication. No additional username or password directories would need to be maintained. One of the big benefits of Windows NT Challenge/Response is that password information is never sent over the wire.
By leveraging off existing systems within an organization's environment, a users self-service experience could be enhanced by automating repetitive tasks. As an example, shipping information could be retrieved from an existing e-mail directory, such as Microsoft Exchange Server via CDO (Collaborative Data Objects). The same could apply for manager approval, where forms could be pre-populated with manager e-mail address information all retrieved from an existing e-mail directory, so that routing/workflow processes can be built. Rather than building this in a new directory/database, this facility may already exist today. Although the information may not be available, it would be in an organization’s best interests to streamline existing systems so that automating the procurement process is made easier and richer.
As mentioned above, existing systems may already maintain user information, such as the Windows NT Directory, but certain additional information may be required in order to extend user attributes. Site Server Membership, running in an Intranet mode, can be used to create additional attributes for existing users maintained within a Windows NT Directory without having to add the same usernames and passwords in another directory. As an example, there may be a requirement to maintain an employee’s driver’s license number for use in a specific application. Site Server would create a mapping automatically between the user of Windows NT and the extended user attributes via the REMOTE_USER name passed by the browser or the logged on username when the user accesses the specific application. Thereafter, every time a user accesses the specific application and is authenticated transparently via Windows NT Challenge/Response, this attribute could be used by the application for additional functionality.
Below is an example of a manual purchasing process with a typical scenario within any paper-based requisitioning environment.
Courtesy Clarus Corporation
In a manual paper-based environment, a Requisitioner would have to obtain product information either from the procurement department or by contacting the relevant suppliers within the vendor base. As an example, if the Requisitioner were trying to purchase a specific electronic commerce book, they would have to inquire with all the book vendors as to product availability and cost.
Once the Requisitioner has obtained the information, they are then able to proceed with creating the requisition by completing a paper-based requisition form. The Requisitioner must know from where to obtain the requisition form, how to complete the requisition form, and what departmental information may be required—including cost centers, authorizers, capex codes, vendor codes, and so on.
Once the Requisitioner has completed the manual requisition form, they are then able to request approval from the relevant authorizer, assuming the relevant authorizer is available, as well as any other parties that may be required for approval. Once the requisition has been approved, a manual purchase order is then generated and submitted to the supplier for fulfillment.
Assuming the correct supplier does receive the purchase order, and assuming the product is currently in stock and the quoted price is valid, only then can the supplier fulfill the purchase order.
The supplier has to rely on the accuracy of the address on the purchase order for delivery of the goods. During the fulfilling stage, the Requisitioner has very little knowledge as to product availability and delivery times. An exception would be if Service Level Agreements had been set-up between the customer and the supplier or if the Requisitioner had manually called the supplier to try and track down the status of a purchase order.
At some point, the goods from the supplier have to reach the receiving office of the Requisitioner. The goods have to be unpacked and forwarded via some internal process to the Requisitioner, assuming the Requisitioner's location can be identified.
Once the goods have been received, a check would probably be issued for the amount as stated in the purchase order and possibly passed onto the deliverer/courier from the supplier. The Requisitioner still does not know if the goods have been received.
The Corporate Purchasing Solutions Framework (CPSF) (http://www.microsoft.com/dns/commerce/corppurchasing.htm) has been created to put in place a technical deployment framework for planning, developing, and implementing corporate purchasing solutions on the Microsoft Commerce platform. This framework also provides consistent approaches, techniques, and applications to enable developers to deliver complete solutions rapidly through continuous knowledge-sharing concerning customer best practices and new ISV solutions.
The CPSF is made of ten areas that affect both the customer (buyer) and the supplier (seller). These ten areas should be considered as questions that a procurement team should ask to better understand what is required within their environment.
Since Procurement Solutions differ from company to company, no single solution can fill this gap, but it is important that each team understand whether they should be building, buying, or doing a combination of both to achieve the best solution.
The following diagram depicts the CPSF from both the buyer and supplier side.
The ten stages that the above diagram tries to address are:
This is probably one of the most difficult steps within the automation process. The team has to identify whether the system being developed will leverage off existing business rules that a financial/accounting system may provide today, or whether some of the rules will have to be re-created (duplicated) to achieve the best solution.
It is important to understand how much integration will be required, what skills are required, and how easy it is to integrate with the financial system. Most financial system vendors provide some mechanism for integration into their system via a middle-ware environment through some COM/Java components/objects. If this is provided, it is important to know how much functionality will be provided by these components, how easily these can be accessed via common development tools, and how much additional development will be required.
A good way to start the project would be to do a user requirement analysis. For the first phase, get the following people into a room: business sponsor, Requisitioner, procurement person, developer, project manager, and a logistics person. The objective for the session should be to try and understand the current procurement flow and the steps required to automate. As a defined deliverable, the session should aim at producing a flow diagram highlighting the steps required for a Requisitioner to purchase goods in an electronic mode. This will also provide a great reference point for future phases in the project, and provide answers to many questions, as the project team gets deeper into the project. As an example, the following is a sample flow diagram If this were an MSF project (http://www.microsoft.com/msf), this stage would be considered the Conceptual Design Phase:
Note The word ‘basket’ is used within the same context as requisition/order form.
Conceptual Design Diagram
User accesses the procurement site and is prompted to input username and password. If the user has registered before they will proceed to step 2.1.1.2. If they haven’t registered before, the user will be prompted with step 2.1.1.1.1.
A screen to capture new user information, alias, password, first name, last name, department, catalog preferences, and postal address information. Once this is completed, the user will be reverted to step ) 2.1.1.2 Preferences
The user is prompted to select one of the following options: order status (2.1.1.14), list saved basket (2.1.1.17), new order (2.1.1.3), or modify the users profile information (2.1.1.1.1).
The user is prompted to select one of the following options: order status (2.1.1.14), list saved basket (2.1.1.17), new order (2.1.1.3), or modify the users profile information (2.1.1.1.1).
A catalog page appears where the user selects items to display from a specific category. Once the user selects the specific category, individual line items appear with an option per line item to ‘select item’ (2.1.1.5). On the catalog page, the user also has the option to specify non-standard items (2.1.1.4) that will be fulfilled manually by procurement. The user also has the option to view their basket (2.1.1.6).
A page appears with an area for the user to supply some descriptive information about the request.
Once the user has selected the specific line item from the catalog, they are presented with a page that displays options for the item. Some of the options are: item description, picture, quantity, price per item, delivery details, and add to basket. If the user selects ‘add to basket’ the item is added to the user's basket and the user is transferred to view basket (2.1.1.6). If the user has returned from the view basket (2.1.1.6), they should not have the option to ‘add to basket’ but rather ‘update basket’ (2.1.1.6).
The shopping basket, order form, or requisition form should contain columns detailing per line item all items for the order. The following should appear in the columns: item name, quantity, price and total. The form should also contain a total field for the entire order. The user should also have the opportunity to edit each line item (2.1.1.5), remove each line item, and include some motivational information about the order. The options that should be presented to the user are: submit order (2.1.1.19, 2.1.1.7), add more items (2.1.1.3), cancel order (revert to 2.1.1.2), and save order for later use (2.1.1.16).
Once the user has submitted the order, an e-mail should be generated by the system and sent off to the user's manager (approver) for approval. The e-mail should contain a link directly to the Requisitioner’s basket.
Once the approver clicks the link in the mail, they are presented with an approver log in (2.1.1.9).
Much the same as (2.1.1.1), the approver is presented with a screen to input their username and password. Once the approver enters their username and password and the approver name and password are validated, the approver is then presented with the approver list (2.1.1.10).
The approver is presented with a list of items that the Requisitioner has requested. The list looks similar to view basket (2.1.1.6) but is customized for the approver. The following fields should be presented to the approver: item, quantity, and price, as well as the option to view detailed information about the order (2.1.1.5). There should also be a field that contains the motivation provided by the Requisitioner. Lastly, the approver should also have the option to approve (2.1.1.11) or reject (2.1.1.18) the order, as well as the ability to add some comment to the order.
Once the approver has approved the order, the system enters that order into its own system. Once this is successfully completed, e-mail status to Requisitioner (2.1.1.12) happens again. The financial system also passes back an order number that is added to each line item for the order.
The system should generate an e-mail to the Requisitioner informing the user of the status. The e-mail should contain a link to view the status of the order in the system (2.1.1.13).
Once the Requisitioner receives the status e-mail (2.1.1.12), they then have the opportunity to click on the link to view the status of their order. Once the user clicks the link, they are presented with a log in screen where they can input their username and password. If this is approved, they are then presented with order status (2.1.1.14).
A table should be presented to the user with information about: order number, basket number, item information, and status (approved/rejected). The line items should be sorted to reflect the specific order that the user had been e-mailed about. Each line item should also provide for an option to view a history of status tracking (2.1.1.15).
The status tracking should contain line-by-line information detailing the status, date, and possibly where in the chain the item may be (received by supplier, item not in stock, and so on).
The system should prompt the user for some unique identifier to assign to their order so that the order can be distinguished from other saved orders. Once this is completed, the user should be presented with new order (2.1.1.3).
The user should be presented with all possible orders/baskets that may have been saved by the user. Once the user selects a saved order/basket, they should then be presented with view basket (2.1.1.6).
If the approver rejects the order, the order is not entered into the system, but e-mail status to Requisitioner (2.1.1.12) occurs.
The user is first prompted with a page to input, as an example, their Responsibility Code(s) and Account Number. This is validated against the back end financial system, and if approved, the user is presented with new order (2.1.1.3).
In the above diagram only one instance is mentioned in which the electronic procurement system has to physically access the financial system—stage 2.1.1.11. All other instances can be handled in an ‘offline’ mode. For example, in step 2.1.1.19, the procurement system validates that the responsibility codes and account codes are correct. Instead of placing unnecessary load on the financial system, the information could be stored in an SQL-based server database, and a component could be written to validate the codes every hour or whenever required.
Similar questions should be asked for leveraging other existing systems:
Now that we have touched briefly on some of the functionality the users/requisitioners may be expecting, we should be aware as to the integration requirements we require into the financial system. We should also be able to begin looking deeper into some of the specific integration issues.
As an example, if we used Walker Financials to demonstrate the functionality a financial system vendor provides, we should be able to gauge what is and what is not possible today.
ActiveX Data Objects to access Walker
The two ActiveX Data Objects (ADOs) required to access Tamaris (the Walker Financial system displayed on the right of the above diagram) are the Connection object and the Recordset object. To open a connection, the programmer builds a connection string and calls the Open method of the Connection object passing it the connection string:
Dim oCon as New ADODB.Connection
oCon.Open "Provider=XEOLEDB;COCODE=SC;CTRLENT=CUBE", "IVP", "USER"Note that the Company Code and Control Entity are defined in the connection string.
Once the Connection has been opened, opening the TABLES Schema Table can access a list of available services:
Dim oRSTables as ADODB.Recordset
Set oRSTables = oCon.OpenSchem adSchemaTables,
The service name is in the field “TABLE_NAME”.
To Execute a service, the programmer first opens a Recordset for that service. To open the WJOURNAL recordset:
Dim varRecsAffected as Variant
Dim oRSJournal as ADODB.Recordset
Set oRSJournal = oCon.Execute “WJOURNAL” , varRecsAffected, adCmdTableDirect
Note the use of adCmdTableDirect and not adCmdTable.
This returns a Recordset with a single record. All fields are blank. At this point data can be entered directly into the fields:
oRSJournal.Fields (“CreditAmount”) = 1000.65
After all values have been set as required, they are sent to the mainframe via a call to UpdateBatch:
oRSJournal.UpdateBatch adAffectAll
UpdateBatch is a Synchronous blocking call that sends the value via the COM Transaction Integrator (COMTI) to CICS. Once the service has completed the function returns, any Diagnostic messages are returned in a Recordset which is accessible from the “Diagnostics” field:
Dim oRSDiagnostics as ADODB.Recordset
Set oRSDiagnostics = oRSJournal.Fields(“Diagnostics”)
The Diagnostics Rowsets contain three columns. They are the Diagnostic Code, the Diagnostic message as text, and Columns associated with the diagnostic referenced by column number:
Dim strDiagnostic as String
Dim nColumn as Integer
strDiagnostic = oRSDiagnostics.Fields(“Text“)
nColumn = oRSDiagnostics.Fields(“Column“)
txtDisplay = “Message = “ & strDiagnostic & “ Column = “ & _
oRSJournal.Fields(nColumn).Name
XEOLEDB uses the concept of hierarchical recordsets to execute two special types of services, called Fill and Iterating Services. A Fill service provides Fetch functionality. Iterating services execute the same service multiple times with successive records of Data:
Dim oRSJournalDetails as ADODB.Recordset
Set oRSJournalDetails = oRSJournal.Fields(“Details”)
In the case of running an Iterating service, there is one blank record at the start. Each additional record must be added using AddNew:
oRSJournalDetails.AddNew
Set the desired number of Detail Records in Field IterCnt. This must be specified explicitly; or else the iterating portion will be ignored.
For a fill rowset, set the minimum amount of information to limit the fill (partial key) and call Update Batch. When the function returns there will be one record in the Details rowset for each iteration of the service. The Number of Detail Records is stored in the Field IterCnt.
In addition to the standard SNA Configuration to talk via a 3270 session, COMTI requires Advanced Peer to Peer Communications (APPC, also referred to as LU6.2). The configuration of APPC is beyond the scope of this document. However, there are some aspects of APPC configuration that are required to set up COMTI to work with XEOLEDB. First, within the COMTI Console (a Microsoft Management Console Based application shipped along with COMTI) configure a remote environment to talk to the CICS APLID for TamarisXE. These settings are the same as used by the Gateway in the WalkLink infrastructure: Local LU alias, Remote LU Alias, and Mode name will all be the same as for a Gateway connection. In addition, the CICS Mirror TP should be set to CSMI.
Once Remote environment configuration is complete use the console to associate the COM TI object Walker.MessageListener.1 with the remote environment. This can be done via drag-and-drop from the Unassigned components. If the component is not visible, it has not been registered on the machine. The component resides within the DLL MsgLstnr.tlb.
Tracing of XEOLEDB provider is enabled using an ActiveX control called XELogCtrl. Add this control to an application or run from the ActiveX control Test container. It allows you to specify Tracing based on the service name, the User ID, and a Trace ID. Trace output is stored in a file specified by the Trace Id (appended with .log) in a directory specified in the Registry value Trace Directory in the Registry Key HKEY_LOCAL_MACHINE\Software\Walker\XEOLEDB.
Deciding where to place supplier information is a tough question. As an example, supplier information could be stored at the supplier’s site or this could be replicated and stored locally at the customer’s site. The following could be a problem with the first option: the dependence on an Internet connection between customer and supplier. If the line went down no one would be able to procure goods from suppliers.
The second option may make more sense but a new problem would be introduced translating information from the supplier’s catalog to the customer’s catalog. This could be achieved in a number of ways as discussed in the Exchanging Information: Mapping section. There may also be an issue with the updating of information, if product information were to change and not be reflected in the buyer’s catalog because an update had not occurred.
An alternative option could be storing some information locally and storing some additional information in the sellers catalog, such as images.
The most important questions to ask are, where does the suppliers’ information reside, how will you get access to it, and who will maintain it.
Site Server 3.0, Commerce Edition (SSCE) provides the ability to query data stores for product information, as well as create simple catalogs. SSCE does not provide catalog management tools for import and export. These either have to be developed manually or purchased from a third party ISV that specializes in these types of tools.
As an example, Saqqara (www.saqqara.com) has a solution that plugs on top of SSCE to provide a Database Catalog Management and Publishing solution that does not detract from any of the features that SSCE provides today.
Courtesy Saqqara Corporation
Step Search Professional (SSP) from Saqqara can be used as the catalog front end for SSCE order management functions to create a complete commerce solution. In other words, the user can locate a product using Step Search Professional, and then order the product using Site Server's functionality. Sections A and B in the Appendix (provided by the Saqqara Web site) describe how the integration between the two systems can be accomplished.
If you refer to the conceptual design diagram, some business transactions should be apparent. The procurement team should now define development requirements without reproducing what is currently available by SSCE.
As an example, a transaction may be required to pass order information onto the financial system for validation and be returned with an order number. This may only require a couple of fields to be validated, such as internal purchase codes and account codes.
You may also choose to have a component that retrieves information from the financial system (such as purchase codes), stores them locally, and periodically checks for updates or changes.
If the integration with the financial system is somewhat seamless and information can be retrieved, modified, and updated, then it should be fairly straightforward to develop these components and to add them to the OPP (Order Processing Pipeline).
Now would be a good time to take a look at the sample Microsoft Market (MSMarket) site that is provided by SSCE, to get a feel for some of the existing functionality that can be leveraged by building on top of MSMarket.
Throughout the design phase, it is important to keep the concept of automation in mind, but automation should not be the beginning and end all for the procurement project. Process integration is about defining what transactions/processes should be automated, what can be done in a synchronous/asynchronous mode, how much information should be created via a structured process, and so forth.
As an example, if a solution were being developed to house a supplier’s catalog at the organization's site, there may be a requirement to automate the transfer of data that is going to be stored in the organization's catalog. An application could be developed that takes a snapshot of the supplier’s catalog, encrypts, compresses, and transmits via FTP (as an example) to the customer's site. The customer may have an application that decrypts, un-compresses, and imports the data into the local catalog. Alternatively, the entire catalog could be housed at the supplier’s site and accessed via a series of stages in the OPP at the customer’s site.
While architecting the solution, you may have a requirement for reliable delivery of information between your organization and your suppliers. If this is the case, MSMQ (Microsoft Message Queue Server) could be used to reliably and asynchronously transfer this information.
In conventional application design, applications in general communicate to other applications directly. With Message Queuing, applications communicate directly with another Message Queue server, which queues the messages/requests and transmits them to another Message Queue server for processing.
Within SSCE, MSMQ is integrated as follows:
Developers develop ASP as normal, but instead of communicating directly with the supplier’s application, the ASP generates MSMQ messages to send to the supplier’s Message Queue server. In a procurement example, the user does not have to wait for the sending application to receive a message from the receiving application at the supplier’s side. Rather, the customer’s application pop-ups a response to the user informing them that the request is being processed.
As an example, users may log onto an Intranet corporate purchasing application and request some product to be purchased. Instead of the application passing this information on to the supplier(s), it may batch the requests for a period of time and then transmit the requests. The user could be updated as to the status changes via e-mail.
Companies that use Web servers other than IIS (Internet Information Server) can also send messages from CGI scripts and other programming mechanisms via MSMQ’s C and C++ APIs. Integration with non-Message Queue servers like IBM’s MQSeries is provided with SNA 4.0.
As organizations have experienced the issues with the Internet and the various protocols that run at an application or circuit level, giving access to all these protocols is generally unheard of. Most organizations today already have some investment in being able to transfer information between other organizations—not leveraging off these is unwise.
There are many different ways to approach the transport issue, but certain questions need to be asked in order to understand what can and can’t be achieved over a period of time.
In a traditional purchasing environment, where one company is purchasing goods from another, faxing as an unstructured mechanism for transporting information may be acceptable. If you were looking at a three-month project as a first phase to try and automate some of the processes within your environment, automation between suppliers may not be the most important. If solving existing back end integration issues are complex and time consuming, then you may want to steer away from complex automation between your suppliers in the first phase. For confirmation, the supplier could log onto a Web site, provided by the customer, to update purchase orders details. Faxing is an easy stepping-stone in bridging the communication gap, but there are some issues.
Information that is transmitted to the supplier via a fax has to be captured in some way into an existing system. This means that the suppliers would have to employ one, two, or more resources just to capture the faxed information. Some OCR (Optical Character Recognition) could be used, but would only be effective if the information stored on the fax is clear and decipherable.
If the supplier has a mechanism of communicating via e-mail, this could be used as an alternative. The customer could develop an application in SSC that would take the contents of an order form and submit it via e-mail to the supplier. The supplier may have an individual that receives the purchase orders, manually processes the purchase orders, and then updates the customer's Web site with the information.
The supplier may choose to develop some automation into their existing systems so that e-mails that are received are processed ‘automatically.’ Provided the information sent by the customer is in an agreed upon format, the supplier could develop a receiving agent that queries the specific mailbox that received these purchase orders. The application could extract the relevant information and start new processes to begin the ordering. To see an example from the SSCE documentation regarding "Creating a Receive Agent," see section C in the Appendix.
The supplier may choose to update the customer whenever something occurs, for example, when the purchase order is received, the receiving agent may generate a receive confirmation and send a message to the customer's procurement system. The customer's procurement system could also have a similar receiving agent that could monitor a mailbox, extract relevant information, and then update the relevant purchase order with a new status type.
Other transport mechanisms could be used. A custom CIP application could be developed to talk to another CIP application or non-CIP application either via a standards-based protocol: HTTP, FTP, SMTP, MSMQ, DCOM (Distributed Component Object Model), EDI or via some proprietary mechanism if it were required. EDI VAN’s could also be leveraged as a transport mechanism.
The following components are provided with SSCE:
Probably one of the most important things to consider within your architecture is the issue around message translation or mapping of information.
As an example, if you want to purchase a book from a book vendor, it would be pretty straightforward to develop catalog and payment integration as discussed, so that supplier information could be interrogated and a product purchased. The same goes for two or more book suppliers. But what happens if you want to simultaneously query/interrogate two book supplier's catalogs and return the cheapest price of the two?
Potentially, you would have to run a query against the first supplier and return a result. You would then run a query against the second supplier and return a result. Then you would query the two values and find the lesser of the two values to display to the user.
If you had the catalogs stored locally, performing this type of query wouldn't be too difficult. But what if the supplier hosted these catalogs?
You would need to agree upon a format in which to query each supplier’s catalog, develop a parser that would be able to take a query from the procurement system, and then run a similar query against the respective supplier's catalog. Since the tables that each supplier maintains and the mechanism for querying the catalog is probably different, you would have to know how to pass the query onto the supplier, receive the response, and re-interpret the response so that the procurement system is able to understand the format of the value(s).
Extensible Markup Language (XML) can be used to provide this kind of functionality. The SSCE documentation makes the following mention of how to use XML for mapping:
The MapToXML component, which usually appears in the Map stage of a Commerce Interchange Transmit pipeline, reads the data stored in an object referenced by the name/value pair in the Transport Dictionary specified by the Object Source Key field. MapToXML converts this data to an XML stream or to an XML-enclosed binary stream. MapToXML then writes the resulting stream to the name/value pair in the Transport Dictionary specified by Results XML Key.
Object Source Key. Specifies the name of a field on the Transport Dictionary from which the component reads the business data object.
Results XML Key. Specifies the name of a field on the Transport Dictionary to which the component writes the XML-formatted business data object.
Preferred Data Format. Specifies the preferred format into which the component translates the business data object.
XML Tags. Specifies that the entire business data object is to be translated into a pure XML stream.
Encoded Binary. Specifies that the binary version of the business data object is to be wrapped in XML tags.
If the object to be read implements IPersistXML or IPersistStreamInit, the output produced by MapToXML is:
<MAPXML><VALUE dt:dt="object" dt:classid="classid:classid">
<ObjectTag>
Object Data
</ObjectTag>
</VALUE></MAPXML>
Or, in the case of encoded binary format, in the following form:
<MAPXML><VALUE dt:dt="object" dt:classid="classid:classid" dt:binary="1">
Binary Object Data
</VALUE></MAPXML>
The value 1 (one) following binary indicates that the result was mapped using the object’s IPersistStream interface; a value of 2 (two) indicates that the result was mapped using the object’s IPersistStreamInit interface.
Suppose a Dictionary object is created, as follows:
Set myObj = Server.CreateObject("Commerce.Dictionary")
myObj.Company = "Microsoft"
When this dictionary is passed to the MapToXML component, the component produces the following output, when the output is specified as XML tags:
<MAPXML><VALUE dt:dt="object" dt:classid="classid:B7990D09-45FD-11D0-8176-00A0C90A90C7">
<DICTIONARY>
<DICTITEM key="Company"><VALUE dt:dt="string" xml-space="preserve">Microsoft</VALUE></DICTITEM>
</DICTIONARY>
</VALUE></MAPXML>
A SimpleList object created as follows:
Set myObj = Server.CreateObject("Commerce.SimpleList")
Call myObj.Add("Microsoft")
Produces the following result when mapped to XML tags:
<MAPXML><VALUE dt:dt="object" dt:classid="classid:B7990D0E-45FD-11D0-8176-00A0C90A90C7">
<SIMPLELIST>
<LISTITEM><VALUE dt:dt="string" xml-space="preserve">Microsoft</VALUE></LISTITEM>
</SIMPLELIST>
</VALUE></MAPXML>
The following example shows an order form that has been converted to XML by the MapToXML component. The example was produced by ordering a laser printer cartridge in the MSMarket sample site.
The XML output reveals the structure of the order form object (the classID of which is 713DD5B1-747C-11D0-BE8A-00A0C90DC855). It contains a dictionary, which in turn contains name value pairs associated with the order. One of these values, Items, contains a SimpleList object, which consists of a list of the items ordered (in this case, only one). The list item, in turn, contains another dictionary, which contains values describing the item ordered (SKU, Quantity, and so on).
<MAPXML><VALUE dt:dt="object" dt:classid="classid:713DD5B1-747C-11D0-BE8A-00A0C90DC855">
<DICTIONARY>
<DICTITEM key="Items"><VALUE dt:dt="object" dt:classid="classid:B7990D0E-45FD-11D0-8176-00A0C90A90C7">
<SIMPLELIST>
<LISTITEM><VALUE dt:dt="object" dt:classid="classid:B7990D09-45FD-11D0-8176-00A0C90A90C7">
<DICTIONARY>
<DICTITEM key="SKU"><VALUE dt:dt="string" xml-space="preserve">OPPRINTER1001605</VALUE></DICTITEM>
<DICTITEM key="Quantity"><VALUE dt:dt="i4">1</VALUE></DICTITEM>
<DICTITEM key="insertby"><VALUE dt:dt="string" xml-space="preserve">JoeStart</VALUE></DICTITEM>
<DICTITEM key="ReqClassCode"><VALUE dt:dt="string" xml-space="preserve">OP</VALUE></DICTITEM>
<DICTITEM key="ISOCurrencyCode"><VALUE dt:dt="string" xml-space="preserve">USD</VALUE></DICTITEM>
<DICTITEM key="RequisitionID"><VALUE dt:dt="i4">10004</VALUE></DICTITEM>
<DICTITEM key="PartNumber"><VALUE dt:dt="string" xml-space="preserve">1001605</VALUE></DICTITEM>
<DICTITEM key="PartClassCode"><VALUE dt:dt="string" xml-space="preserve">PRINTER</VALUE></DICTITEM>
<DICTITEM key="Partdescription"><VALUE dt:dt="string" xml-space="preserve">Print Cartridge GGG Printer Black</VALUE></DICTITEM>
<DICTITEM key="isstandardpart"><VALUE dt:dt="string" xml-space="preserve">Y</VALUE></DICTITEM>
<DICTITEM key="UnitCode"><VALUE dt:dt="string" xml-space="preserve">EA</VALUE></DICTITEM>
<DICTITEM key="list_price"><VALUE dt:dt="string" xml-space="preserve">2519</VALUE></DICTITEM>
<DICTITEM key="AccountCode"><VALUE dt:dt="string" xml-space="preserve">728002</VALUE></DICTITEM>
<DICTITEM key="InternalOrder"><VALUE dt:dt="string" xml-space="preserve">1</VALUE></DICTITEM>
<DICTITEM key="placed_price"><VALUE dt:dt="i4">2519</VALUE></DICTITEM>
</DICTIONARY>
</VALUE></LISTITEM>
</SIMPLELIST>
</VALUE></DICTITEM>
<DICTITEM key="RequisitionID"><VALUE dt:dt="i4">10004</VALUE></DICTITEM>
<DICTITEM key="ReqStatusID"><VALUE dt:dt="i2">2</VALUE></DICTITEM>
<DICTITEM key="insertby"><VALUE dt:dt="string" xml-space="preserve">JoeStart</VALUE></DICTITEM>
<DICTITEM key="DateCreated"><VALUE dt:dt="dateTime.iso8601">19980206T11:34:34000</VALUE></DICTITEM>
<DICTITEM key="DateChanged"><VALUE dt:dt="dateTime.iso8601">19980206T11:34:43000</VALUE></DICTITEM>
<DICTITEM key="ReqClassCode"><VALUE dt:dt="string" xml-space="preserve">OP</VALUE></DICTITEM>
<DICTITEM key="ISOCurrencyCode"><VALUE dt:dt="string" xml-space="preserve">USD</VALUE></DICTITEM>
<DICTITEM key="VendorNumber"><VALUE dt:dt="string" xml-space="preserve">0002001243</VALUE></DICTITEM>
<DICTITEM key="locale"><VALUE dt:dt="i4">1033</VALUE></DICTITEM>
<DICTITEM key="order_id"><VALUE dt:dt="string" xml-space="preserve">DJUH21SKKES12MHD00L1R0D8N7</VALUE></DICTITEM>
<DICTITEM key="ReqName"><VALUE dt:dt="string" xml-space="preserve">Joe Start</VALUE></DICTITEM>
<DICTITEM key="ReqEmailalias"><VALUE dt:dt="string" xml-space="preserve">JoeStart</VALUE></DICTITEM>
<DICTITEM key="ApprovalManagerAlias"><VALUE dt:dt="string" xml-space="preserve">MaryJJJ</VALUE></DICTITEM>
<DICTITEM key="ReqPhone"><VALUE dt:dt="string" xml-space="preserve">454-9894</VALUE></DICTITEM>
<DICTITEM key="TransmissionRoom"><VALUE dt:dt="string" xml-space="preserve">2343</VALUE></DICTITEM>
<DICTITEM key="TransmissionLocation"><VALUE dt:dt="string" xml-space="preserve">100</VALUE></DICTITEM>
<DICTITEM key="TransmissionLine1"><VALUE dt:dt="string" xml-space="preserve">Joe Start</VALUE></DICTITEM>
<DICTITEM key="TotalAtSubmit"><VALUE dt:dt="i4">2519</VALUE></DICTITEM>
<DICTITEM key="needapproval"><VALUE dt:dt="boolean">0</VALUE></DICTITEM>
</DICTIONARY>
</VALUE></MAPXML
Alternatively, you could use an EDI translator for mapping. There is a component called MakePO that is used to generate forms, such as purchase orders (PO), from an Active Server Pages template file and from information on the OrderForm. In order to demonstrate how to create EDI messages, a sample template is provided with the installation of the SSCE SDK that can be used with MakePO to generate an ANSI X12 850 message, which is a purchase order (commonly referred to as an 850.)
The template is contained in the file PO850Template.txt, which can be found in \Microsoft Site Server\SiteServer\Commerce\SDK\Commerce\Samples and is also displayed in Appendix D.
The 850 template can be used with the MakePO component in the MSMarket SubmitViaInterchange pipeline, which is used to send orders for office supplies over the Commerce Interchange Pipeline. The template may also be used with the MakePO component in the Map stage of the MSMarket InterchangeTransmit.pcf pipeline.
The 850 template for MakePO is compatible only with MSMarket. It can be used in the Purchase Order Submit stage of the SubmitViaInterchange pipeline or in the Map stage of the InterchangeTransmit pipeline in MSMarket.
If your architecture required all mapping to be done via EDI, it is recommended to purchase an EDI Translator from the one of the many EDI Translator ISV’s that have built EDI translators that integrate within SSCE.
As the project team starts to better understand what is required from the procurement solution, and small proof-of-concepts are built to understand the process, security questions will most likely be raised.
It is important to understand what information is considered sensitive, and define exactly what needs to be secured. Everything can be secured or nothing secured, depending on the requirements.
As an example, if you were building a solution that communicated to your number one supplier (office supplies as an example), you may choose to create a VPN (Virtual Private Network) between each other. The Windows NT-based platform provides all the facilities to create a secure solution. In this scenario, PPTP (Point-to-Point Tunnelling Protocol) could be used to create a VPN between the customer and the supplier.
Users could be authenticated, data could be encrypted, and multiple protocols could be used. The above is an example of a "protect everything" approach.
Alternatively, communication to the supplier may be done via HTTP. Either all communications via the CIP could be in the clear, or upon connection or transfer of sensitive information, SSL (Secure Sockets Layer) could be introduced to encrypt the contents sent over the wire. During the SSL handshake, a unique encrypted session key is generated by the transmitting Web server, for bulk encryption of information, and transferred to the receiving Web server securely via a public-key operation so that information is transferred securely.
Certificates, which are required for the above operation, could either be created by a third-party CA (Certificate Authority) or by using Microsoft Certificate Server 1.0.
Another option might be to encrypt the contents and send the encrypted package in the clear. The PO (Purchase Order) could be sent as a file attachment and encrypted before transmitting. Or the PO could be sent via an e-mail that could be digitally signed (to prove that the information has not been tampered with) or digitally encrypted (so that no one can read the contents unless they know the secret key) or both.
SSCE provides the components which can be used for the encrypting and signing of information. These components are:
DigitalSig usually appears in the Digitally Sign Stage of a Commerce Interchange Transmit pipeline, digitally signs the message. The component creates a message digest using the hash function and then encrypts the digest using a certificate. The component encodes both the signature and the message or only the signature, and then writes the encoded data to the Transport Dictionary.
EncodeMIME usually appears in the Add Header Stage of a Commerce Interchange Transmit pipeline, encodes the business data object into Quoted printable or Base 64, wraps it in MIME, and then writes it back into the Transport Dictionary.
EncodeSMIME usually appears in the Digitally Sign Stage or the Encrypt Stage of a Commerce Interchange Transmit pipeline, encapsulates the business data object in S/MIME (secure MIME) format for transmission in e-mail using an SMTP mail server. The EncodeSMIME component encodes the object using IETF EDIINT draft Internet standard encoding, and then writes it back into the Transport Dictionary.
EncryptPKCS usually appears in the Encrypt Stage of a Commerce Interchange Transmit pipeline, encrypts a business data object using the Public Key Crypto System (PKCS) security standard.
For information to be encrypted securely, you may want to import the supplier’s certificate so that information can be encrypted with the supplier’s public key and decrypted only by the supplier using their private key. You can use the CIP Manager to manage the relationship with the respective suppliers by importing certificates and defining the levels of trust.
The image below portrays the stages that could be required for this type of operation:
The previous sections discuss how to build integration into back end financial systems, how to build integration into suppliers’ catalogs, how to transport orders, how to translate information, how to protect the transport of information, and so on. But one thing that hasn’t been covered yet, is how to leverage existing systems for collaborative tasks.
For example, a customer generates a request for a product from their internal procurement system. That request is processed and at some stage, is passed on to the supplier for fulfilling. But the inbetween steps of approval and status updates still need to occur.
For approval, manager/authorizer/approver information could be retrieved either from an existing e-mail directory, such as Microsoft Exchange Server, from an existing database that may house this information, or from a financial system that may contain authorizer e-mail addresses relating to internal purchasing codes or departments.
The MSMarket sample demonstrates an approval process based on information retrieved from an ODBC database, but this could be modified to perform a look-up against Microsoft Exchange Server using CDO or against a financial system. The username and password validation could either be validated against the MSMarket database (default) or using Windows NT Challenge/Response, which would have to be switched on in MSMarket Manager. For Windows NT Challenge/Response, the REMOTE_USER HTTP header value would be used to do a look-up against the MSMarket database to retrieve additional information (approver email, and so on) about the requisitioner.
The MSMarket SubmitViaInterchange Pipeline as displayed above, provides a stage that performs a look-up to verify the approval status a user may have. In this example, this is based on a monetary value that is assigned to an individual user. If the value for the purchase order is greater than the approval status amount, then an email is sent to the manager for approval. See Appendix E for an example of approval.vbs.
Another example of collaboration may be the requirement to have status updates provided to requisitioners, so they can constantly view where in the process their order may be.
When the supplier receives a purchase order, there may be an automated process to trigger a receive confirmation as discussed in 2.1.5. But there may not be a full process to provide all status updates throughout the flow from supplier courier customer.
A custom interface could be developed to provide the ability to update purchase order status information at the customer’s site. Or some automated process could be developed at the supplier’s site to provide this kind of functionality.
Throughout the design process you will find that most of the component services being developed run in-house. Meaning, if a product look-up occurs, the component for the look-up is housed locally, and the catalog in which the product information is contained could either be housed locally or at a remote supplier’s site.
This is a great approach if you have the resources, skills, and funds available, and provided your supplier wants to provide integration into their system which may already be enabled via some other means which might not be easy to integrate with your solution.
In the above scenario, the component developed could run locally, but the resource that requires accessing could be housed remotely. This way, outsourcing a specific service like cataloging.
A good example of this might be EFT (Electronic Funds Transfer), so that you can automate the payment process to your suppliers. Instead of issuing checks when goods arrive, you may choose to electronically transfer the respective amount. Although you would probably have to develop a component(s) that would talk to the EFT system, the back end process would not be maintained by you.
Alternatively, you may find doing business with multiple suppliers, managing multiple message translation mechanisms, and message transport mechanisms a costly exercise.
You could potentially look at outsourcing this entire process so that you, as a customer, don’t have to worry about managing multiple message translation mechanisms and message transport mechanisms.
A third-party company could potentially deal with all of this for you, since they may already have all the processes in place to conduct business with existing suppliers. As an organization, you would be able to leverage off this without having to build the solution from the beginning.
For a proof-of-concept configuration, SSCE should be installed on a dedicated computer, and the databases installed on a dedicated SQL Server-based computer. This implementation is chosen to meet performance requirements and for flexibility. SSCE administrative tools can be installed on a dedicated administrative (client) computer, to further meet performance requirements and flexibility. Alternatively, all of the above can be installed on one machine, for testing purposes.
| Hardware | Site Server | SQL Server | Client Machine |
| *Either an Intel Pentium 100 MHz or faster processor or a Digital Equipment Corporation (DEC) Alpha processor (Intel Pentium 166 MHz recommended) | X | X | X |
| Network Adapter Card | X | X | X |
| CD-ROM drive | X | X | X |
| VGA or Super VGA monitor compatible with Windows NT Server version 4.0 set to 1024 x 768 | X | X | X |
| Microsoft® Mouse or compatible pointing device | X | X | X |
| Memory | 64MB (128 recommended) | 64MB (128 recommended) | 32MB (64 recommended) |
| 1 GB (2 GB recommended) of disk drive space for the platform software as well as: | Database (100MB per 10K users) | P&M server components | P&M tools |
*Refer to the Windows NT Hardware Compatibility List for the processor architecture that corresponds to the platform you are installing. For example, if you are installing SQL Server for Intel as the database on your Membership Directory computer, you can choose any of the 32-bit Intel x86-based microprocessors, such as the 80486 or the Pentium.
The following table depicts the required software that must be installed on each computer in this configuration before installing P&M.
| Software | Site Server | SQL Server | Client Machine |
| Microsoft® Windows NT® Server 4.0 | X | X | optional |
| Windows NT Workstation version 4.0 | optional | ||
| *Microsoft® Windows® 95 or 98 | optional | ||
| Windows NT 4.0 Service Pack 3 | X | X | X (with Windows NT) |
| Windows NT file system (NTFS) | X | X | optional |
| *Microsoft® SQL Server™ version 6.5 with TCP/IP and Named Pipes on | X | ||
| SQL Server Service Pack 4 | X | ||
| SQL Hotfix 297 | X | ||
| Microsoft® Internet Explorer 4.01SP1 | X | X | |
| Windows NT 4.0 Option Pack | X | ||
| Site Server 3.0 | X | ||
| Site Server 3.0 Commerce Edition | X | ||
| Windows NT 4.0 Service Pack 4 and Y2K Update | X | X | |
| MDAC 2.0 SP2 | X | X | |
| Site Server 3.0 SP2 | X | ||
| Microsoft SQL Server Service Pack 5a (without MDAC 2.1) | X |
*SQL Server configured this way is required in this particular network environment. Microsoft SQL 7.0 can be used. Please read the SSCE SP2 read me.
In order for the Site Server, SQL Server and client computers to communicate, they must have the following connectivity settings:
The following tables depict the protocols and ports used by Site Server components. Any routers, filters, or firewalls you install between the computers in your network must use the security protocols and port settings defined by these tables.
| Site Server Component | Communicates With | Protocol Used |
| LDAP Service | SQL Server | OLEDB (ODBC) |
| Authentication Service | LDAP Service | LDAP |
| AUO | LDAP Service SQL Server ADSI |
LDAP ODBC ADSI |
| Sample membership site | Web browser | HTTP |
| Web Administration (Web Admin.) |
Web Server LDAP Service |
HTTP, HTTPS LDAP |
| Authentication Service | Membership Directory | LDAP |
| Design Time Controls (DTCs) | ADSI LDAP Service Web browser |
ADSI LDAP HTTP |
| Membership Directory Manager (MDM) | LDAP Service | LDAP |
| P&M Service Administration | LDAP Service Web Server, remote servers |
LDAP HTTP, HTTPS, DCOM, ADSI |
| Site Server Documentation | Web browser | HTTP |
| Web Server | Web browser Admin. Objects (.asp) |
HTTP, HTTPS LDAP, DCOM, ADSI, ODBC |
The following table shows the default Site Server port numbers to run your protocols; however, these can be changed depending on what protocols your company is using, and which ports are already being utilized.
| Protocol | Default Port # |
| HTTP | 80 |
| HTTPS | 443 |
| SMTP | 25 |
| **DCOM | 135 (for SCM) + Dynamic (>1024) |
| ODBC | 1433 |
| LDAP | 1002* |
| LDAP (SSL) | 636 |
*LDAP running on port 389 is a standard used by many different applications and is therefore left available when the Site Server components are installed. The default port for the Site Server LDAP Service is 1001 plus the instance number of the Membership Server the LDAP is associated with.
**DCOM uses port 135 for Service Control Manager and assigns one dynamic port per server process, between the range of 1024 and 65535.
Solution architects often approach the design of a Web-based direct selling commerce application by focusing on two elements:
These are important design elements but there are some common architectural features of a direct selling solution that are often overlooked, or not well specified, thus increasing project risk.
Unfortunately for Web merchants, only a fraction of all traffic in a direct selling solution will be devoted to actually buying something. While it is vital that the ordering process works reliably, until some other components are defined and understood, the risk of losing orders is considerable.
Fundamentally, solution architects should approach their designs assuming that the site must interoperate with existing business processes and systems. The first question a solution architect should ask is, “What’s out there that I can use?” Unless this is a completely new business, there’s probably going to be an order entry system in place. There’s a General Ledger. There may already be considerable business intelligence in existing systems for payment processing, taxation, sales analysis, marketing, catalog management, and fulfillment. While electronic commerce and the Web certainly provide organizations an opportunity to re-engineer these existing processes, time-to-market pressures in Web selling more often than not win out over such fundamental change.
A comprehensive specification for an electronic commerce site should include a solid plan covering each of the following architectural areas.
Solution architects often start by asking, “How many servers do I need?” Yet it is difficult to ensure success without having some idea of the target load and performance. The best way to approximate this is to build a profile. There are two aspects to the profile: a site profile and a user profile.
A site profile attempts to estimate metrics such as:
This certainly isn’t a complete list—the idea is to get some broad idea of the expected load.
In addition, the user profile attempts to build a set of usage scenarios to answer the question, “What’s the typical user session going to be like?” The Microsoft Site Server 3.0 Performance Kit, available on the Web at http://www.microsoft.com/siteserver/commerce/30/downloads/VCTurbo.htm, has many sample questions and metrics that can help you determine your user profile, as well as some sample user scenarios and profiles.
These shopping/browsing scenarios are user session-based, rather than focusing solely on a metric, such as hits/day. For example, a user profile scenario might describe the traffic produced by a typical user over a twenty-minute session. Articulating these scenarios will help the development team design test cases and performance benchmarks.
The profile does not automatically lead to a definitive count of the number of servers one needs to buy. However, no scaling assumptions in the solution architecture make much sense until they are stacked up against some idea of the expected load.
Catalog management is the most overlooked architectural area in direct selling sites. In all but the smallest stores, catalog management touches so much of the underlying business that it should probably be treated as an entire application on its own. By catalog management, we are referring to the process of getting the list of products and associated details (including pricing, pictures, specials, inventory levels, as well as a flexible searching capability) onto the Web.
The key catalog challenges that often surprise architects are:
The pricing may be in one system, the details in another. The existing catalogs may exist only partially in electronic form; vital information may be in the art department on paste-up boards. There may be a special catalog that feeds from Steve in marketing’s old FoxPro® database that you never knew about when you first met the customer.
Many catalogs have been built with telesales or print as the intended delivery media. The Web poses many new challenges. For example, the existing product description may be all-uppercase text with cryptic, encoded words previously intended only for telesales operators. Some products may have graphic images, and some may not (products without pictures do not sell on the Web). Some product catalogs have many items that should not be displayed on the Web; many times complex rules are required to filter them out. Finally, some catalogs may simply not have enough information for a Web buyer to make a coherent buying decision, such as products with many parts and complex configuration rules that were previously known only to sales personnel.
Many of the source systems that might feed a Web catalog update at different intervals. Inventory updates may occur at one frequency while pricing changes occur at another. A price change might make it through the source systems quickly while changes to the box shot (picture) occur rarely. The marketing department (and often, there’s a separate and new department called “Web marketing”) rightly believes that they need to update the site, particularly the “specials” or homepage items, on an immediate basis. How does that reconcile with the existing system’s schedules?
Although the existing catalogs are often disjointed and messy, the Web demands a very accessible and stable catalog that can be rapidly updated. The process of assembling a suitable Web catalog is often akin to a data warehousing effort, complete with source systems, data cleansing, and transformation along with an entirely new, neutral data model designed for the Web catalog.
Some of the pertinent architectural decisions involve answers to the following:
Once these kinds of questions can be answered, the architect can begin to make a buy or build decision. It's possible that third party catalog management and/or configurator solutions, such as those available from Mercado Software (http://www.mercadsoftware.com) or Saqqara Systems (http://www.saqqara.com), as described in section 2.1.2 above, will meet the need. Tools such as these should be evaluated first.
If the catalog maintenance process requires integration with existing systems, then a product like Microsoft SQL Server ™ 7.0, with the graphical Data Transformation Services tools, can be used to design the catalog maintenance process and schedule regular update jobs.
For building search capabilities that unify disparate catalog data, Microsoft Site Server 3.0 Search is invaluable. Site Server Search can be configured to build catalogs from disparate data sources through ODBC queries. The resulting catalog can be automatically propagated to one or more servers that handle user queries. A very powerful COM component can be used to query the catalogs and sort results.
Catalogs can certainly become complex. Solution architects should be aware of these potential risk areas and plan accordingly.
Every site that takes orders needs to identify customers in some way. The “authentication model” refers to the specification for how the users are identified and tracked. This is only partially a technical issue.
The user authentication model obviously has broad implications for the rest of the design. For example, a site that needs to authenticate each user via SSL encryption before allowing access is going to have a different performance characteristic than an open, “anonymous” site. A site that requires extensive personalization has a completely different design than a more static site.
Generally, some of the technical questions center on:
But beyond technical considerations lies some important architectural implications because existing businesses have many overlapping lists of customers.
For this paper, transaction services shall refer to the process of getting a valid order to its ultimate destination. In many ways, this is the easiest part of the commerce solution, but architects tend to spend a disproportionate amount of time designing order transactions when they should more properly be focusing on the catalog.
The major architectural point to consider is: Does the business have an existing order entry structure that I can utilize?
Many commerce solutions today take orders from the Web and drop them into another system. This is often an excellent risk-mitigation strategy as considerable effort and money has been spent building the business logic into these existing systems in order to handle order management and fulfillment. There are many technologies for getting the order from the Web into the system, which basically involve around synchronous or asynchronous approaches.
The synchronous approaches involve ODBC/OLE DB database connections from the Web to the legacy systems, middleware integration technologies like the Microsoft COM Transaction Integrator, IPC/RPC approaches like sockets, or even terminal emulation. In this manner the Web transaction is entered directly into a legacy system.
Synchronous approaches tend to be exciting, and they certainly enable a commerce environment with much less "friction." But they aren't always the best idea. For example, many order entry systems are scaled and tuned for a certain amount of load. The Web may represent an unknown quantity that could adversely impact that load. Also, many existing transactional systems are just too slow for today's impatient Web buyer.
These cases may call for asynchronous approaches. Asynchronous techniques include queuing orders in a separate store and feeding them into the legacy system later, as well as interfacing through batch. Yes, batch! There's no crime in building a batch input from a Web order and feeding it to the legacy system in that manner. You may even win a few friends in the data center with this approach.
Some of the architectural questions to answer:
Closely related to the catalog management considerations mentioned above is the notion of specifying a publishing model for the commerce application. The publishing model is the design for how content will make it from its source into a displayable format on the Web.
In a direct selling Web application, most of the publishing concerns the catalog. The basic architectural question is: Where will the catalog live?
There are two high-level approaches:
Realistically, most site architects understand that to be successful they may need a hybrid of the two approaches.
For example, the typical shopping scenario involves an initial drill-down through product categories, followed by a browse of a fairly long list of products, followed by a final drill down into specific product detail. A site architect who has done their profile as described at the beginning of this section will begin to understand that the key to buying is to get the user to the detail page quickly and without incident. Therefore, perhaps they will optimize the initial drill downs by building static pages, but when the user clicks to get the detail, a database query may result.
One other architectural question that should be specified at this time is caching. Perhaps instead of static pages, large sections of the catalog should be loaded into the Web server's memory, perhaps into an array or some sort of dictionary object. This could have significant favorable performance impacts, but the publishing model has become more complex (as procedures need to be developed to load and refresh the in-memory cache).
Successful direct selling applications are characterized by a requirement to adapt to quickly changing market conditions. As such, there are two key drivers:
The site architect must therefore design administrative services in place that allow the site to be managed. This is not the same as the system administration activities associated
with monitoring throughput, security, IP addresses and the like, but rather the day-to-day business functions of the site, such as:
The Microsoft Site Server 3.0 Commerce Edition store-building wizards will construct a store with a set of "manager pages" that illustrate many of these capabilities. These manager pages may suffice, and are an excellent starting point for further customization. In addition, the Microsoft Site Server Commerce Edition Ad Manager application is an extremely powerful example of an administrative application, complete with source code that can be used out of the box or customized as necessary.
Yet many real-world scenarios will require customized administrative interfaces to handle the different user constituencies. For example, product managers may create catalog content. Sales personnel sign up customers and win banner ad campaigns. Marketing folks determine which products to feature.
There are two general approaches site architects take to accommodating these interests:
The first approach may seem unacceptable but quite often the Web marketing team has considerable HTML/script experience, so allowing very selected personnel access to the page source code may be efficient. The second approach, whereby the Web team applies all changes, is not scalable. If the online venture is at all successful the Web team will be quickly overwhelmed with cosmetic and marketing-oriented requests.
The best approach is to provide the proper level of self-service via applications assigned for each user.
In some cases this can take the form of a content authoring tool that shows one interface to site authors in order to collect content, but then "flows" that content into templates. Many architects have built their own homegrown publishing tools using Microsoft SQL Server™ as a content store. Interwoven (http://www.interwoven.com) is a third party example of a content publishing, workflow, and deployment tool.
In other cases, this could mean providing Microsoft® Visual InterDev™ with access permissions to the appropriate virtual root on a development server.
One major architectural point about these sorts of administrative tools is that if at all possible, Web administrative tools should be integrated with the rest of the business. For example, a customer management system that allows sales personnel to track prospects might also be used to provision a business-to-business Web site with custom pricing for that customer. To the salesperson, the process should be seamless.
Techniques for securing Web applications and servers on the Internet are well known and documented in many places; including the product documentation and Web sites such as http://www.microsoft.com/security.
Securing an online store involves protecting the system software and application content while making it accessible to customers. Because the Internet provides wide access to the information on your computers, it presents a significant security risk. A security plan needs to be developed that addresses how to provide security for the following areas:
Administration. The security foundation rests with the underlying platform; therefore, it is necessary to consider the security implications of your configuration. This includes not only the physical installation, but also the network configuration and administrative access.
Consideration should be given to the following questions:
Applications and content. Planning the security of your store in this context is defining the accessibility of Web content and applications to users. As you plan and classify content, determine the level of security that each type requires. Bear in mind that personalizing the store based on user attributes selectively distributes content, but does not protect it.
Most stores have public areas where any user can access content. This content generally doesn’t require an authentication method although Cookie Authentication is often employed. Protected content and applications are accessed through the configuration of authentication methods, from providing a password to requiring a client certificate. Access control also means determining the specific level of access granted or denied to an object, directory, file, or application, and including all groups or users that you want to be able to access that resource.
User data. Data security continues to loom large in the minds of customers so how do you emphasize the security of your site? It seems obvious that sensitive information is accessible only under a high-security authentication method such as client certificates or SSL technology and that data be isolated from the store and placed on a secure network. But it doesn’t end there. Sensitivity to user concerns by publishing your polices on the security and use of customer data will go a long way.
Some items to consider:
The point of this exercise is that regardless of how secure you’ve made your store, it is really only as secure as your customers perceive it to be; ignore their concerns at your own peril.
In order to determine the health and success of an online store, analysis should be approached from the perspective of every stakeholder in the organization. What this means is that IT, Finance, Sales, Operations, any department that gets touched by the online store, probably has resulting information needs aligned with their business goals. In addition, they also probably have reporting tools and databases that need to accept the data collected from the store. In most cases then, the question is “What’s already available?” From there, decisions can be made regarding any requirements for additional reporting and analysis tools.
Although not a comprehensive list, some typical information needs are:
The goal of analysis should be to provide the necessary information within the timeframe required to make sound business decisions. Collecting data will do no good if it comes too late in the decision-making process, so building the analysis framework should be done with the reporting deadlines in mind, whether they be daily, weekly, monthly, or ad-hoc.
Most of the technical and design advice found in the rest of this document applies equally well to direct selling commerce applications. The goal of this section of the paper was to touch on some of the architectural issues in direct selling which are not often understood completely during the specification stage of the project. After understanding the scope of these issues, solution architects will be better able to specify a technical architecture that will answer their questions about scalability. The next section examines scalability in more detail.
This section of the document outlines a roadmap for building a Site Server Commerce Edition 3.0 (SSCE, also known as Commerce Server) site that is highly scalable. The objective is to show how a site can be scaled with a target capable of supporting 100,000+ concurrent users with as few servers as possible using a number of scaling and architectural techniques. The end result is a server farm that is a highly scalable site that can grow well beyond its original design.
Please note that this document does not address techniques to achieve a high-availability site. Please refer to MCIS 2.0 Resource Kit, among others it includes a document addressing high-availability sites. Topics such as warm backups, transaction log shipping, SQL Server clustering, and so forth are explored in that document.
The following sections expand on several techniques of achieving high-scalability:
Note that the above techniques do not necessarily need to be performed in the order listed. Architectural improvements are best considered and designed early in the project's lifecycle, having an efficient architecture allows a site to be built and operated at a lower cost. Scaling by hardware vertically vs. horizontally is primarily a management vs. cost consideration. Scaling vertically allows site management to be simplified initially but at higher hardware costs. However, once capacity is reached horizontal scaling is required. Scaling horizontally allows capacity to be built at a low cost initially, and once management becomes sufficiently complex, vertical scaling is required.
A high performance and high scalability site lends to a deployment with fewer and consolidated servers. The concept of consolidated Web servers is attractive from the management standpoint compared to an unconsolidated server farm. However with modern management software such as HP OpenView, Tivoli, or CA Unicenter TNG, the complexity and cost of server farm management is greatly reduced.
Although attractive, there is danger in over-consolidating servers to the extreme. At the upper limits of high-scalability there are disadvantages to a heavily consolidated Web server farm such as achievable with midrange or mainframe class hardware. The section, Arguments Against Consolidated Web Servers, expands on arguments against over-consolidation of servers.
It is commonplace now to find servers with huge amounts of memory that can practically cache the entire site's content. Disk I/O throughput has also increased, along with NTFS file cache. This improves I/O throughput. N-way SMP hardware is also readily available in the marketplace. Thus resources are available in abundance to scale SSCE 3.0 vertically by the use of hardware.
Through performance and capacity planning benchmarks performed by Microsoft, SSCE 3.0 has been found to be characteristically CPU-bound. Its primary bottleneck was found to be CPU processing capacity. This is due to ASP page processing that is highly CPU intensive. To see benchmarks and details on server capacity and performance, refer to the Capacity and Performance Analyses white papers on Microsoft’s web site for Site Server 3.0 (http://www.microsoft.com/siteserver/site/DeployAdmin.htm).
One way to scale SSCE 3.0 vertically is to increase the processing power. This can be achieved by utilizing a higher-class processor. These range from Pentium II, Pentium III, and their Xeon derivatives (server-specific versions with large level II caches). SSCE 3.0 also runs on the Compaq/Digital Alpha class hardware platform that is on par with any Unix hardware platform. The net effect is that more users can be accommodated on a single server.
Windows NT 4.0 Server is readily capable of running on 4-way SMP hardware. Thus another way to address the vertical scalability is to run SSCE 3.0 on these 4-way SMP servers. It might be tempting to run SSCE 3.0 on an 8-way SMP hardware in order to increase throughput even more. However, SSCE 3.0 shows diminishing returns of performance on hardware with more than 2 processors. The aggregate throughput is higher but comes at the cost of diminishing returns on investment (i.e. per processor throughput is less on 4-way than on 2-way SMP hardware), higher aggregate throughput is achieved with an increase in cost.
The recommendation is to scale the hardware with processors up to 4-way SMP hardware and proceed further with horizontal or architectural scaling techniques. A large amount of memory helps the IIS/ASP server cache content and compiled script as well as NTFS disk cache buffers to improve throughput. Multiple SCSI disk controllers increase disk I/O throughput. Multiple 100mbps-network cards increase network I/O throughput. Optical fiber interconnects with ATM could also be considered, but can be costly.
The following diagram illustrates scaling from a single CPU server Pentium II class, to dual CPU server with a Pentium III Xeon class processor, and all the way to a 14 CPU server with an even higher class Alpha 21264 processor.
Figure 1. Scaling vertically
Scaling vertically helps the site consume less physical footprint initially. Further scaling is achieved by scaling horizontally.
Scaling horizontally means deploying additional computers running SSCE 3.0. SSCE 3.0 was built with this in mind; the components can readily be run and/or distributed across multiple servers.
With horizontal scaling comes the complexity of distributing load evenly across multiple servers. This is addressed by utilizing load-balancing techniques, among others: Windows Load Balancing Services (WLBS—OS software) and DNS round robin (network software). The benefit of load balancing is two fold; it provides redundancy of services and presents higher aggregated capacity by directing load to multiple servers.
There are application level considerations when employing load balancing. SSCE 3.0 Site Builder wizard by default generates load-balancing friendly sites by employing URL or cookie-based shopper identifications. The wizard also generates sites with session/state management located centrally on the commerce store database away from the ASP servers. This allows user requests to be directed to any available server at any given time without loss of user session or state. This requires that IIS's session management be disabled. The advantage is that without session variables, no resources are taken up to manage user sessions. If the application is coded from the ground-up, to effectively scale horizontally it is required that IIS’s session variables are not used, and that IIS’s session management is disabled.
In cases where an application has been coded with IIS session management (i.e. makes use of Session variables), load balancing can still be achieved by usage of WLBS. WLBS can be configured to direct traffic based on the client/source IP address. This has the net effect of sending a client back to the same destination server for each request. This provides the desired effect as Session variables are local to each server and the client will then see the correct set of variables and state. There are serious issues with this technique for some proxy server farm architectures that are detailed in the Architectural Improvements section.
Alternatively, session state can be maintained in the farm with load balancing using WLBS, provided dynamic LDAP replication (a feature provided by Site Server 3.0) is switched on. This enables all Web servers in the farm to maintain dynamic data that is used to store session information that expires, and information about users who are currently logged on. It includes user identifiers and Internet Protocol (IP) addresses, and may also contain per-user application information, such as items in a shopping cart or pages visited.
The following shows how each component of SSCE 3.0 is scaled horizontally:
Scale by adding more computers to function as ASP servers. Externally the servers are exposed with a common domain name with single virtual IP address mapping in DNS that actually maps to a load-balancing system. The load balancing system directs the traffic to multiple servers. Load balancing for ASP servers directs TCP connections to a server and keeps it directed to the same server until the TCP connection session is closed (an HTTP request is an example of a TCP connection).
Scale by adding more computers to function as LDAP servers. The same scheme as ASP servers are used. Externally the servers are exposed with a common domain name with single virtual IP address mapping in DNS that actually maps to a load-balancing system. The ASP server Personalization & Membership instances point to the common domain name. The load balancing system directs the traffic to multiple servers. Load balancing for LDAP servers directs TCP connections to a server and keeps it directed to the same server until the TCP connection session is closed.
Scales by partitioning the Membership Database and hosting each database partition on dedicated computers running Microsoft SQL Server. Note that partitioning the Membership Database requires advanced planning; the database must be partitioned before populating it with data (e.g. member objects, and so on). The partitioned databases can be run on a single computer initially and at a later stage, be placed on dedicated computers running Microsoft SQL Server. If the Membership Database has already been populated, a custom migration tool needs to be written to repopulate the partitioned Membership Database. In this scenario its best to start of with a new Membership Database and migrating objects from the existing Membership Directory.
The DBStorage database stores the shopping basket, orders, and receipts. It was not provided with a partitioning option as an out of the box feature. However, with minimal code consideration outlined in the next section (Architectural Improvement) this database can be partitioned and scaled horizontally.
The following diagram illustrates scaling horizontally; multiple IIS/ASP servers, multiple LDAP servers, partitioned Membership Databases, and Commerce Store Databases are utilized.
Figure 2. Scaling horizontally
Scaling horizontally helps the site build to higher capacity. Further scaling requires architectural improvements shown in the next section.
An architectural improvement revolves around making conscious decisions about how the application is built and deployed to improve the efficiency of the site. The basic idea is to separate workload based on load-factors, and dedicating servers specifically suited for each type of workload. This allows a server to execute lightweight operations with higher capacity serving a higher number of concurrent users per server, and to direct operations with heavier load-factor but smaller capacity requirements to a smaller number of servers.
Heavyweight content such as ASP server, MTS, and Commerce Pipeline component execution are served up on dedicated servers such that the entire bandwidth of the server is efficiently utilized and does not burden servicing of static HTML/GIF content requests.
Static HTML/GIF content request can be served up by IIS many times faster than processing an ASP page request. A server dedicated to serving up HTML/GIF content may be able to handle a lot more concurrent user requests as compared to a similar server dedicated to serving up ASP/Commerce Pipeline. (Benchmarks suggest the difference to be on the order of five times; however, actual performance will vary with the complexity of each site.)
Thus 80% of the traffic is serviced with computers at high capacity, while 20% of the other traffic is serviced with computers at lesser capacity. However as the 20% heavy duty operations traffic also represent a lesser concurrent number of users, there is a lower requirement for computing capacity (translates to lower count of computers, or lower specification computers). The net effect is that fewer computers are required to serve up the site.
There are many places where this scheme can be applied, such as static content (HTML/GIFs) vs. dynamic content (ASP/Commerce Pipeline), business rules (MTS components), disk I/O (cache most active files), and so on. This section does not go as far as esoteric profiling of MTS component execution, disk I/O caches, or request queues.
A recent Intel study shows that any Web-based electronic commerce site services customer requests which can broadly be classified as one of the following five transactions:
| Browse | 80% |
| Search | 9% |
| User registration | 2% |
| Add item | 5% |
| Buy | 4% |
The study shows that users tend to perform browsing, user registration, and searches nine times as much as actually adding items to their shopping basket and checking out. This shows a 1:9 ratio. Thus for a population of 100,000 users, there would be 10,000 users performing basket and checkout operations while 90,000 users are browsing, registering, or using search operations. The IIS logs of a site can be analyzed to determine the actual browse to buy ratio.
There are other architectural improvements that can be applied to achieve higher performance with better scalability. The following sections describe each architectural improvement in detail.
It is essential that the application code does not make use of IIS Session variable, and disables IIS session management. IIS session management consumes an amount of memory for each user; the more values are stored in the session variable the more memory is consumed. For small amounts of values this may not matter much, but for larger amounts of memory (such as an object model) this can make a significant difference. For example, if each user consumed 1MB of memory stored as a session variable, if 1000 users were online this represents approximately 1GB of memory. This is an enormous amount of resources consumed. Usage of session variables severely limits scalability in this case to 1000 concurrent users given a machine with 1.5GB of memory. Without this resource exhaustion it’s conceivable that a higher number of concurrent users can be serviced up to the limits of CPU utilization ceiling.
Another disadvantage that arises out of session variables is that each client requires affinity to the server where it started up its session since that is where its session variables reside (on that local server). This requires session stickiness, eliminates on-the-fly redundancy (if server goes down or needs to be taken down, user sessions are destroyed), and eliminates dynamic load-balancing (this causes uneven user experience, some users experiences a really slow site while others zip through the site smoothly).
This affinity to the server can be mitigated by the use of the WLBS load-balancing software. WLBS uses the source IP address as the method to “stick” the request to a destination server. There are issues with this technique of "sticky sessions" load balancing. If the client makes use of a load balanced proxy server that presents a different IP address externally from each proxy server to access the Internet, WLBS will not be able to maintain session "stickiness" resulting in chaotic user experience. This is particularly problematic for users of a major ISPs. The workaround has been to dedicate a server for their traffic. While this may be a workable solution in the short-term, if the traffic continues to grow it may overwhelm the single dedicated server in the future.
This follows from the introduction presented earlier. Consider the situation of 100 Web servers (capable of serving 1000 concurrent users with mixed operations each) serving up 100,000 concurrent users. If instead you use 9 servers (10,000 concurrent users each serving static HTML content) can serve 90% of the users, and 10 servers (1,000 concurrent users each serving dynamic content) serve 10% of the users, the total number of servers drops to an efficient 19 Web servers.
IIS is very capable of handling a high rate of requests for static HTML/GIF content, while ASP page processing requires significant CPU processing and is serviced at a lower rate. In order to achieve the most efficient use of server utilization, operations that have similar load-factor characteristics or capacity requirements can be aggregated while those that differ should be separated.
Please refer to MCIS 2.0 Resource Kit for more detailed information. Also, Appendix F shows one simple technique to calculate the number of servers that may be needed for a Web farm.
There are many types of data that is rendered to HTML by ASP pages from data sources that are not strictly static but are not highly dynamic either. Examples of such data include product attributes (information, description, price, and so on), site announcements, sale announcements, and more.
These types of information can very easily be rendered as static HTML pages by a nightly process (or as necessary) and served up as static HTML/GIF content for a much higher throughput. The benefit is reduced overhead by avoiding ASP processing, and SQL Server data fetch.
In scenarios where the product information is relatively static but product price is driven by a database look-up, such as pricing by zip code, this technique can still help by framing product information in a separate HTML content frame from product price, which is processed by an ASP page. Another solution is to use an ISAPI filter that understands HTML template language and performs a look-up to an in-memory database similar to the way in which early database integration was done using the Internet Database Connector (Windows NT 4 Option Pack) with .idc and .htx files. This has the benefit of avoiding full ASP processing and retaining high-rate HTML servicing.
In cases where dynamic look-ups are required such as a price look-up based on zip code or customer identifiction (in cases where customized price lists are utilized), an in-memory database can be used to cache the look-up table. This helps to reduce overhead of data fetch across the network. The in-memory database can be refreshed by a nightly process (or as necessary).
In many cases, real-time inventory status as an item count available is unnecessary and can be replaced by an in stock “flag." These look-ups can also be cached in-memory with a nightly refresh. Caching this information helps reduce data fetch overhead placed on the SQL Server database.
The net effect of caching data in-memory results in faster look-up time with reduced latency (previously due to data fetch across the network). This in turns helps increase performance.
Look-up tables that are small in size are ideal for the application of this technique. Building out memory capacity of the hardware can help accommodate larger tables. An analysis of IIS and SQL Server logs can yield statistical data on which frequently used look-up tables would benefit from caching. Windows NT 4.0 Server sites will need to develop a custom Windows NT Service based facility.
As SSCE 3.0 is CPU bound, reducing CPU utilization helps improve ASP/Commerce Pipeline processing performance. This reduction can be found by identifying and separating complex, processor-intensive business rule computation as MTS components to dedicated MTS application servers.
There is an argument to be made on the trade-off between in-process execution of components vs. out-of-process execution of components marshaled via DCOM. To be precise only a benchmark can provide the ultimate performance numbers.
However, in general if the business rule is significantly processor intensive and is greater than the cost of marshalling via DCOM, then the component is a candidate for development as an MTS component. For example, if prior to distributing the component execution time was ten seconds and after distribution the component execution time drops to five seconds (i.e. ASP code call returns in five seconds instead of ten seconds), it shows that there is a net benefit in distributing the component to a dedicated MTS server. Dedicating execution of MTS components on a separate server provides additional computation capacity on the ASP/Commerce Pipeline that goes toward increasing performance of ASP/Commerce Pipeline processing.
If the business rule consists of only a few lines of code that is not processor intensive, it is probably not worth executing it on a dedicated server. It’s probably best to either leave it as an ASP function snippet and save object activation/invocation costs, or if it is sufficiently complicated in ASP, then code it as an ASP COM component using Visual C++ and ATL and activate it locally (use ATL wizard’s “Both” threading model).
This technique leverages the use of asynchronous communications to achieve a high rate of “fire and forget” operations/transactions to avoid the latency imposed by an operation such as a data fetch or extended computation.
In many cases, a different business unit or an entirely different company performs the actual fulfillment at a different geographical location (drop ships). The site frequently needs a way to notify the remote location of an order to be fulfilled, or a shipment status to be returned.
Rather than utilizing a database operation/transaction such as a periodic batch database extract and shipment of the results to the remote site, Microsoft Message Queue (MSMQ) services, or Microsoft Exchange, e-mail can be utilized to send notification and/or accept status information from the remote site.
The front-end servers accept the request and very quickly hand-off the information to MSMQ or via e-mail to a remote location. This results in a high rate of processing and provides a fast response turnaround time by front-end servers. The remote sites receive more timely updates than can be expected from a periodic batch database extract.
Sites usually require some sort of reporting on orders, transactions, usage, and so forth. These reports are typically generated from online logs or database records copied from the production database. MSMQ can be utilized to provide near real-time copies of data, which can be sent in “fire and forget” mode to a server dedicated to the capturing of reporting records. The benefit is that a lengthy, time consuming database record copy operation is no longer necessary. The records have already been shipped to the reporting database via MSMQ. The shipped records can even be processed/aggregated immediately at the remote location to provide near real-time report updates with reduced storage costs if the shipped record is immediately thrown away.
Commerce Server orders and receipts can also be recorded asynchronously rather than using an in-line database transaction; this allows the ASP page to avoid transaction latency and continue processing. The disadvantage is that the customer does not see an immediate order confirmation number and has to wait for e-mail to arrive, or wait until the orders and receipts end up in the database. This approach works best at sites with periodic load peaks to reduce transaction response time. The downside of using e-mail is that customers often provide erroneously formatted e-mail addresses which results in the non-delivery of notifications.
Defer credit card, tax, or other processing until it is necessary to do so. The processing can be performed at a later stage or in batch mode on a dedicated server. In many cases a legacy system already exists to perform these operations. This allows the front-end servers to process requests at a high rate to provide a fast response turnaround time. Failures and exception reports can be sent to the shopper via e-mail. Be aware that some states/countries may require full disclosure of a purchase (including sales and taxes).
Internet sites are typically built in Rapid Application Development (RAD) mode, or with a very short timeframe between releases with many iteration cycles. This frequently results in code that is thrown together without proper consideration to efficiency, clarity, or even reusability. The resulting code base can be duplicative, inefficient, contain lengthy code execution paths, and so on. All these flaws contribute to less than optimal ASP code processing.
A simple utility can easily be created to insert instrumentation code that records execution time per line of code into ASP source files. Running the ASP code will then generate an execution profile of the source that can be used to identify where pockets of code can benefit from further optimization.
Business rules are typically prototyped with a Commerce Pipeline Scriptor component and never converted to compiled code. With each scriptor component a scripting engine is invoked with its associated activation and invocation overhead. A significant increase in performance can be achieved by converting the logic to a Visual C++ and ATL compiled Commerce Pipeline component. For best performance, take care to use the “Both” instead of the default “Apartment” threading model.
Further increases in the Commerce Pipeline can be achieved by removing stages that are not required, and/or breaking up the pipeline for separate execution where possible or necessary. Reducing the number of stages required reduces the activation/invocation overhead and results in a higher rate of processing.
It is also possible to bypass the Commerce Pipeline completely and utilize custom MTS components throughout. This technique is not recommended in general unless absolute performance is necessary. The disadvantage of bypassing the Commerce Pipeline is loss of access to third party ISV pipeline components resulting in significantly increased development costs (e.g. custom tax, shipping, inventory, and/or accounting integration components).
This technique bypasses the Commerce Store completely and writes baskets, orders, and receipts directly to an SQL Server database with a custom schema. It involves custom development of code to replace the functionality DBStorage provides. This technique essentially bypasses marshalling inefficiencies associated with DBStorage. Although it appears simple to implement, careful design and benchmarking are required to ensure that appropriate performance advantage is achieved.
The disadvantage of this technique is loss of flexibility that the dictionary object provides. For each additional field that is required, this technique requires modification to the database schema and read/write code, resulting in longer timeframes and increased development costs.
This technique really extends the technique of separating static content from dynamic content request servicing on dedicated servers. SSCE 3.0 provides Catalog Builder a service to build index catalogs that can be run on a dedicated server, and Search Server a service to search built catalogs that can also be run on a dedicated server. Each Catalog Builder can propagate up to 32 Search Servers, thus providing an efficient way to create and search catalogs.
Further increases in search capacity can be achieved by dedicating search servers for the site and increasing capacity in incremental steps.
Usage of SQL Server 7 with its native OLEDB provider is recommended. SQL Server 7 provides many improvements including self-tuning and high-performance online/live backup. Analyze the current SQL statement stream to optimize indices further.
It is possible to scale individual databases utilized by SSCE 3.0 by placing them on dedicated servers that are scaled vertically:
The following techniques may be necessary only for extremely high-traffic sites:
The SSCE 3.0 Commerce Store databases may also be scaled horizontally:
The following diagram illustrates architectural improvements; dedicated static HTML/IIS servers, dedicated Search/User Reg. ASP servers, dedicated Basket/Checkout servers, multiple LDAP servers, multiple Membership databases, and partitioned Commerce Store database on multiple SQL-based servers are utilized.
Figure 3. Architectural Improvement
There is a point to be made when consolidating Web servers. Although it is possible to achieve a low server count there are also significant disadvantages if this exercise is carried too far.
An electronic commerce Web site is typically required to operate at high capacity, high traffic, and high-availability. Fewer servers mean that each server carries responsibility for a higher amount of capacity, traffic, and availability. The result is that the loss of any one server has a higher impact on the site overall.
The disadvantages of consolidating a Web server farm include:
The above disadvantages can be mitigated by the use of redundant servers, disks, power supplies, and so on. However, in very short time the site begins to look very similar to an unconsolidated Web farm.
A highly scalable site can be built with as few servers as possible by scaling vertically, horizontally, and by architectural improvements. For the Internet, it is actually beneficial to scale horizontally. Scaling horizontally provides Internet sites with redundancy, predictable incremental capacity increases, along with predictable incremental capacity degradation in cases of failures.
Tailoring and directing operations with similar workload factors to dedicated servers allows the site to efficiently scale up with the least amount of servers.
Using the techniques described in this document SSCE 3.0 can be scaled to serve 100,000 concurrent users with 24 front-end servers. Contrast this with the original number of 100,000 / 1,000 = 100 servers using a conventional site architecture.
Back end SQL servers can also be scaled as necessary, which adds to the total server count. However, it has been rare to find a site that requires the back end server to be scaled horizontally. It is very easy and common to satisfy capacity requirements by scaling the SQL Server vertically either by adding processors, memory, or going to a higher-class processor.
Note that the 100,000 concurrent users number is not the maximum limit. It is possible to accommodate 200,000 users with 24 * 2 = 48 servers plus the necessary back end servers. At this point the scaling goes horizontal.
In the future, Windows 2000, IIS 5 with Web clusters, IIS 6, and other technologies (1 GHz Alpha, Merced chips, Windows 2000 64-bit) will allow much higher scalability and performance. With current technologies in development, within 1-2 years it may be possible to achieve one million concurrent users.
Refer to the following technical case studies to see examples of solutions built on the Microsoft Commerce platform:
Direct Marketing and Selling
Supply Chain Integration
Corporate Procurement
(Courtesy of Saqqara)
Both Step Search Professional and Site Server consist of a collection of COM objects and ASP scripts. Each ASP displays (or processes) a page that performs a specific function in the buying process, such as comparing two products, asking for credit card information, and so on. Therefore, the easiest way of connecting the two systems is to simply use URL links. For example, in the product data sheet page, a "Buy" button can be added to point to Site Server's "add to cart" ASP page. Once the "add to cart" ASP is invoked, the added product becomes part of the shopper's order and can be processed normally by Site Server's ASP pages. The following code shows how the "Buy" button in the product datasheet ASP page points to the shopping cart in Site Server.
<A HREF="xt_orderform_additem.asp?sku=<%= Server.URLEncode(objProduct.ProductNumber) %>">&Fam=<%= objFamily.NavigationName %>&ProductID=<%= objProduct.ID %><IMG SRC="/images/buy.gif"></A>
Note that the URL that invokes the shopping cart script passes the product number (SKU), family navigation name, and product database ID to indicate which product should be added to the cart. The ID or the product number can be used in the remaining shopping cart stages to retrieve more information about the product from the Step Search database. The “xt_orderform_additem.asp” file provided by Microsoft should be modified to add the ID and family to the order form. To do this, change the existing one line which calls AddItem to the following three lines:
set item = mscsOrderForm.AddItem(product_sku, product_qty, 0)item.id = Request("ProductID")
item.Family = Request("Fam")(Courtesy of Saqqara)
Site Server uses the Order Processing Pipeline to manage orders (see Microsoft's Site Server documentation). Each stage of the pipeline performs a different function in the order processing sequence. Each stage consists of zero or more COM objects associated with that particular stage. The first stage, called "product info" uses a default COM component supplied with Site Server, which retrieves the product description from the product database. This default component (called QueryProdInfo or QueryProdInfoADO) uses a default SQL query to retrieve the desired product information from the database.
The integration of Step Search Professional and Site Server Commerce Edition can be achieved in one of two ways:
The same process can be used to create replacement components for the different stages of the Order Processing Pipeline. For example, a replacement component can be used for the Price Adjust stage so that it calculates a price adjustment based on whether the product is on sale. The Step Search Professional database can have, for example, an extended attribute that contains the discount percentage for each product on sale. The replacement component in the Price Adjust stage can then read the discount value and calculate the discounted price.
Below is an example of a VBScript scriptor component used to link Step Search to Site Server (this component replaces the QueryProdInfo component in the Product Information stage).
Option Explicit
function MSCSExecute(config, orderform, context, flags)
Dim objConnection, objItem, objProduct, objDataAttribute
Dim dblProductPrice
REM --
REM -- Create a connection to the Step Search database
REM --
Set objConnection = CreateObject("ADODB.Connection")
objConnection.open "mydsn", "username", "password"
REM --
REM -- Instantiate the required Step Search objects
REM --
Set objProduct = CreateObject("SSPCatalog.Product")
For each objItem in orderform.items
REM --
REM -- Get product information from Step Search
REM --
call objProduct.Initialize(objConnection, Clng(objItem.ID))
if objProduct.ID = 0 then
call objProduct.InitializeByProductNumber(objConnection, _
objItem.sku, objItem.Family)
end if
if objProduct.ProductNumber = "" or objProduct.ID = 0 then
objItem.delete = 1
else
Set objDataAttribute = _
objProduct.GetSingleAttribute(objConnection, "PRODUCT_PRICE")
dblProductPrice = objDataAttribute.Value * 100
REM --
REM -- Fill the order form with the Step Search product information
REM --
objItem.[_product_sku] = objItem.sku
objItem.[_product_name] = objProduct.Name
objItem.[_product_list_price] = dblProductPrice
end if
Next
MSCSExecute = 1
end function
sub MSCSOpen(config)
end sub
The SendSMTP component can send e-mail to any mail recipient. However, in order to receive a business data object and pass it to an application on the receive side, the receiving mail system must invoke the receive pipeline.
If you are using Microsoft Exchange 5.5, you can run a Visual Basic script to invoke an OrderPipeline object (but not a MtsPipeline or MtsTxPipeline) on the Exchange server when mail arrives in the designated mailbox to process the received messages. SSCE includes the Recvsmtp.vbs example script, which is installed on your local hard drive in \Microsoft Site Server\SiteServer\Commerce\. The following procedure describes how to use this script with an Exchange 5.5 mail server and Microsoft Outlook® 8.03 installed on a client computer that connects to the Exchange server.
On the File menu, point to New, and click Folder. In the Make this folder a subfolder of box, expand Public Folders, and click All Public Folders. In the Name box, type a name for the new folder. Click OK to create the folder.
In the Administrator window, select the System Folders container for your organization. Click Events Root, and then select the EventConfig folder that corresponds to the Microsoft Exchange Server computer on which the folder owner should be granted permission. On the File menu, click Properties. Click Client Permissions. Click Add, and then select the user (Client1) who will have permission to run applications or other event handlers on this server. In the Roles box, select the Editor role. This enables the user to create scripts that handle events on this server.
In the Administrator window, select the Public Folders container for your organization. Select the CIReceive folder that you created in Step 3. On the File menu, click Properties. Click the Advanced tab. Clear the Hide from address book check box. Click the Email Address tab. Write down the e-mail address corresponding to SMTP (CIReceive@org1.com, in this example). Use this e-mail address for the SMTP Host when configuring the SendSMTP component for your Commerce Interchange transmit pipeline.
Using the Outlook client, select the CIReceive public folder. On the File menu, click Properties. Click the Agents tab, and then click New. Type a name for the new agent. Select the event A new item is posted in the folder to enable the agent to respond when a new item is posted. Click Edit Script. If Microsoft Visual InterDev is installed, it starts with a blank script that you can edit. If it is not installed, Notepad starts with a blank script. Copy the Visual Basic script from \Microsoft Site Server\SiteServer\Commerce\recvsmtp.vbs, and then paste it into the blank file. Make any desired changes, and then save the file and exit from Visual InterDev or Notepad. Click OK to save the new agent.
<%%
'
' Example ANSI X12 850 EDI Template for MakePO component (VBScript)
' Microsoft Commerce Server v3.00
'
' Copyright (c) 1998 Microsoft Corporation. All rights reserved.
'
' This example illustrates the use of MakePO for generating
' an ANSI X12 850 EDI Message for MSMarket Office Supplies.
' It can be placed into the Purchase Order Submit stage of the' SubmitViaInterchange pipeline or in the Map stage of the' InterchangeTransmit pipeline in MSMarket.
'
' The database is used to implement the interchange control number
'
' Database Setup (MS SQLServer):' 1) Create a control number table with an initial value. Example:
' CREATE TABLE control_number (next_control_number int)
' INSERT INTO control_number VALUES(0)
' 2) Create a stored proc to increment the sequence. Example:
' CREATE PROCEDURE sp_get_next_control_number
' @id int = 0' AS
' UPDATE control_number SET @id = next_control_number = next_control_number + 1
' SELECT @id id
' 3) Configure the Connection_String in the code below.
Dim t
Dim InterchangeCreationDate
Dim InterchangeCreationTime
Dim GroupCreationDate
Dim GroupCreationTime
Dim InterchangeControlNumber
Dim GroupControlNumber
Dim SenderType
Dim SenderID
Dim ReceiverType
Dim ReceiverID
Dim VendorName
Dim VendorContact
Dim VendorPhone
Dim Dbg
t = Now()
InterchangeCreationDate = YYMMDD(t)
InterchangeCreationTime = HHMM(t)
GroupCreationDate = YYMMDD(t)
GroupCreationTime = HHMM(t)
InterchangeControlNumber = XDigits(IncrementControlNumber(),9)
GroupControlNumber = CLng(InterchangeControlNumber)+100000000
SenderType = "01" 'DUNS Number
SenderID = "081466849 "
ReceiverType = "09" 'DUNS+4 Number
ReceiverID = "123456789ABCD "
VendorName = "VendorName"
VendorContact = "VendorContact"
VendorPhone = "425-555-1212"
Dbg = 0
'++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
' Begin X12 850 Template
'++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
%%>ISA~00~ ~00~ ~<%%=SenderType%%>~<%%=SenderID%%>~<%%=ReceiverType%%>~<%%=ReceiverID%%>~<%%=InterchangeCreationDate%%>~<%%=InterchangeCreationTime%%>~U~00200~<%%=InterchangeControlNumber%%>~0~P~^
GS~PO~<%%=Trim(SenderID)%%>~<%%=Trim(ReceiverID)%%>~<%%=GroupCreationDate%%>~<%%=GroupCreationTime%%>~<%%=GroupControlNumber%%>~X~003020
ST~850~000000001
BEG~00~NE~<%%=Dictionary("RequisitionID")%%>~~<%%=Dictionary("DateChanged")%%>
CUR~BY~<%%=Dictionary("ISOCurrencyCode")%%>
REF~DD~<%%=Dictionary("order_id")%%>
PER~BD~<%%=Dictionary("ReqName")%%>~EM~<%%=Dictionary("ReqEmailalias")%%>
PER~BD~~TE~<%%=Dictionary("ReqPhone")%%>
PER~MG~~EM~<%%=Dictionary("_shopper_manageremailalias")%%>
DTM~002~<%%=Dictionary("RequiredByDate")%%>~~~<%%=CenturyCode(now)%%>
N1~ST~<%%=Dictionary("_shopper_name")%%>~92~<%%=Dictionary("_shopper_buildingdescription")%%>
N2~Office: <%%=Dictionary("_shopper_roomdescription")%%>
PER~DC~<%%=Dictionary("_shopper_name")%%>~TE~<%%=Dictionary("_shopper_workphonenumber")%%>
PER~DC~~EM~<%%=Dictionary("_shopper_emailalias")%%>
N1~VN~<%%=VendorName%%>~92~<%%=Dictionary("VendorNumber")%%>
PER~IC~<%%=VendorContact%%>~TE~<%%=VendorPhone%%>
<%%
' The following section repeats for each line item in the PO:
dim index
dim quantities
for index=1 to items.items.count
%%>PO1~<%%=index%%>~<%%=LineItem(index,"Quantity")%%>~<%%=LineItem(index,"UnitCode")%%>~<%%=LineItem(index,"list_price")%%>~~VP~<%%=LineItem(index,"PartNumber")%%>
CUR~BY~<%%=LineItem(index,"ISOCurrencyCode")%%>
PID~F~~AB~~<%%=LineItem(index,"Partdescription")%%>
REF~32~<%%=LineItem(index,"AccountCode")%%>
REF~CA~<%%=LineItem(index,"CostCenter")%%>
PER~EB~<%%=LineItem(index,"insertby")%%>
<%%quantities = quantities+CLng(LineItem(index,"Quantity"))
next
' The following section follows the line items and is not repeated:
%%>CTT~<%%=items.items.count%%>~<%%=quantities%%>
AMT~TT~<%%=Dictionary("TotalAtSubmit")%%>
SE~<%%=ST2SELineCount()%%>~000000001
GE~1~<%%=GroupControlNumber%%>
IEA~1~<%%=InterchangeControlNumber%%>
<%%
'++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
' End X12 850 Template
'++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
'++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
' Helper Functions
'++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
' Extract an element from the business object, raise an error if missing
' (MakePO exposes the business object as 'items')
Function Dictionary(element)
Dictionary = items.Value(element)
If isNull(Dictionary) then
if dbg=1 then
Dictionary = "??"&element&"??"
else err.Raise -1,"PO850","Dictionary element "&element&" not found"
end if
End if
Dictionary = CStr(Dictionary)
End function
' Extract a line item element from the business object, raise an error if missing
' (MakePO exposes the business object as 'items')
Function LineItem(index,element)
LineItem = items.items(index-1).Value(element)
If isNull(LineItem) then
if dbg=1 then
LineItem = "??item("&index-1&")."&element&"??"
else
err.Raise -1,"PO850","LineItem element "&element&" not found on item "&index-1
end if
End If
LineItem = CStr(LineItem)
End function
' Helper function that formats a number to x digits
' +Doesn't work with negatives
' +Doesn't handle overflow
Function XDigits(number, digits)
dim pow
dim sweep
pow=1
for sweep=2 to digits
pow=pow*10
if number < pow then
XDigits = XDigits & "0"
end if
next
XDigits = XDigits & CStr(number)
End Function
Function TwoDigits(number)
TwoDigits = XDigits(number,2)
End Function
Function ST2SELineCount
ST2SELineCount = 14 + 6 * items.items.count + 3
ST2SELineCount = TwoDigits(ST2SELineCount)
End Function
Function YYMMDD(t)
YYMMDD = TwoDigits(Year(t) mod 100) & TwoDigits(Month(t)) & TwoDigits(Day(t))
End Function
Function HHMM(t)
HHMM = TwoDigits(Hour(t)) & TwoDigits(Minute(t))
End Function
Function CenturyCode(t)
CenturyCode = Year(t) - (Year(t) mod 100)
CenturyCode = CStr((CenturyCode/100)+1)
End Function
Function IncrementControlNumber()
dim conn
dim cmdTemp
dim rs
Connection_String = "dsn=stores;user=sa;pwd=;database=stores"
Set conn = CreateObject("ADODB.Connection")
conn.Open Connection_String
Set cmdTemp = CreateObject("ADODB.Command")
cmdTemp.CommandType = 1
Set cmdTemp.ActiveConnection = conn
cmdTemp.CommandText = "sp_get_next_control_number"
Set rs = CreateObject("ADODB.Recordset")
rs.Open cmdTemp, , 1, 1
IncrementControlNumber = rs("id").Value
End Function
%%>
Option Explicit
REM ######################################################################### REM REM APPROVAL.VBS REM Microsoft market REM Microsoft Commerce Server v3.00 REM REM Copyright (c) 1998 Microsoft Corporation. All rights reserved. REM REM #########################################################################
'
' approval.vbs is the scriptor component that sets the approvallimit flag and formats the manager approval , needs approval, approved / rejected emails.
'
Function mscsexecute(config, orderform, context, flags)
Dim mscsDataFunctions, mscsMessageManager, subaction, domain, convertratio
Set mscsDataFunctions = context.DataFunctions
Set mscsMessagemanager = context.MessageManager
domain = context.domain
if not(len(domain)> 0) then orderform.[_purchase_errors].Add(mscsMessageManager.GetMessage("config_domain_absent"))
mscsexecute = 3
exit function
end if
subaction = context.subaction
convertratio = context.convertratio
If ("SUBMIT" = subaction) Then
REM -- order needs approval, so format manager email and disable requisitioner email.
orderform.TotalAtSubmit = orderform.[_total_total]
Dim approvallimit
approvallimit = CLng(orderform.[_shopper_approvallimit])
REM -- convert the approvallimit from base currency into the currency for the locale and then into the base unit for the new locale currency.
approvallimit = mscsDataFunctions.ConvertMoneyStringToNumber(CStr(CLng(approvallimit * CSng(convertratio))), CLng(orderform.locale))
If (approvallimit < orderform.[_total_total]) Then
orderform.needapproval = True
orderform.[_to_approver_email] = orderform.ApprovalManagerAlias & domain
orderform.[_approver_subject] = replace(MSCSMessagemanager.GetMessage("sub_email_subject"), ":1",orderform.requisitionid)
orderform.[_approver_body] = MSCSMessagemanager.GetMessage("sub_pre_order_approval") & chr(10) & chr(13) & context.approvalurl
Else
orderform.needapproval = False
orderform.[_to_approver_email] = "None"
End If
orderform.[_to_requestor_email] = "None"
Else
REM -- order accepted/rejected, so format requisitioner email and disable manager email.
orderform.[_to_requestor_email] = orderform.reqemailalias & domain
orderform.[_requestor_subject] = replace(MSCSMessagemanager.GetMessage("sub_email_subject"), ":1",orderform.requisitionid)
If ("ACCEPT" = subaction) Then
REM -- flip the approval flag so components down the pipeline will execute.
orderform.needapproval = False
orderform.[_requestor_body] = MSCSMessagemanager.GetMessage("sub_post_order_approval")
ElseIf ("REJECT" = subaction) Then
orderform.[_requestor_body] = MSCSMessagemanager.GetMessage("sub_post_order_rejection")
End if
orderform.[_to_approver_email] = "None"
End If
mscsexecute = 1
End Function
Note The data below is just an estimate (as each site and solution varies) and is only here to demonstrate a method for calculation.
Lets assume an IIS server is capable of concurrently serving:
| Static HTML/GIF content requests | 10,000 users |
| Search ASP page requests | 1,500 users |
| User Registration ASP requests | 1,500 users |
| Add item to basket ASP requests | 1,200 users |
| Checkout purchase ASP requests | 1,200 users |
Using the above numbers that we have assumed, it makes sense to dedicate a server for static HTML/GIF contents, a server for both Search ASP page requests and User registrations, a server for both add item and checkout operations.
For a user population of 100,000, with the above-assumed numbers and page usage percentages the server topology looks as follows:
| Static HTML/GIF content servers: | 100,000 * 80% | = 80,000 users / 10,000 users | = 8 servers |
| Search + User reg. ASP servers: | 100,000 * 11% | = 11,000 users / 1,500 users | = 8 servers |
| Add item + Checkout ASP servers: | 100,000 * 9% | = 9,000 users / 1,200 users | = 8 servers |
| Total of front-end servers required: | = 24 servers |
The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.
This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENT.
© 1999-2000 Microsoft Corporation.
All rights reserved.
Microsoft, ActiveX, FoxPro, FrontPage, Jscript, Outlook, Visual Basic, Visual C++, Visual InterDev, Visual J++, Visual Studio, Win32, Windows and Windows NT are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries/regions.
Other product and company names mentioned herein may be the trademarks of their respective owners.