ColdFusion 4.5.1 Windows NT Performance Fact Sheet and Sizing Guide

About ColdFusion

ColdFusion is a high-performance, scalable, open platform for delivering Web applications that range from simple database-driven pages to full e-commerce solutions on intranets, extranets and the Internet.

First released in 1995, ColdFusion has become the leading cross-platform Web application server. ColdFusion was the first Web application server on Windows NT, and as a pioneer in the application server market, ColdFusion has established a broad base of support with thousands of customers from small consulting groups to multinational corporations. Used by almost half of the Fortune 500, ColdFusion provides the fastest way to develop and deliver scalable enterprise and e-commerce applications.

Introduction

This document outlines performance and configuration considerations of a ColdFusion Web application. It summarizes testing that Allaire performed in its Enterprise Scalability Lab and details several examples of sizing ColdFusion Web application deployments. It includes the results of performance and scalability tests on the ColdFusion “Tack2++” application, a version of the “Tack2Plus” sample application included with ColdFusion, as well as configuration information and a copy of the test script actions. These tests can serve as a general guide for customers deploying ColdFusion-built applications. Customers can use this Performance Fact Sheet to assist with deployment planning to help ensure that sufficient resources are available for successful deployment.

Performance & Scalability Testing

Web site deployments vary in size and complexity, from several servers to several hundred servers, depending on many factors. ColdFusion deployment scenarios typically involve multiple machine configurations, facilitated by the built-in server-clustering capabilities of ColdFusion Server Enterprise edition. The clustering capabilities of ColdFusion Enterprise provide a high degree of scalability and fault-tolerance with dynamic load balancing, and network and service-level failover.

Testing was performed on 1, 2, 4, and 8 Compaq dual-processor 1850R machines[1] running ColdFusion Server 4.5.1, and 2 quad-processor Compaq 6400R machines in a Compaq DISA configuration running MS SQL Server 7.0 Enterprise Edition as the database server. The 2, 4, and 8 machine cluster configuration was performed to provide examples of:

1) ColdFusion’s linear scalability

2) Sample test data from which customers can extrapolate for use as a general guide when sizing and deploying ColdFusion application deployment configurations

Customers using these results to size their ColdFusion application server environments must test their own applications using an enterprise-level load-testing tool. This step is critical to determine sizing requirements for their specific application. Effective load testing techniques will determine the applications’ performance and scalability characteristics. It can also help find and remove application and site performance bottlenecks, and test the application’s "wide" (multiple machine) vs. "deep" (multiple processors) scaling characteristics. Widely varying traffic patterns, peak traffic periods, and future growth must also be taken into account to properly size a production server configuration prior to deployment.

Customers are strongly urged to load test as soon as a working prototype of the application is available, and should continue load testing through development on the entire working application before it is placed into production.

For customers wishing assistance with this process, Allaire Consulting offers a 5-day onsite Performance Analysis & Tuning consultation, which can help ensure the success of application deployment. Allaire Training also offers a 2-day Performance Analysis and Tuning course designed to help customers gain the necessary knowledge to properly load test their applications, identify and fix site bottlenecks and help ensure deployment success.

The key variables measured or controlled in the load testing performed for this Fact Sheet were:

Page Response Time

The time experienced by the Web browser from URL submission until all page elements have been downloaded and page rendering is complete.

Simultaneous Users

Concurrent site users simulated using the Segue SilkPerformer 3.5 load-testing tool.

Number of Servers

The number of Spectra server machines that make up a cluster of servers.

Spectra Requests/time – The number of Spectra requests processed per second, minute, hour, and 24-hour period.

Page Views/time

The number of browser pages displayed per second, minute, hour, and 24-hour period.

HTTP Hits/time

The number of browser HTTP hits experienced per second, minute, hour, and 24-hour period.

The ColdFusion Tack2++ Web Application

The Tack2Plus sample application included with ColdFusion is a reasonable, although uncomplicated, ColdFusion e-Commerce Web application for a fictitious company selling surf and sail-related gear on the Internet. It was tested to provide data that customers can use as a starting point for sizing their production server clusters.

The Tack2++ application tested for this Fact Sheet is new version of the original Tack2Plus sample application included with ColdFusion. It preserves all of the functional characteristics of the original Tack2Plus application, but makes use of ColdFusion’s template and query caching features, as would a real-world production ColdFusion Web application.

The Tack2++ application was load tested in order to test capacity in terms of both application throughput and number of supportable simultaneous users. While results will vary with customer applications of varying scope and complexity, customers can expect to realize similar results with similar applications and server configurations.

Four types of users were simulated during the load testing to model as closely as possible the real-world load mix an e-Commerce site is likely to experience. These user types are described below:

User1: Brief, curious visitors who visit the site’s main page, enter the site, then leave.

User2: Browsers, who visit, browse and drill into several product categories, then leave.

User3: Shoppers who visit, browse product categories, perform a site search, add a few items to their shopping cart, then leave, abandoning their shopping cart.

User4: Committed shoppers, who visit, browse product categories, perform a site search, add items to their shopping cart, commit to checkout and place their order for purchase before leaving the site.

The tests were performed with the following user load mix, expressed as a percentage of the total load:

User1: 25%

User2: 25%

User3: 25%

User4: 25%

The user load mix used in this round of testing can be considered more demanding on the server than a typical load mix normally experienced on public e-commerce sites, where studies have shown approximately 90% of all traffic consists of browsing activities, and less than 5% of all users actually add items to the shopping cart and checkout.

ColdFusion Clustered NT Server Test Results

The ColdFusion Tack2++ application proved linearly scalable through the 8 machine cluster test. That is, page response time grew in direct proportion to the amount of load applied in the tests. The application serviced 5,280 simultaneous clients at a rate of about 62 million HTTP hits, or about 25 million ColdFusion-generated page views per day. At this rate, the average page response time was about 4 seconds, with server requests evenly distributed throughout a 24-hour day. This translates to a rate of roughly 1.9 billion HTTP hits, or 763 million page views per month, with server requests evenly distributed throughout a 30-day month.

The test data also reveals that ColdFusion request throughput remained nearly constant at this rate while maintaining an average page response time of about 1 second for 4,200 simultaneous users, 2 seconds for 4,600 simultaneous users, and 8 seconds for 5,280 simultaneous users.

The simultaneous users simulated for these tests represent, as closely as possible, real-world users with actual user-recorded think time between page clicks. The use of actual user-recorded think time in load testing is important when testing server and application capacity. The use of artificially inflated think time between pages may result in the misrepresentation of the true number of simultaneous users an application can support.

Figure 1 illustrates the linear scalability of the application in HTTP hits serviced per day for 1, 2, 4, and 8 servers with an average page response time of about 1 second.

Figure 1 - Tack2++ Cluster HTTP Throughput

Figure 2 illustrates the linear scalability of the application in terms of page views serviced per day for 1, 2, 4, and 8 servers.

Figure 2 - Tack2++ Cluster Page View Throughput

Figure 3 illustrates the linear scalability of ColdFusion as the number of simultaneous users able to be serviced for each response time target threshold (1, 2, 4, and 8 seconds) for 1, 2, 4, and 8 servers.

Figure 3 - Tack2++ Cluster Simultaneous Users

Figures 4 and 5 detail the summarized load testing data for the clustered server Tack2++ NT tests. Since the Tack2++ application consists of exactly one ColdFusion template request for each application page view (i.e. no raw HTML pages or HTML framesets are used), the number of ColdFusion request is equal to the number of page views. The data for ColdFusion requests and page views reflect this one-to-one relationship.

Figure 4 - Tack2++ (1) and (2) Server Cluster Load Test Data

Figure 5 - Tack2++ (4) and (8) Server Cluster Load Test Data

ColdFusion Single NT Server Test Results

On a single NT machine, the Tack2++ application proved capable of servicing 810 simultaneous users with an average page response time of 7.60 seconds. The Tack2++ application also maintained an average page response time of less than 1.0 second as load increased to beyond 525 simultaneous users.

Figure 6 below illustrates the response time measured as the number or simultaneous users increased from 1 to 810 for a single Compaq 1850R with 2 x 500Mhz processors and 512MB RAM.

Figure 6 - Tack2++ Single Server Response Time

Figure 7 below illustrates single server HTTP throughput as the number or simultaneous users increased from 1 to 810. Throughput reached the optimal range of about 7.8 million HTTP hits at 525 simultaneous users and remained relatively constant through the 810 simultaneous user load level.

Figure 7 - Tack2++ Single Server HTTP Throughput

Figure 8 illustrates single server page view throughput as the number or simultaneous users increased from 1 to 810. Throughput reached the optimal range of about 3.2 million page views at 525 simultaneous users and remained relatively constant through the 810 simultaneous user load level.

Figure 8 - Tack2++ Single Server Page View Throughput

Figure 9 details the load testing data summary for the single server NT Tack2++ tests. The test data reveals that a single Compaq 1850R dual-processor NT server capable of servicing about 7.8 million HTTP hits, or about 3.2 million page views per day, with server requests evenly distributed throughout the day. This translates to a rate of roughly 234 million HTTP hits, or 98 million page views per month, with server requests evenly distributed throughout the month.

This rate of requests is achieved while maintaining sub-second (≤ 1.0 second average page response time, and servicing 525 simultaneous users. Again, the optimal throughput range of ~3.2 million page views, or ~7.8 million HTTP hits is realized at about 525 simultaneous users.

Figure 9 - Tack2++ Single Server Test Data

Using the ColdFusion Tack2++ Test Results for Product Server Sizing

Customers whose ColdFusion applications have characteristics similar to those of the Tack2++ application apply these results directly to to assist in sizing their production server configurations. However, customers whose ColdFusion applications have characteristics dissimilar to those of the ColdFusion Tack2++ application can also use these test results. This can be accomplished by factoring into account the performance degradation or improvement the application demonstrates relative to the ColdFusion Tack2++ application.

Sizing Methodology

By arriving at one of the key variables measured or controlled in the load testing performed for this Fact Sheet, and knowing the desired average page response time, you can Look up or extrapolate the estimated number of servers likely to be required to service a given ColdFusion application’s anticipated load.

The sizing methodology used in the sizing examples can be summarized in five basic steps:

Step 1: Select a load metric to use for sizing.

Step 2: Estimate the peak-period load the site is expected to experience.

Step 3: Adjust the peak-period load estimate.

Step 4: Identify a peak-period page response-time limit.

Step 5: Look up the estimated number of servers required from ColdFusion Tack2++ test data.

Below is an overview of the sizing methodology:

Step 1: Select a preferred load metric to use for sizing:

a) Page views

b) HTTP hits, or

c) Simultaneous users

Customers with existing load estimates based on other metrics, such as unique site visitors per month, may need to gather additional data or perform additional calculations to arrive at one of these metrics to use this sizing methodology effectively.

Step 2: Estimate the peak-period load the site is expected to experience:

a) Page Views

Estimate the average number of page views per hour the Web site will need to support during the hour of peak site load.

For example, 120,000 page views per hour during the hour of peak site load.

b) HTTP Hits

Estimate the number of HTTP hits per hour the Web site will need to support during the hour of peak site load

For example, 350,000 HTTP hits per hour during the hour of peak site load.

c) Simultaneous Users

Estimate the number of people (users) who will use the Web site concurrently during the hour of peak site load.

For example, 800 simultaneous users during the hour of peak site load.

Customers may need to convert their existing page view and HTTP hit load estimates from monthly, weekly, or daily to hourly to use this methodology effectively. One possible approach to this conversion is shown below:

Peak load page view conversion:

For example:

Peak load HTTP hit conversion:

For example:

Step 3: Adjust the peak-period load estimate:

If the ColdFusion application being sized is more complex (more resource-intensive) or less complex (less resource-intensive) than the ColdFusion Tack2++ application, the peak-period load estimate can be multiplied by an “adjustment factor” to help account for its performance characteristics when using the Tack2++ load test data.

For example, the following adjustment factors might be used, depending on site complexity and resource utilization:

In these examples a peak-period load of 100,000 page views per hour, 350,000 HTTP hits per hour and 800 simultaneous users is multiplied by the adjustment factor to yield the adjusted peak-period load.

This value may more accurately reflect the difference between the ColdFusion Tack2++ application data and the application being sized.

Any number of additional adjustments to the peak load estimate can be made to account for application-specific attributes different from those of the Tack2++ application tested, such as:

o Number of HTTP requests, HTML frames, or ColdFusion requests per page view

o Number of DB queries per page view

o Other resource-intensive or prolonged external protocol calls to external servers, etc.

Step 4: Identify a peak-period page response-time limit:

Determine a limit on how many seconds a site user may have to wait for a page during periods of peak load. For example: 1, 2, 4, or 8 seconds. Generally, the lower the tolerable response-time limit is set, the more servers and resources are required to sustain the page response-time limit.

Step 5: Look up the estimated number of servers required from ColdFusion Tack2++ test data:

Using the adjusted peak-period load estimate and the desired page response-time, Look up the number of servers required from the ColdFusion Tack2++ cluster load test data results provided in this Fact Sheet.

Since the ColdFusion Tack2++ application proved scalable across server machines, an alternative sizing technique is to:

a) Look up the desired page response-time in the ColdFusion Tack2++ single server test results data,

b) Determine what load level the single server was capable of handling with this response time, and

c) Calculate the estimated number of servers required by dividing the single server load capacity into the adjusted peak-period load estimate, as:

For example, from Figure 9, a single server running ColdFusion Tack2++ proved capable of servicing 134,988 page views per hour with a response time of about 4 seconds.

If the adjusted peak-period load estimate is 550,000 page views per hour, then:

This second approach can be useful if the desired load level and response time is not directly listed in the test data from the ColdFusion Tack2++ cluster tests. This can happen when the number of servers required exceeds the eight machines tested.

The examples that follow should be used only as a general guide for performing production server sizing. Customers must perform similar load testing and sizing exercises using load test data from their specific ColdFusion applications to ensure accurate server sizing. Sizing exercises should be performed on a regular basis once a site is placed into production, and should factor in existing load patterns and growth projections to ensure successful load handling and a satisfied user base.

Customers should always consider peak period loads in their estimates, allowing for occasional load spikes, which can often occur unexpectedly, but have a dramatic, and often lasting, negative effect on users’ perceptions of the site.

EXAMPLE 1: Using Page Views

A customer is deploying a ColdFusion application and they expect to receive 70 million page views per month. The customer wants to maintain an average server page response time of no more than about 4 seconds during peak periods. They are expecting to see most of this load distributed somewhat evenly throughout a 6-hour period each day of the month.

The application makes extensive use of logging and database queries. The customer believes the application makes “heavy” use of server resources compared to the ColdFusion Tack2++ application tested in this fact sheet.

To estimate the number servers required to adequately service this load within the desired response time periods during peak load:

Step 1: Select a load metric to use for sizing:

The customer has estimated load in page views.

Step 2: Estimate the peak-period load the site is expected to experience:

Step 3: Adjust the peak-period load estimate:

Using the example adjustment factors provided earlier:

Step 4: Identify a peak-period page response-time limit:

The customer has identified 4 seconds as the desired peak-period page response-time limit.

Step 5: Look up the estimated number of servers required from ColdFusion Tack2++ test data:

Using the ColdFusion Tack2++ cluster test data in Figure 3 and Figure 4, an attempted Look up of 1,555,556 page views per hour at 4 seconds reveals the 8 server cluster tested serviced 1,079,978 page views at 4 seconds, less than required for this customer’s application.

Since the ColdFusion Tack2++ application proved scalable across server machines, the estimated number of servers can be calculated by dividing the single server load capacity for 4 seconds from Figure 9 (134,988) into the adjusted peak-period load estimate:

EXAMPLE 2: Using HTTP Hits

A customer is deploying a ColdFusion application and their best estimates put the expected load at a rate of 225 million HTTP hits per month. The customer wants to maintain an average server page response time of no more than about 4 seconds during peak periods. They are expecting to see most of this load distributed somewhat evenly throughout a 6-hour period each day of the month.

The customer believes the application makes “medium-heavy” use of server resources compared to the ColdFusion Tack2++ application tested in this Fact Sheet.

To estimate the number servers required to adequately service this load within the desired response time periods during peak load:

Step 1: Select a load metric to use for sizing:

The customer has estimated load in HTTP hits.

Step 2: Estimate the peak-period load the site is expected to experience: