Developer Center Article

In March of 2002, Oracle modified the Pet Store application and published revised results for Pet Store on their J2EE application server. While they were trading barbs with Microsoft, Oracle also made comparisons with other vendors such as BEA. BEA did their own test and published their results. In May 2002, Microsoft published new results from a third-party testing company (VeriTest) backing up their conclusions.

Because Pet Market runs on ColdFusion MX, and the Flash front-ends run on J2EE and Microsoft .NET back ends, the team set aside performance comparisons between platforms and focused on the usability and efficiency of the rich client front-end.

For Pet Market, the team focused on a few key metrics:

In this article, we will make comparisons for the first three metrics between the Pet Market blueprint application, Java Pet Store, and Microsoft .NET Pet Shop. However, in the interests of staying above the fray, the team tested Pet Market scalability in absolute terms. To do justice to an "apples to apples" comparison in performance across platforms, operating systems, databases, and applications is outside the scope of this release.

The conclusions:

Rich Internet Applications consume less bandwidth
Rich Internet Applications built using Macromedia Flash MX and Macromedia Flash Remoting consume less bandwidth. This is primarily a result of two factors:

Using a standard list of tasks to perform in each implementation, the team measured the bandwidth consumed by the Pet Market blueprint application running on ColdFusion MX, the Microsoft .NET Pet Shop, and the Java Pet Store. The team used the Microsoft WAS (an automated testing tool also known as "homer") to report kilobytes sent and ran four different scenarios, three times each, across each implementation. Total bytes sent and received for each script and each version of the application were recorded. These were the scenarios:

Details on the individual tasks of the scripts can be found in the references at the end of this document.

This chart summarizes the results of the bandwidth comparison.

In each case, Pet Market consumed less bandwidth than the Java Pet Store and the Microsoft .NET Pet Shop. In particular, scripts 1, 3, and 4 (Extended browsing, Indecisive shopping, and Extended browsing and purchase experience) showed a significant advantage because of the increased number of requests and responses from the server. Pet Shop and Pet Store achieved comparable results to Pet Market only in script 2 (the Directed shopping experience) where the number of requests and responses was minimized by the test script.

In general, Pet Market consumes less bandwidth because the server sends a smaller amount of information over the network to the rich client front-end. Because the Pet Market front-end is already loaded, it executes code for formatting and presentation, so only data needs to travel down the wire. In contrast, the Java Pet Store and .NET Pet Shop have to send both data and presentation code over the wire.

This illustrates a fundamental dynamic of Pet Market: the longer the application is in use, the greater the benefit in terms of bandwidth savings. More importantly, once the files for Pet Market are cached on the local machine, the bandwidth advantage increases because Pet Market does not require an additional initial load.

However, it is important to examine the bandwidth consumed in the initial load of the Macromedia Flash front-end of Pet Market. The table below states the amount of data stored (in kilobytes) in the browser cache for the three applications after the application is first launched.

Comparing Pet Market with other e-commerce sites (as measured by the kilobytes of files in the browser cache) shows that Pet Market is within the range of acceptability in terms of initial load.

Both the Java Pet Store and the Microsoft .NET Pet Shop offer a spartan online shopping experience compared to Pet Market or e-commerce stalwarts like Amazon.com, Barnes and Noble, or Buy.com.

However, the fact remains that the initial load of Pet Market consumes more bandwidth than the Java Pet Store or the .NET Pet Shop, despite the much richer browsing experience. But this bandwidth hit is a one-time cost, because once Pet Market is cached, the application only needs to be initialized. Most importantly, as the application is used, the operating bandwidth advantage for Pet Market rapidly makes up this cost.

The table below summarizes the bandwidth saved by Pet Market (in kilobytes) when compared to the Java Pet Store and Microsoft .NET Pet Shop.

In short, each script reveals a significant bandwidth advantage in favor of Pet Market. While these savings are not significant taken individually, over time and multiple users this can add up to significant savings.

More importantly, the Pet Market application provides a richer experience within a smaller bandwidth footprint. For each individual type of pet, Pet Market loads a ~5K image and short textual description. There are images and descriptions for clown fish, goldfish, hippo tang, koi, and yellow sailfin tang in the fish category.

In contrast, the Java Pet Store and .NET Pet Shop do not load product images at all—and still they are less bandwidth-efficient. Both Pet Store and Pet Shop consume bandwidth with approximately 6K of HTML code required to be loaded for each type of pet. In Pet Market, the image and text data provide a shopper with much more useful information for a lower bandwidth cost (see below).

Pet Market provides more graphical and text information for each type of pet.	Pet Shop provides no graphical information for each type of pet.

While there are additional file size and bandwidth optimizations that could be applied to Pet Market (shorter variable names, smaller images, optimized code for some user interface components) to increase the advantage Pet Market enjoys, the Pet Market team focused on making this release of the application a good learning tool.

In summary, Pet Market provides a richer experience in a smaller amount of bandwidth, simply because Pet Market does not need to load a new page formatting code for each piece of the application.

Usability metrics
Using the same four task-oriented scenarios, the Pet Market team measured how long it took for a user to complete each scenario. The usability reviews by Jakob Nielsen and User Interface Engineering cover the reasons for increased user success and reduced time to complete the scripts, but the team wanted to measure the actual time taken.

The same four scripts were used, and the team measured the time taken by users to complete each script with a stopwatch and averaged the results.

The results are presented below:


Two of the scripts, 1 and 4 (Extended browsing experience without purchase and Extended browsing and purchase experience) demonstrate the dramatic usability advantage that Pet Market enjoys over the Java Pet Store and the .NET Pet Shop. In the Pet Shop and Pet Store, users exploring the product catalog underwent a long series of page loads as they navigated up and down the hierarchy to view different types of animals. This kind of "backtracking" adds significant overhead.

In script 4, the changes in context (the checkout process, search, history) required by page reloads for the Java Pet Store and Microsoft .NET Pet Shop added additional time (above and beyond the page load) for the task.

The Java Pet Store and Microsoft .NET Pet Shop were almost as usable as Pet Market in scripts 2 and 3 (Directed shopping experience, and indecisive shopping experience). This is due to the relatively simple tasks and the few number of page loads required in the script.

To be fair, the Java Pet Store and the .NET Pet Shop were not designed for usability, while Pet Market went through extensive design reviews and user testing. However, this data does indicate the kind of impact that Rich Internet Applications and user centered development processes can have on usability.

Lines of code and developer productivity
The following chart compares the lines of code in the ColdFusion MX and Pet Market implementation of Pet Market application to the Java Pet Store and the .NET Pet Shop. The types of code are divided into configuration, database, middle tier, and presentation tier. As might be expected, Pet Market has a fair amount of code in the Macromedia Flash front-end, while the middle tier is extremely compact when compared to the Java Pet Store and the Microsoft .NET Pet Shop. This is due to the server side markup language of ColdFusion (CFML) which can efficiently encapsulate "glue code" such as database calls required for different database back ends.



Statistics about lines of code are open to interpretation and need to be taken in the context of the implementation. For example, the following are some of the factors that affect the lines of code measurement:

However, it is moderately useful to note that Pet Market compares very favorably to the other two pet stores in terms of lines of code for a much richer and more usable experience.

As a proxy for developer productivity, lines of code metrics can be misleading. However, developer productivity was a key design center of the Macromedia MX product family. The off the shelf interface components in Macromedia Flash MX alone saved the Pet Market team an estimated 3 weeks of development time.

These scalability tests were conducted with ColdFusion MX and Pet Market 1.0 on the following configuration:

Version 1.2 enhancements

Performance tuning
As part of the engineering for the Pet Market 1.2 release, we decided to do some additional work on tuning the performance of the application.  There were three areas that we wanted to evaluate for performance improvements.

Load time
One area that we examined was the loading time of the application.  It turned out that the biggest opportunity for load time reduction was in the Macromedia Flash Player detection logic. The JavaScript code used in this part of the application is heavily commented.  Removing the comments eliminated 10K of text download during application startup.
 
Movie initialization
Once the application is loaded, the various components of the application initialize.  We wanted to reduce the time to first interaction in the application.  Some simple changes sliced 20 to 30% off of the time to first interaction:

• Allow interaction with cart prior to initialization of the top navigation elements.
• Stagger initialization of checkout widget over multiple frames to improve startup time and responsiveness
• Improve Broadcaster class to spread UI events of multiple frames

Flash Remoting API changes
We decided to do a detailed analysis of how the Flash Remoting API calls were used to see if there were opportunities for performance improvement.  We discovered the best opportunity when analyzing the cart.  Whenever our client did a cart add, change, or delete operation, it always asked for the count and dollar total of the items in the cart.  By changing the add, change, and delete API functions to return the count and dollar total, it eliminated one round trip to the server and improved the perceived responsiveness of the cart.

Summary
Measuring the Pet Market against the Java Pet Store and the .NET Pet Shop is a topic that deserves a more comprehensive test suite than our time here allows. While we believe that Pet Market enjoys considerable advantages over page-based user interaction models in terms of usability, there are other advantages to the Rich Internet Application model in terms of bandwidth usage and architecture. Most importantly, the Pet Market team realizes that the best practices for Rich Internet Applications will evolve over time.

While this article provides some core metrics for bandwidth usage, usability, lines of code, and performance, the true measure of Pet Market will be in its ability to help developers explore the new paradigm of Rich Internet Applications.

References
Web application server configuration:

Related links

Scripts for usability tests