Stating Response Time Requirements

Traditionally, response time is often defined as the interval from when a user initiates a request to the instant at which the first part of the response is received at by the application.  However, such a definition is not usually appropriate within a performance related application requirement specification.  The definition of response time must incorporate the behavior, design and architecture of the system under test.   While understanding the concept of response time is critical in all load and performance tests, it is probably most crucial to Load Testing, Performance Testing and Network Sensitivity Testing.

Response time measuring points must be carefully considered because in client server applications, as well as web systems, the first characters returned to the application often does not contribute to the rendering of the screen with the anticipated response, and do not represent the users impression of response time. 

For example response time in a web based booking system, that contains a banner advertising mechanism, may or may not include the time taken to download and display banner adds, depending on your interest in the project. If you are a marketing firm, you would be very interested in banner add display time, but if you were primarily interested in the booking component, then banner adds would not be of much concern.

Also, response time measurements are typically defined at the communications layer, which is very convenient for LoadRunner / VUGen based tests, but may be quite different to what a user experiences on his or her screen.  A user sees what is drawn on a screen and does not see the data transmitted down the communications line.  The display is updated after the computations for rendering the screen have been performed, and those computations may be very sophisticated and take a considerable amount of time.  For response time requirements that are stated in terms of what the user sees on the screen, WinRunner should be used, unless there is a reliable mathematical calculation to translate communications based response time into screen based response time.

It is important that response time is clearly defined, and the response time requirements (or expectations) are stated in such a way to ensure that unacceptable performance is flagged in the load and performance testing process.

One simple suggestion is to state an Average and a 90^th Percentile response time for each group of transactions that are time critical.   In a set of 100 values that are sorted from best to worst, the 90^th percentile simply means the 90^th value in the list.  The specification is as follows:

The above specification, or response time service level agreement, is a reasonably tight specification that is easy to validate against.

For example, a customer 'display order details' transaction was executed 20 times under similar conditions, with response times in seconds, sorted from best to worst, as follows -

    2,2,2,2,2, 2,2,2,2,2, 3,3,3,3,3, 4,10,10,10,20        Average = 4.45 seconds,  90^th Percentile = 10 seconds

The above test would fail when compared against the above stated criteria, as too many transactions were slower than seven seconds, even though the average was less than five seconds.

If the performance requirement was a simple "Average must be less than five seconds" then the test would pass, even though every fifth transaction was ten seconds or slower.

This simple approach can easily extended to include 99th percentile and other percentiles as required for even tighter response time service level agreement specifications.

Send mail to webmaster@loadtest.com.au with questions or comments about this web site.
Copyright © 2004 RPM Solutions Pty Ltd - abn 64 007 217 941             Last modified: August 04, 2004   

The content on this web page represents and demonstrates the way that RPM Solutions Pty Ltd understands and conducts load testing engagements.

This page is not an instructional guide - and RPM Solutions Pty Ltd does not take responsibility for the use of the information provided herein.