Newsletter sign-up

View all newsletters

Sign up for our Enterprise Java Newsletter

Enterprise Java

JMeter tips

Improve the quality of your JMeter scripts

By Chi-chang Kung, JavaWorld.com, 07/11/05

Page 4 of 6

In the context of Web applications, response time refers to the time elapsed between the submission of a request and the receipt of the resulting HTML. Technically, response time should include time for the browser to render the HTML page, but a browser typically displays the page piece by piece, making the perceived response time less. In addition, typically, a load-test tool calculates the response time without considering rendering time. Therefore, for practical purposes of performance testing, we adopt the definition described above. If in doubt, add a constant to the measured response time, say 0.5 seconds.

There is a set of well-known rules for determining response time criteria:

• Users do not notice a delay of less than 0.1 second
• A delay of less than 1 second does not interrupt a user's flow of thought, but some delay is noticed
• Users will still wait for the response if it is delayed by less than 10 seconds
• After 10 seconds, users lose focus and start doing something else

These thresholds are well known and won't change since they are directly related to the cognitive characteristics of humans. Though you should set your response-time requirements in accordance with these rules, you should also adjust them for your particular application. For example, Amazon.com's homepage abides by the rules above, but because it prefers a more stylistic look, it sacrifices a little response time.

At first glance, there appears to be two different ways to specify response-time requirements:

• Average response time
• Absolute response time; that is, the response times of all responses must be under the threshold

Specifying average response-time requirements is straightforward, but the fact that this requirement fails to take into account data variation is disturbing. What if the response time of 20 percent of the samples is more than three times the average? Note that JMeter calculates the average response time as well as the standard deviation for you in the Graph Results listener.

On the other hand, the absolute response-time requirement is quite stringent and statistically not practical. What if only 0.5 percent of the samples failed to pass the tests? Again, this is related to sampling variation. Fortunately, a rigorous statistical method does consider sampling variation: the confidence interval analysis.

Let's review basic statistics before going further.

The central limit theorem

The central limit theorem states that if the population distribution has mean μ and standard deviation σ, then, for sufficiently large n (>30), the sampling distribution of the sampling mean is approximately normal, with mean μ_mean = μ and standard deviation σ_mean = σ/√n.

Note that the distribution of the sampling mean is normal. The distribution of the sampling itself is not necessarily normal. That is, if you run your test script many times, the distribution of the resulting average response times will be normal.

Figures 5 and 6 below show two normal distributions. In our context, the horizontal axis is the sampling mean of response time, shifted so the population mean is at the origin. Figure 5 shows that 90 percent of the time, the sampling means are within the interval ±Zσ, where Z=1.645 and σ is the standard deviation. Figure 6 shows the 99-percent case, where Z=2.576. For a given probability, say 90 percent, we can look up the corresponding Z value with a normal curve and vice versa.

 

About Us | Advertise | Contact Us | Terms of Service/Privacy

Copyright, 2006-2008 Network World, Inc. All rights reserved.