Load Impact

How do I interpret my graph?  Is it good or bad?

Interpreting test results can be tricky. If a result is good or bad depends entirely on what you are testing, and how, aswell as what your requirements are. Also, individual test results can vary, making interpretation harder.

The first, and perhaps most important advice we can give you is that the more tests you run, the more you will learn.

Running a single test, once, with a single configuration, and then assuming you have the complete picture is usually a bad idea. There are too many factors that affect the performance of a site. Human brains, however, are extremely good at picking out patterns in a large set of input data, and that means that if you run a lot of tests, you will be able to get a very good overall picture of how your site performs. Take the statistical route, and collect the results from many different tests, and you will learn a lot.

Having stated the importance of running several tests, at different times and with different configurations, we will try to explain why graphs look like they do and what you should think about when you encounter a certain type of graph.


Also - be sure to read What are the values that are plotted in the graphs?


First of all, let's show the ideal graph: the exponentially growing graph



This graph is ideal from a testing perspective because it shows us the performance limits of the system we are testing (the target system). It clearly shows that the target system is running out of some critical resource at the load levels we are subjecting it to. The load levels used here are 30,60,90,120 and 150 clients. The response time is increasing in a linear fashion up until 120 clients, when it suddenly starts increasing exponentially instead. This particular target system therefore has a breakpoint at 120 clients - it cannot scale much beyond that point. The response time up until 120 clients is at or below 1 second. At 150 clients it is 3 seconds and at 180 clients it goes beyond 10 seconds.

From this graph we can deduce that the target system can handle loads up to 120 clients without too much difficulties, but then things start falling apart very quickly. Most every system has a performance curve that looks like this, it is usually just a matter of finding the right load levels and the right type of testing to subject the system to, in order to conjure up this curve. When you have this curve you get a pretty good idea of the limits of your system.

Exponential growth basically means that something is growing very fast - you can read more about it at Wikipedia


So what if the curve isn't a nice, exponential curve then?

Here is another type of graph: the straight graph



The straight graph shows a system that is using up some resource, and therefore response time is increasing with increased load, but performance degrades gracefully (or "linearly"). Graceful degradation is a very good characteristic for a system, and something that is difficult to achieve technically. Many software systems out there will not achieve graceful degradation at all. Some will be able to degrade gracefully for a short while and then convert to exponential performance degradation once they reach a critical point (like what happened in our ideal graph above).

One common situation in which you will observe graceful/linear degradation of performance, is when your system is running out of network bandwidth. Because TCP/IP is so well designed, it allows graceful degradation of performance for quite some time before the exponential curve takes over.

Note that graceful/linear degradation can still be bad, if the curve is steep - that means you will still run into unacceptable response times pretty quickly. Therefore, you should look at the response time values reported and how fast they are increasing. If you see that you will quickly run into unacceptable response times, then the curve tells you your system's performance is not good enough. If, on the other hand, the curve is growing quite slowly, then you should increase the load level to try and see if there is an exponential curve somewhere ahead, at higher load levels.


What if my curve is falling, then rising? 

This is the falling and rising graph:



This graph shows that we are not subjecting the target system to enough load to get good results. There may be several reasons why the response times are decreasing at first - here is an article about that - but the important thing is that the graph shows you that at/beyond 40 clients, you are starting to actually load the target system, causing response times to increase. In this particular case you should reconfigure your test to start at 40 clients and go up to maybe 100, then rerun it and see how it turns out.

Here is another example of such a graph from a test that was being run with the load levels 1, 4, 7, 10 and 13 clients:



Then we changed the test to run with 10, 20, 30 and 40 clients instead, and got a different result graph:



As we can see this looks like the beginnings of the old exponential graph. If we ran the test with 50, 60, 70 clients we would probably see quite substantial response time growth.


What if my graph is completely flat?

The flat graph:



This usually means you are nowhere near being able to stress the target system. If you try to run a load test on google.com you will get a flat curve. Their site is powerful enough that any change in response times as a result of the load we generate is all but impossible to measure. If your site runs on powerful servers, with lots of Internet bandwidth, or if your system is very efficient you can also get a fairly flat curve. At least running the free test that only simulates up to 50 simultaneous clients, this is not too unlikely to happen.

The solution is to test your own system instead of Google. If you already were, then you probably need to increase the load you are using.


What if my graph looks really strange?  Like rising, falling, rising, falling in a completely random manner?

Sometimes graphs can be misleading. For example: if you get the response time values 210, 218, 213, 220, 214 they will result in a graph that looks as if it is rising-falling-rising-falling but the differences between the values are actually small enough to fall within error margins, so an experienced operator would likely judge the graph to be "flat".

Last edited by Ragnar (2009-03-29 22:39:55)