LoadImpact

Load Impact executes 100,000th load test!

We are extremely happy to announce that we recently passed the magic number: 100,000 load tests have been executed since launch early 2009! This is an incredible achievement, and we want to thank all our users for the huge interest you have shown in Load Impact. A hundred thousand load tests is just incredible, and something we never dreamed we would reach this quickly when we launched the service just a little over a year ago.

This blog entry is actually a bit late (as usual), as we executed load test #100,000 this wednesday (the 14th). This means that we have a lucky winner of our iPad competition also. All registered Load Impact users got the opportunity to guess when we would execute load test #100,000, and the one who got closest would win an iPad. The lucky winner is Ben Lamb from the UK (you will also be notified by email, Ben), who guessed that the test would start at 11:00 on the 14th. The test actually started at 7 pm, so Ben was only 8 hours off.

However, we also have another winner! We noticed that one contestant had actually made a better guess than Ben. Vladimir Mischenko from the Ukraine also got the day right, and his guess was 11:07:23!  (Only 7 minutes from Ben's guess. They must both have the same astrologer, or something). The only problem with Vladimir's guess was that he thought the test would start on April the 14th, 2009!

Of course, one could argue that this is our fault for providing a web input form where the default year selected was 2009 - pretty silly for a contest that was launched in 2010. But we prefer to pretend that it was by design. We just wanted to see if people were alert. However, we feel a bit sorry for Vladimir also, so we have decided to give him an iPod touch as a consolation prize (you will also be notified by email, Vladimir).

To all our users we would like to say thank you for your support in making Load Impact the world's most popular load testing tool! Expect to see some nice new features and improvements to the service this year. We are, for instance, very excited about our new load generator application (contact support@loadimpact.com if you want to try it out early) that is being beta-tested right now. It is programmable using Lua and which will allow fully dynamic transactions. More about that later.

Thank you everybody, and don't forget: load testing makes your site better!

Fast sites get better search engine ranking

Late 2009, Matt Cutts from Google, made a speech were he said that Google might start using page speed as a factor when ranking search results, meaning that a faster web page would get a higher Google ranking than a slower one. Google were then already using landing page speed as a component when calculating Adwords quality scores, and from there the step was probably not so great to start using it when presenting search results also.

On friday, the Google webmaster blog announced that Google is now actively "using site speed in web search ranking". The fact that the world's most popular search engine thinks web site performance is a key factor to user satisfaction means a lot of people who previously were not too bothered with performance are now likely to start scrambling to improve the performance of their sites. We are of course only too happy about this, as web site speed - testing and optimization - is what we are all about.

In practise, this means that a fast webpage will now get a higher search ranking than a slow one. Or, more accurately, a fast webpage will be slightly favoured by Google in the comparison with a slower webpage. There are of course a whole range of other factors used when computing the ranking of a page.

As a web site owner or developer, it is going to be ever more important to keep your site and your pages fast and responsive. There have been several reports about how slow sites will cause you to lose business. If your web service is slow, your users are likely to abandon you for a competitor whose service is faster and more responsive. When industry players like Google also openly start favouring speed it is a clear message that now speed is everybody's concern.

What to do?  Well, if you're a web developer you should start thinking about performance when writing code. This is not something you can do as an afterthought (or you can, but it will take a tremenduous amount of extra time) - it has to be part of your creative process.

The trick isn't to write code that performs well when you test it by yourself. The trick is to write code that scales - code that performs well when there are 10, 50 or 100 people accessing your site at the same time. Many developers aren't very good at this, because it requires a different mindset than just writing code "that works". Instead of just thinking "will this work?" you have to start thinking "will this work with 1,000 users on the site?".

It doesn't have to look like this. A software that scales well
is like a road that gets wider when traffic increases.

Both as a web developer and a site owner you will want to test your web pages to see that they perform well and that they scale well. There are various tools, techniques and methods you can use. In the upcoming weeks we will be writing a couple of articles on how to improve performance of your site, aimed both at developers and site owners. Keep an eye on this space.

Increased data transfer quotas for Load Impact premium accounts

The data transfer quotas for Load Impact Premium users (Basic, Professional, Advanced) have been increased to allow more testing using the premium accounts.

The reason to have quotas in the first place has been to on one hand prevent abuse and on the other hand to restrict resource usage. As we have seen that there haven't been a lot of abuse issues surrounding the service, and we have plenty of spare capacity to run tests, we have decided to increase the quotas in order to allow our premium users to get as much testing done as possible.

Previously, these were the data transfer limits:

Load Impact BASIC:

- 40 GB data transfer per 30 days
- 20 GB per target IP per 30 days
- 5 GB per target IP per 24 hours

Load Impact PROFESSIONAL:

- 100 GB data transfer per 30 days
- 50 GB per target IP per 30 days
- 15 GB per target IP per 24 hours

Load Impact ADVANCED:

- 500 GB data transfer per 30 days
- 200 GB per target IP per 30 days
- 50 GB per target IP per 24 hours

Now, the data transfer limits are:

Load Impact BASIC:

- 50 GB data transfer per 30 days
- 50 GB per target IP per 30 days
- 10 GB per target IP per 24 hours

Load Impact PROFESSIONAL:

- 300 GB data transfer per 30 days
- 300 GB per target IP per 30 days
- 50 GB per target IP per 24 hours

Load Impact ADVANCED:

- 1000 GB data transfer per 30 days
- 1000 GB per target IP per 30 days
- 200 GB per target IP per 24 hours

Read more about the usage quotas for the different account types

Understanding the effects of memory starvation.

This is the third part in a series of posts about Wordpress and performance. In part 1,
we took a look at Wordpress in general. In part 2 and part 2.5 we reviewed a couple of popular caching plugins that can boost performance. In this part, we'll start looking at how various plugins can have a negative effect on performance and if anything can be done about it.

In the comments for one of the previous posts in this series, Yaakov Albietz asked us if we used our own service www.loadimpact.com for the tests. I realize that I haven't been that obvious about that, but yes, absolutely, we're exlusively using our own service. The cool thing is that so can you! If you're curious about how your own web site handles load, take it for a spin using our service. It's free.

We started out by looking for plugins that could have a negative effect on Wordpress performance, thinking, what are the typical properties of a bad performer plugin? Not so obvious as one could think. We installed, tested and tinkered with plenty of suspects without finding anything really interesting to report on. But as it happens, a friend of a friend had just installed the Wordpress Multi Language plugin and noted some performance issues. Worth taking a look at.

The plugin in question is Wordpress Multi Language (WPML). It's got a high rating among the Wordpress community wich makes it even more interesting to have look at. Said and done, we installed WPML and had it for a spin.

The installation is really straight forward. As long as your file permissions are set up correctly and the Wordpress database user have permissions to create tables, it's a 5-6 click process. Install, activate, select default language and at least one additional language and your done. We're eager to test, so as soon as we had the software in place, we did our first test run on our 10 post Wordpress test blog. Here's the graph:



Ops! The baseline tests we did for this Wordpress installation gave a 1220 ms response time when using 50 concurrent users. We're looking at something completely different here. At 40 concurrent users we're getting 2120 ms and at 50 users we're all the way up to 5.6 seconds or 5600 ms. That needs to be examined a bit more.

Our first suspicion was that WPML would put additional load on the MySQL server. Our analysis was actually quite simple. For each page that needs to be rendered, Wordpress now have to check if any of the posts or pages that appears on that page have a translated version for the selected language. WPML handles that magic by hooking into the main Wordpress loop. The hook rewrites the MySQL query about to be sent to the database so that instead of a simple "select foo from bar" statement (over simplified), it's a more complex JOIN that would typically require more work from the database engine. A prime performance degradation suspect unless it's carefully written and matched with sensible indexes.

So we reran the test. While that test was running we sat down and had a look at the server to see if we could easily spot the problem. In this case, looking at the server means log in via ssh and run the top command (if it had been a Microsoft Windows box, we'd probably have used the Sysinternals Process Exporer utility) to see what's there. Typically, we'd want to know if the server is out of CPU power, RAM memory or some combination. We were expecting to see the mysqld process consume lots of CPU and verify our thesis above. By just keeping an unscientific eye on top and writing down the rough numbers while the test was running, we saw a very clear trend but it was not related to heavy mysqld CPU usage:

20 users: 65-75% idle CPU 640 MB free RAM
30 users: 50-55% idle CPU 430 MB free RAM
40 users: 45-50% idle CPU 210 MB free RAM
50 users: 0%   idle CPU  32 MB free RAM

As more and more users was added we saw CPU resource usage go up and free memory availability go down, as one would expect. The interesting things is that at 50 users we noted that memory was extremely scarce and that the CPU had no idle time at all. Memory consumption increases in a linear fashion, but CPU usage suddenly peaks. That sudden peak in CPU usage was due to swapping. When the server comes to the point where RAM is running low, it's going to do a lot more swapping to disk and that takes time and eats CPU. With this background information in place, we just had to see what happended when going beyond 50 users:

That's very consistent with what we could have expected. Around 50 concurrent users, the server is out of memory and there's a lot of swapping going on. Increasing the load above 50 users will make the situation even worse. Looking at top during the later stages of this test confirms the picture. The kswapd process is using 66% percent of the server CPU resources and there's a steady queue of apache2 processes waiting to get their share. And let's also notice that mysqld is nowhere to be seen (yes, this image is only showing the first 8 processes, you just have to take my word for it).

The results from this series of tests are not WPML specific but universal. As we put more and more stress on the web server, both memory and CPU consumption will rise. At some point we will reach the limit of what the server can handle and something got to give. When it does, any linear behavior we may have observed will most likely change into something completely different.

There isn't anything wrong with WPML, quite the opposite. It's a great tool for anyone that want a multi language website managed by one of the easiest content management systems out there. But it adds functionality to Wordpress and in order to do so, it uses more server resources. It seems WPML is heavier on memory than on CPU, so we ran out of memory first. It's also interesting to see that WPML is actually quite friendly to the database, at no point during our tests did we see MySQL consume noticeable amounts of CPU.

Conclusion 1: If you're interested in using WPML on your site. Make sure you have enough server RAM. Experience of memory requirements from "plain" Wordpress will not apply. From the top screen shot above, we conclude that one apache2 instance running Wordpress + WPML will consume roughly 17 Mb RAM, we havent examined how that differs with number of posts, number of comments etc, so lets use 20Mb as an estimate. If your server is set up to handle 50 such processes at the same time, you're looking at 1000 Mb just for Apache. So bring out your calculators and calculate how much memory your will need for your server by multiplying the peak number of users you expect with 20.

Conclusion 2: This blog post turned out a little different that we first expected and instead of blaming on poor database design we ended up realizing that we were watching a classic case of memory starvation. As it turned out, we also showed how we could use our load testing service to provide a reliable source of traffic volume to create an environment where we could watch the problem as it happens. Good stuff, something that we will appear as a separate blog post shortly.

Feedback

We want to know what you think. Are there any other specific plugins that you want to see tested? Should we focus on tests with more users, more posts in the blog, more comments? Please comment on this post and tell us what you think.

This is the second and a half part in a series of posts about Wordpress and performance. In part 1,
we took a look at Wordpress in general. In part 2 we reviewed a couple of popular caching plugins that can boost performance. In this follow up post, we'll tell you a bit about what we learned after part 2 was published.

Revisiting W3 Total Cache

After publishing our results in part 2, we received a concerned Twitter message from Frederick Townes, the W3 Total Cache (W3TC) author. He thought we had done something wrong since the Disk enhanced cache mechanism used in W3TC should be at least as effective as the Wordpress Super Cache plugin (WPSC). After a brief Twitter discussion, we understood that he was absolutely right. The mod_rewrite magic that WPSC uses to achieve the amazing results was indeed present in W3TC as well (and I might as well add that the mod_rewrite rules added by Freredick's plugin is neater than the ones added by the test winner).

The mistake we made in our first test is that we didn't realize the significant difference between the two different disk based page caching methods available. There's "Basic" caching which is the one we tested, and there's "Enhanced mode". In Basic mode, W3TC will work pretty much the same way as the standard wp-cache plugin which involves invoking a PHP script. In our server benchmark, we've already seen that our server will consume in the region of 80ms for doing that so we're glad if we could avoid it in the elegant manner that Wordpress Super Cache does.

In Enhanced mode, surprise surprise, avoiding invoking PHP is exactly what W3 Total Cache will do. The basic concept is the same in W3TC and WPSC, both plugins will add a bunch of lines to the .htaccess file that tells Apache2 to go look for a static copy of (a.k.a cached) version of the requested file/resource. And as noted above, W3TC does this with a slightly more elegant addition to .htaccess. In our defense, even though the documentation provided with W3TC is good, we didn't find anything in particular that explained this significant difference between Basic and Enhanced.

How Load Impact can affect the results

Naturally, we took W3TC back to the lab to see how fast it is in enhanced mode. But before telling you about the results, we want to explain a few details about how Load Impact works. When we ask Load Impact to simulate the load of 50 concurrent web users, that is exactly what Load Impact will do. The second the test starts, exactly 50 virtual users will load the page at the same time and Load Impact will note how long time it takes before the web server responds and the content is completely transferred. Then, each virtual user will make a random pause and try again. Depending on the accuracy settings for the test, this will be repeated over and over again. In a "Fast" test, there will be very few repetitions and in a "Accurate" test, there will be lots and lots of repetitions. The more repetitions, the more data points to use when calculating the average load time. This configuration setting allows users to prioritize test completion time over accuracy if they want to. This behavior actually have some impact when testing cache plugins. When we test, the first time when 50 virtual users comes storming to the test web server at once Apache will fire up as many child processes as it's configured for, 30 in our case. All of these processes will go look in the cache and quite likely discover that there is no cached version of the requested file. So PHP is invoked, Wordpress will generate the page and the cache plugin kicks in and stores the rendered version of the page in the cache. Not only does creating the cached version take more time than a normal page request does, in our scenario, there's a risk that this is done up to 30 times. And to make things even worse, 30 child processes writing to a file based cache exactly the same time will cause a lot of file locking problems that will end up taking even more time.

The conclusion is that depending on the state of the cache when the test begins, the response time of the first 30 data points may vary. And this is exactly what we saw when we took W3 Total Cache back to the lab.

Testing W3 Total Cache again

We retested W3TC again and arrived at these numbers:

Wordpress baseline: 1220 ms

W3 Total Cache (basic disk): 256 ms (-79.0%)

W3 Total Cache (enhanced disk): 188 ms (-84.6%)

That's a solid improvement so we contacted Frederick again with the good news only to be turned down again, "something is still wrong" he told us. Then we redid the test for Enhanced mode  and over again with minor tweaks to the W3TC settings. After every tweak, we cleared the cache so that any cached pages specifics wouldn't interfere with the current settings. We saw slightly higher average load times as well as slightly lower, but we were never close to the 112 ms record set when using the Wordpress Super Cahce plugin. Until the "warm vs cold" cache issue hit us and we did a test with a warm cache. And boom! The average load time decreased all the way down to 109 ms, better than what WPSC would acheive. So let's add how W3TC performs using enhanced disk caching:

Using Enhanced disk cache:

Average load time 50 users: 109 ms

Baseline difference: -1111 ms

Baseline difference %: -91.1%

Summary

Results

Before updating the results table,  we retested the other results as well, but the number we ended up with in the retests was all within a 5ms difference from the original test result, so we're sticking with the results from our first round of tests. But we're reducing to using just 2 significant figures: 

That's it.