Understanding the effects of memory starvation.
This is the third part in a series of posts about Wordpress and performance. In part 1,
we took a look at Wordpress in general. In part 2 and part 2.5 we reviewed a couple of popular caching plugins that can boost performance. In this part, we'll start looking at how various plugins can have a negative effect on performance and if anything can be done about it.
In the comments for one of the previous posts in this series, Yaakov Albietz asked us if we used our own service www.loadimpact.com for the tests. I realize that I haven't been that obvious about that, but yes, absolutely, we're exlusively using our own service. The cool thing is that so can you! If you're curious about how your own web site handles load, take it for a spin using our service. It's free.
We started out by looking for plugins that could have a negative effect on Wordpress performance, thinking, what are the typical properties of a bad performer plugin? Not so obvious as one could think. We installed, tested and tinkered with plenty of suspects without finding anything really interesting to report on. But as it happens, a friend of a friend had just installed the Wordpress Multi Language plugin and noted some performance issues. Worth taking a look at.
The plugin in question is Wordpress Multi Language (WPML). It's got a high rating among the Wordpress community wich makes it even more interesting to have look at. Said and done, we installed WPML and had it for a spin.
The installation is really straight forward. As long as your file permissions are set up correctly and the Wordpress database user have permissions to create tables, it's a 5-6 click process. Install, activate, select default language and at least one additional language and your done. We're eager to test, so as soon as we had the software in place, we did our first test run on our 10 post Wordpress test blog. Here's the graph:
Ops! The baseline tests we did for this Wordpress installation gave a 1220 ms response time when using 50 concurrent users. We're looking at something completely different here. At 40 concurrent users we're getting 2120 ms and at 50 users we're all the way up to 5.6 seconds or 5600 ms. That needs to be examined a bit more.
Our first suspicion was that WPML would put additional load on the MySQL server. Our analysis was actually quite simple. For each page that needs to be rendered, Wordpress now have to check if any of the posts or pages that appears on that page have a translated version for the selected language. WPML handles that magic by hooking into the main Wordpress loop. The hook rewrites the MySQL query about to be sent to the database so that instead of a simple "select foo from bar" statement (over simplified), it's a more complex JOIN that would typically require more work from the database engine. A prime performance degradation suspect unless it's carefully written and matched with sensible indexes.
So we reran the test. While that test was running we sat down and had a look at the server to see if we could easily spot the problem. In this case, looking at the server means log in via ssh and run the top command (if it had been a Microsoft Windows box, we'd probably have used the Sysinternals Process Exporer utility) to see what's there. Typically, we'd want to know if the server is out of CPU power, RAM memory or some combination. We were expecting to see the mysqld process consume lots of CPU and verify our thesis above. By just keeping an unscientific eye on top and writing down the rough numbers while the test was running, we saw a very clear trend but it was not related to heavy mysqld CPU usage:
20 users: 65-75% idle CPU 640 MB free RAM
30 users: 50-55% idle CPU 430 MB free RAM
40 users: 45-50% idle CPU 210 MB free RAM
50 users: 0% idle CPU 32 MB free RAM
As more and more users was added we saw CPU resource usage go up and free memory availability go down, as one would expect. The interesting things is that at 50 users we noted that memory was extremely scarce and that the CPU had no idle time at all. Memory consumption increases in a linear fashion, but CPU usage suddenly peaks. That sudden peak in CPU usage was due to swapping. When the server comes to the point where RAM is running low, it's going to do a lot more swapping to disk and that takes time and eats CPU. With this background information in place, we just had to see what happended when going beyond 50 users:
That's very consistent with what we could have expected. Around 50 concurrent users, the server is out of memory and there's a lot of swapping going on. Increasing the load above 50 users will make the situation even worse. Looking at top during the later stages of this test confirms the picture. The kswapd process is using 66% percent of the server CPU resources and there's a steady queue of apache2 processes waiting to get their share. And let's also notice that mysqld is nowhere to be seen (yes, this image is only showing the first 8 processes, you just have to take my word for it).
The results from this series of tests are not WPML specific but universal. As we put more and more stress on the web server, both memory and CPU consumption will rise. At some point we will reach the limit of what the server can handle and something got to give. When it does, any linear behavior we may have observed will most likely change into something completely different.
There isn't anything wrong with WPML, quite the opposite. It's a great tool for anyone that want a multi language website managed by one of the easiest content management systems out there. But it adds functionality to Wordpress and in order to do so, it uses more server resources. It seems WPML is heavier on memory than on CPU, so we ran out of memory first. It's also interesting to see that WPML is actually quite friendly to the database, at no point during our tests did we see MySQL consume noticeable amounts of CPU.
Conclusion 1: If you're interested in using WPML on your site. Make sure you have enough server RAM. Experience of memory requirements from "plain" Wordpress will not apply. From the top screen shot above, we conclude that one apache2 instance running Wordpress + WPML will consume roughly 17 Mb RAM, we havent examined how that differs with number of posts, number of comments etc, so lets use 20Mb as an estimate. If your server is set up to handle 50 such processes at the same time, you're looking at 1000 Mb just for Apache. So bring out your calculators and calculate how much memory your will need for your server by multiplying the peak number of users you expect with 20.
Conclusion 2: This blog post turned out a little different that we first expected and instead of blaming on poor database design we ended up realizing that we were watching a classic case of memory starvation. As it turned out, we also showed how we could use our load testing service to provide a reliable source of traffic volume to create an environment where we could watch the problem as it happens. Good stuff, something that we will appear as a separate blog post shortly.
Feedback
We want to know what you think. Are there any other specific plugins that you want to see tested? Should we focus on tests with more users, more posts in the blog, more comments? Please comment on this post and tell us what you think.
This week saw the Red Herring Top 100 Europe awards take place in Paris. Load Impact was one of the 200 finalists chosen to present and on thursday, we were declared to be one of the 100 winners of the prestigious award.
