LoadImpact

A while ago, we published a blog entry here about browser behaviour and performance where we told the story of how most browsers seem to transmit AJAX POST requests (i.e. HTTP POST requests originated from Javascript code) as two separate TCP packets, while a small number of browsers - most notably recent versions of Firefox - transmit the same request using only a single TCP packet.

This behaviour was discovered by a Yahoo developer originally, and later on investigated and confirmed by Joseph Scott on his blog. We went further and mapped a wide range of browsers on several different operating systems, to find out in what circumstances the request was sent as two packets. The results can be read in a PDF report we created, if you're interested. (If you are not, then you should probably stop reading now, because for non-geeks, this post is only going to get worse from here)

Because we are such mega nerds when it comes to performance stuff, we decided to dig deeper into this, and find out just how big a performance impact the behaviour had. I.e. we wanted to see how much slower a request would be, on average, when the browser transmitted two instead of one TCP packet. Logically, more packets mean more overhead and a bigger risk of one packet getting lost and having to be resent, causing extra delay. Yahoo actually advice people against using POST in Ajax requests because of this behaviour - see Yahoo's ySlow performance rule #15 - Use GET for Ajax requests - but they don't tell you what it means for performance if you do/do not use POST. To us, just like to Yahoo apparently, it seemed rather obvious that sending two packets was worse than sending one packet, but maybe it shouldn't be a general performance recommendation unless it has a significant impact (if not, then other recommendations would perhaps have been more important to get across to ySlow users than this #15).

So, we wanted to see how big the impact actually was. By sending a large number of AJAX requests using both the GET and POST methods, we reasoned that we should be able to see some statistical difference between the response times of the two. Especially if we also simulated some network delay and packet loss on the link, the difference had to be quite noticeable. Our goal was to be able to say e.g. that using GET instead of POST in an Ajax request can reduce average transaction time by up to X% or more, depending on network conditions.

We set up the test, and spent several days running hundreds of thousands of Ajax GET and POST requests using different browsers, and emulating different network characteristics (i.e. network delay and packet loss). In the end we found something we weren't prepared for - the tests showed that sending two packets seemed to slightly outperform sending one packet. (Again, read the PDF report if you want more details) 

After scratching our heads, asking around, double-checking the data, reading up on TCP and its most common implementations, then scratching our heads some more, we still couldn't understand how this came to be. There might have been problems with our setup somehow, but it seems far-fetched that any problems with our setup would give a 2-packet transaction an advantage over a 1-packet transaction. We decided to publish the results and see if anyone could explain what we were seeing, or tell us what we were doing wrong. We have since had a couple of people offer things such as "two packets allow faster resend if the first packet is lost" or "sending more packets might help TCP compute the RTT better". But none of the explanations seem to suggest anything that could outweigh the penalty you get from the fact that the 2-packet transaction suffers twice as much from packet loss as the 1-packet transaction. If it does - maybe that's the assumption we make that is wrong.

At the Velocity 2010 conference, I ran into a well known web performance guru and while talking to him a guy from the Yahoo ySlow team showed up, so I mentioned our findings to them both. The guru said he didn't believe me, and tried to change the subject. The Yahoo guy didn't say anything but walked off as soon as he got the chance. The obvious conclusion from this encounter is that I have to work on my people skills. Or maybe they were just stressed, or suffering from caffeine deficiency (it was a coffee break I think) - who knows? Maybe I was suffering from caffeine deficiency. I hope I don't come off as one of those annoying people who can't stop trying to find faults in stuff other people do, in an attempt to feel superior. Because I am not - I just can't help trying to improve things everywhere, all the time. It's like a drug to me. I have to figure out how things work, and then try to improve them. Have to. I can probably be a pain about it - when people do something really well and expect praise I will usually tell them "Nice! The only thing you might want to change for next time is [whatever]..." and many take that as a sign that I don't like what they've done. For me, constructive criticism is the finest form of praise!

Anyway, enough of the boring introspection and back to the important stuff - investigating the problem! I decided that if people don't believe me, and they don't have time to set up a test bed and run the tests for themselves, the obvious solution is that I set up a test bed they can run tests on remotely. If I can get the critics to observe the strange behaviour first-hand, I might force them to acknowledge that it does happen, if nothing else! Then more people will be likely to try and come up with an explanation. I would be happy to learn that there is some problem with our setup, for instance, because then I would at least know. Not knowing is frustrating. How can I work to improve something unless I understand it first?

So, I have now a test system up and running where anyone can try executing 50 Javascript GET, then 50 Javascript POST operations and see which are faster. I also set up the test on both Apache 2 and Lighttpd, in case this problem is webserver-dependent somehow. I thought, why not name it "Scott's problem", after the Joseph Scott who first bothered to describe it in detail. So the address to the test pages are:

• http://scotts-problem.loadimpact.com (Apache 2)
• http://scotts-problem.loadimpact.com:81 (Lighttpd)

Load Impact has been chosen as one of Europe's 100 hottest companies

This week saw the Red Herring Top 100 Europe awards take place in Paris. Load Impact was one of the 200 finalists chosen to present and on thursday, we were declared to be one of the 100 winners of the prestigious award.

The winners were the companies judged to be the most innovative and to be positioned to grow at an explosive rate. Companies were evaluated on both quantitative and qualitative criteria, such as financial performance, technology innovation, quality of management, execution of strategy and industry integration.

We are of course extremely happy to get even further recognition that we are on the right track. Still, the single biggest positive signal of all is of course, and has always been, our astounding usage statistics. Over 114,000 load tests executed, at the time of writing, is pretty telling. Our users are giving us all the confirmation we could ever ask for, and for that we are ever grateful! We will do our best to live up to all the hype and provide you with the world's best load testing service both now and in the future!

We here at Load Impact are always interested in learning more about browser behaviour, and when we saw Steve Souders new Browserscope project, we really liked it and decided to help out, if we could. Browserscope profiles web browsers, and tries to determine what browsers have support for new features, are suffering from known security issues, or have certain performance-affecting "quirks". It basically runs an extensive test suite on your browser and tells you how good your browser is. Go and check it out, we think it is very cool.

Naturally, we are most interested in the performance-affecting "quirks" browsers have. These mostly fall under the "Network" category in Browserscope (go to browsercope.org and click on the Network tab) and can be stuff like how many concurrent TCP connections a certain browser will open to one and the same host. Internet Explorer 7 will only open max 2 concurrent connections to a webserver, while Internet Explorer 8 will open up to 6 concurrent connections. That means that IE8 can download 6 files at the same time, while IE7 can only download 2 files at the same time. It definitely affects performance, and as such, is very interesting to us (note that the opening of many connections might be a good thing from a client perspective, but worse for the server that has to keep a ton of connections open - we don't advocate opening more connections as a universal way to increase performance...)

We have actually modeled our Web page analyzer on these performance-affecting characteristics that Browserscope has identified for us. Our web page analyzer can emulate different well-known browsers when it analyzes a web page, and the way it does that is by mimicking the behaviours you can read about under the Browserscope Network test category. It emulates browser-specific performance-affecting behavour such as the max number of concurrent TCP connections to open, etc. and thanks to this it can provide fairly accurate predictions of page load times in most major browsers.

Anyway, we decided to try and create a Browserscope test to see what browsers support HTTP trailing headers. This is a feature of HTTP 1.1 that very few browsers have implemented. It means that a server can transmit extra HTTP headers after the HTTP body has been sent. It is useful if, for example, a server generates a large PDF file dynamically and wants to include a checksum (using the Content-MD5 HTTP header), but wants to start sending the file immediately and avoid buffering the whole of the file in RAM. If the browser at the other end support HTTP trailing headers, the server can generate the file while sending it, and while computing the checksum on the fly. Then when the file has been transmitted, the server just add the Content-MD5 HTTP header at the end.

The new test is live at browserscope.org now. It lies under the Network category and is called "Headers in trailers". As you can see, it is only Opera (and Palm Pre!) that seem to support the feature

And now for something really strange

That was a case of finding out whether browsers support a standard HTTP feature or not, but there was also another, more exotic browser characteristic we wanted to dig into. The strange phenomenon that most modern browsers always send Ajax HTTP POST requests as two or more TCP packets. That is, even if the request is small enough to fit into a single packet.

This was apparently first observed by a Yahoo developer, then re-posted a couple of times until Joseph Scott tested various browsers to see which were affected, and found out that most browsers had this tendency, with Firefox being a notable exception. Many have speculated about the possible performance impact of this, and Yahoo have actually incorporated it into their Best Practises for Speeding Up Your Web Site as a separate rule/tip, where they advise people to use GET instead of POST requests in javascript code, because the POST requests may be transmitted using an extra TCP packet. But there is precious little hard data about what this browser behaviour actually does for performance, and we also thought the browser survey done by Joseph Scott was a bit incomplete, so we set out to investigate a bit more.

First, we performed a major survey where we learned that Sea Monkey, Galeon and most versions of Firefox (on various platforms), were the only browsers that sent short HTTP POST requests as a single TCP packet. The rest of the bunch always sent one TCP packet containing just HTTP headers, and one or more TCP packets containing the HTTP body, when a javascript performed an HTTP POST request (note that this  only applies to javascript-generated requests).

Then we wanted to find out how big the performance impact was. We imagined that an Ajax application performing a lot of small HTTP POST requests over a high-latency, high-packetloss link would see its performance deteriorate badly due to the extra time spent resending lost packets, if the user used one of the browsers always sending two packets instead of one. More packets equal more packets lost, which means more waiting for resend timeouts and then more resending of packets.

So, we set up a test bed where we simulated packet loss and network delay between client and server. Then we let the 1-packet browser Firefox compete against the 2-packet browser Safari, to see which one fared the worst. The results were a bit surprising. It turned out that while both browsers saw their average transaction times deteriorate substantially with increased network delay and increased packet loss, Safari did slightly better than Firefox overall. The increase in average transaction time due to packet loss in Safari were not as great as the same increase in Firefox. If you're having a slow day and want more detail, here is a PDF report from the tests.

And you people who aren't very interested in arcane TCP matters can probably stop reading now :-)

Why is sending two packets just as good as, or better than, sending one? I have been discussing this with some networking people on an email list recently, and two possible answers came up:

• if packet #1 sent by Safari is lost, retransmission will occur faster if packet #2 gets delivered, as the receiver will ACK packet #2, whereupon the sender will see that receiver is still waiting for packet #1 and most likely also lower its retransmission timeout for packet #1.
• sending more packets allows TCP to keep better track of network roundtrip times, which are used to set reasonable retransmission timeout values.

What is really interesting here is that the results seem to go against the Yahoo recommendations. There seems to be no discernible performance hit for using POST requests. At least not on the client side. We would love to hear from people who are more familiar with current TCP implementations and who can offer some explanations for this behaviour.

With page speed now a factor affecting your Google ranking, it seems that more web services are interested in having a page analyzer or load testing tool on their website. However, creating your own load testing application takes time, and even integrating open source tools can be a drain on precious resources.

So, we came up with a 5 minute solution to tackle this problem.

We have created an interactive flash object, so it is now possible for you to integrate Load Impact's load testing and page analysis tools on your website. You can even start a load test from your blog. Because each website has a different layout, we went ahead and created 2 different designs for each tool. You can see the two different styles below:

Page Analyzer Tool (Design 1)

Load Testing Tool (Design 2)


Take a peek to see all the flash tools that we have created.

To place these flash objects on your website, you just need to copy and paste the html code available here into your web page or blog. Nifty, huh?

Of course, if you don't mind spending a little more time on integrating our services with yours, do drop us a mail and we will be happy to introduce you to our co-branded Premium Partnership Program, where we offer attractive margins (kick-back) per subscription sold.

Otherwise, feel free to try out the flash tools and let us know what you think!

The background

A few weeks ago, one of the companies I'm partly involved in purchased a very popular website in the Swedish real estate field. The website we bought was essentially a Wordpress site with some interesting customizations. The new website ranks very good in Google and the other major search engines and the amount of organic traffic that flows in to that site is simply fantastic.

As soon as the deal was done, the team copied the website to a new production environment and started working with the graphic appearance so that the new site would resemble the existing old site. Half way through the work, the CEO of started asking questions about the performance. In short, it was terrible. A page would take anything from 4 to 10 seconds to load and sometimes it would even time out entirely. A few different efforts was made to explain and remedy the problem, but it turned out that there was a lot of guessing involved and none of the guesses made was the right one. Yesterday, I did the right thing and took it to the test bench.

Take it to the lab

They say that once you know how to use a hammer really well, all problems starts to look like a nail. I guess that's starting to become true for me, only in my case the hammer is loadimpact.com. My idea was to copy the Wordpress blog in question to an environment where I could examine it under load, so I did just that. I looked in the back of my closet and found this old 2.1 GHz Quad code with 8Gb RAM machine that was just collecting dust (kidding, I use it all the time). The server has a complete LAMP stack so both the web server and the mysql server resides on the same hardware. 

A few short words about how to move a Wordpress installation. It's not complicated, but you can end up with a few really non intuitive problems if you don't get it right. However, if you do it right, it's a 5 minute task (plus the actual time to move large files across the Internet). Here's how I got it right the second time:

Move all Wordpress files to the test environment. There isn't really that many ways to fail but you need to make sure that the web server(I'm assuming everyone uses Apache2 here) on the new host has read access to the destination folder. 

Get a DNS entry for the new destination. In my case, all the customizations made to the blog was assuming that the installation resided in the root folder of the webserver, all images was referred to with src="/images/foo.jpg" etc. So I setup a subdomain that pointed to the web server and set up a new virtual host that pointed directly to the Wordpress installation on the new web server

Copy the database. There's some room for error here. I used phpmyadmin to create a database dump from the production environment with the intention to restore it on a database accessible from the test environment. When restoring it in the test database, I used the command line:

mysql dbname -u username -p  < dumpfile.sql

That went really well, the database was recreated in no time and everything looked really good. But. That doesn't work. Wordpress uses uft-8 as default character set and in the above line the mysql command line tool will assume that the file is ISO-Latin-1. Chances are that it won't really hurt you at all but if it does, the errors will be far from intuitive. I ended up with error messages like:

Warning: array_keys() [function.array-keys]:

The first argument should be an array in wp-includes/widgets.php on line 657

The fact that you're looking at a character encoding problem here isn't really obvious right? But the explanation is farily simple. Worpress uses the PHP internal functions serialize and unserialize to transform complex structures like arrays and objects into strings that are suitable for storing in a database. The format used will for instance describe a string like this: s:10:"räksmörgås". The number 10 tells PHP how many characters to expect in the following string. When exporting the above example into a text file, the file on disk will look something like this: s:10:räksmörgÃ¥s. Now, if mysql thinks that the file you are importing is encoded in ISO-Latin-1, the content of the database will look exactly as it did on disk. Later when Wordpress is asking PHP to unserialize the expression s:10:räksmörgÃ¥s PHP will kindly reply that it can't be done and Wordpress will end up feeding funny looking strings into functions that would expect for instance and array as input parameter. The above error message I got was one of those examples. But the remedy is really simple, import the dumped file with this command instead:

mysql dbname -u username -p --default-character-set utf8 < dumpfile.sql

Adjust to the new environment. The last thing that  needs to be done when moving a Wordpress installation is to make sure Wordpress knows that it have moved now. There are two important places to change. First, the file wp-config.php needs to be updated with the new database credentials. Second, the table wp_options (could use a different prefix than "wp_" in your installation) stores the URL of the blog in two places, you will find them both using the following query:

SELECT * FROM wp_options WHERE option_name in('siteurl','home');

So, with the above steps taken care of, the Wordpress installation now works as expected in my test envornment. It could have been done in 5 minutes, but I ended up spending a little more due to the database character encoding experiment, but at least I learned something from it and hopefully so did you by reading about it.

Take it to the lab

Time to bring out the tool box. In my case, it's loadimpact.com account and a ssh account on the target machine. What I like to do is to put load on the application  and wait for white smoke to appear somewhere. Typically, I'm looking for signs showing that the application is too hard on the database, is using too much CPU or is memory hungry.  Of we go:

Yep, there's a problem with this blog alright. Even at 10 concurrent users, we have response times around 7 seconds. Interestingly though, the load times seem farily constant as we put additional load. At 50 concurrent users, we see about the same average response time as with 30 or 40 users. As this test was running, I was watching the server to see where the white smoke would appear. I didn't really have a favorite suspect so I began with using top:

The screenshot shows what top looked like with 50 concurrent users (remember that the response times here was in the range of 10 seconds). There's a couple of apache2 processes that's actually not that hard on resources. 4% CPU each and 0.3% memory (on a fairly powerful machine though). The mysqld process seem calm, it shows up in the list but uses very little CPU and memory resources. This actually looks fairly good. But let's go over the ususal suspects:

RAM memory actually looks really full, only 200 Mb  free, but that's actually just how Linux works with memory (the same numbers on a Windows machine would be awful). Before I see the kswapd process showing up in top, I won't really consider available RAM to be a problem. So conclusion #1 is that we don't have a problem with RAM memory.

CPU also looks good. None of the visible processes in top uses that much CPU. Also, take a look at the header. 95.1%id actually means that the server CPU is idling 95.1% of the time. Conclusion #1 is also easy, we don't have a problem with CPU usage.

Database usage looks good as well. Most of the work is carried out by the mysqld process that we already saw was very quiet. Since we actually have the web server and the database server on the same hardware, excess database usage would also show up as higher total CPU usage. But to be sure, I'll have a look using the tool mytop:

 mytop shows the currently running queries that mysql is handling at the moment. If we had any queries that took a really long time without consuming a lot of CPU, we'd catch them here. The screenshot above is takes while there was 50 concurrent users hitting the blog and the response times was in the 10 second range. It's clear that the delay in response times has nothing to do with poor database performance.

So, none of my three usuals suspects seem to be guilty this time. What else could there be?

The fourth suspect

The fourth suspect that I came to think about was network performance. Long response times paired with little activity on the server could indicate that there is some other resource on the network that is taking the load. My test server isn't showing any signs of activity, but what about other servers or the network itself? The next handy tool would be to bring out iftop. Here's what the screenshot looks like:

iftop will tell you what other hosts your server is connected to at the moment and how much bandwidth each of those connections are using at the moment. Looking at the list I could explain the reason behind each and everyone of the foreign hosts except for two of them. The last one on the list is an internal host on the same network and that's simply a file server that shouldn't come in to play at all. snik1.gatorhole.com is the server that's putting the load on my machine, so it's only natural that it shows up here. One of the other hosts is the one I'm using ssh from, so that one was also expected. The two I'm curious about was lillamy.ballou.se and s114.loopia.se.

Ballou.se is the domain name of our hosting provider for the new site, I find it interesting that we'd be hitting it with traffic. Loopia.se is the domain name of the OLD hosting provider for the site we're trying to test. That's even more interesting! Perhaps the application have some hard coded configuration that makes it look up things in the old environment!

Any such configuration would either be stored in the database or in the file system so I went there to look. First, I was using grep to search the file system for any references to the old domain name or hosting provider but it turned up blank. Next I was a bit lucky. Knowing that Wordpress stores a lot of information in the wp_options table I did a search for it in phpmyadmin:

SELECT * FROM `wp_lo_options` WHERE option_value like '%lokaler.nu%'

"lokaler.nu" is part of the domain name of this application and the DNS records are actually still pointing to the old server. I got roughly 10 rows back and after looking at them a bit the one that caught my attention was a the configuration for a RSS Widget. This widget would get the latest 6 news items from the blog itself. Instead of using a Widget such as "Recent Posts" or similar, the administrator setting this up had opted to get the news via RSS. Just replacing that Widget for the more natural "Recent posts" was exactly what I was looking for. The next test was much better:

 By removing the RSS Widget and getting the results directly from the internal database was the key. The graph is now much more normal and for a normal 10-20 concurrent users we're in the comfortable 1-2 second range. We still have issues to look at and the next step is to install one of the better Wordpress cache solutions, but the most critical performance issue is resolved. Done.

 Conclusions

So, did we learn anything new today? Well, yes I'd like to think so. First and foremost, using a load testing tool to find out what's wrong with an application is actually a very good idea. Putting load on a web application is a good way to make the performance problems stand out a lot more. In this case, the tools used to examine what's going on on the server all takes a little time to produce interesting numbers. A single snapshot of top is useful, but looking at it for 60 seconds is an order of magnitude better. The same goes for all the other tools involved. Even getting to see the s114.loopia.se name show up in iftop would have been impossible or at least required a lot of luck if we had to generate the traffic using manual refresh in Firefox/Firebug.

Second, if you're using an RSS Widget in your Wordpress blog, please be careful. If you're using it to draw news from your own blog, consider using a Widget that can get data directly from the database instead. If you are pulling news from an external source, make sure you use a working cache module.