LoadImpact

Load Impact nominated as one of the rising stars of the north

SIME, Northern Europe's largest Internet business conference, has nominated LoadImpact.com as one of the possible winners of their grand prize. The yearly event honors excellence in online communication, business practice and technology, recognizes the individuals and organizations responsible and inspire the Scandinavian Internet industry to world-class performance.

For more information, go to the nomination page: http://sime.nu/09/awards/rising-stars-of-the-north/load-impact/

Three new US-based load generator nodes

Today Load Impact has activated three new load generator nodes in the US. They are located in Washington DC, Dallas and Seattle. With the existing Chicago site, this brings our total geographic presence in the US up to four sites, spread out over east-, west-, south and central north america.

View Google map showing US-based load generator nodes

This is an important step in our aim to provide the best online load testing solution to users in the US and all over the world.

The Seattle site will have fairly good connections westwards also, covering east asia and Oceania better than before, while the Dallas site will provide better connectivity to customers in central and south america.

To use the new load generator nodes, you have to be a Load Impact premium customer. That allows you to change the load generator node used when you configure your tests on the edit test page. At the bottom of the test configuration page, you will find the load generator node selection box:

For a list of all the load generator nodes and the IP address ranges they use, see this FAQ entry.

Implementing CSS support for the Page Analyzer

Our new web page analyzer is getting more popular by the day. Just after its release we even had a call from a well-known Internet organization who wanted to license it for their internal use, to gather general statistics about web site performance. They were impressed with the analyzer, but quickly noticed that it had a flaw - it would always load all resources that were referred to in a CSS file, no matter if the resources were needed to render the web page in question, or not.

Many sites today have a single CSS file that is used for the whole site. That CSS will contain references to e.g. various images that are used for backgrounds on different pages on the site. However, a web browser that loads the CSS file, will parse it and realize that while the background object a.jpg is needed on the current page, another background object b.jpg isn't needed. It is used on some other page on the site, but not on the page that is currently being loaded and rendered. The web browser will see this and refrain from loading b.jpg.

What this all means is that our page analyzer would often load a lot more objects (images etc) than a real browser would, which of course meant that the total page load time would be slower than you'd experience with a real browser. In short, our browser emulation would not be very accurate in many cases.

An interesting side note here is that no other page analyzers seem to fully support CSS. We have looked around and found that all available analyzers seem to have this same problem.

We decided to see what we could do about implementing true CSS support in our analyzer, and started by looking at available CSS libraries for Python (the analyzer is written mainly in Python). We found two contenders - CSSutils and CSSEngine. After looking at both, we decided to go with CSSEngine as the code base seemed to be a better starting point for what we wanted to do. The drawback with that library was that it hasn't been maintained the last 3 years or so.

We cleaned up the code a bit, made it more forgiving so it wouldn't croak and die when it encountered the sometimes horrible mess that constitutes CSS files on the Internet, and made it CSS3-aware (CSS version 3 is fairly new, so the library had no support for it).

Then we started testing our new CSS parsing capabilities, and found to our disappointment that it was slow. For the sites out there with large CSS files, the parser was really slow. Glacial, in fact. What to do? We started out by using Cython to convert the Python code to C and then compile it. This improved speed 3-4 times, but it was still too slow. Parsing a really large CSS file would take 15 seconds or more, and that wasn't acceptable.

So the hunt went on to improve speed. It turned out that the culprit was the CSS parsing, mainly the execution of all the scattered regular expressions. We found Ply, a python implementation of the popular unix programs Lex and Yacc. After some more searching we found css-py, a lexer and parser-grammar written using Ply. Unfortunately css-py seems to have been unmaintained for over a year and both the lexer and parser-grammar needed alot of modification to support our needs, handling broken CSS and CSS3. In the end we used parts of the lexer but had to rewrite the parser-grammar.

With the help of Ply, we managed to speed up execution enough that we were happy with the performance. The end result then was that the parser/cascader became a mix of code from CSSEngine, py-css and newly written code. It supports the following:

• Selectors according to the CSS3 selectors module
• Namespaces according to the CSS3 namespaces module
• Cascading according to CSS2.1, also referred to by CSS3
• Most color spaces (rgb, rgba, hsl, hsv) according to the CSS3 colors module
• Large parts (but not all) of the CSS3 values module

So, while the user agent emulation we had before was very likely better than anything any other analyzer could come up with, it is now better yet. We now emulate most important, performance-affecting characteristics of modern browsers (and other user agents) and can therefore provide a fairly accurate analysis, that tells the user how fast his/her page loads in different browsers. This is something that few, if any, other analyzers can claim being able to do.

New proxy recorder feature

As a premium (paying) Load Impact user, you get access to the HTTP recorder feature. This is a zero-configuration recorder that you will likely not find anywhere else, because it is quite tricky to implement. Unfortunately, it also cannot handle 100% of all advanced javascript out there. Simple Javascript that uses e.g. XMLHttpRequest to load static URLs works fine, but when Javascript starts generating dynamic URLs it gets more complex, and then our zero-configuration recorder might fail to record all transactions that happen.

When the zero-configuration recorder has failed, the only remaining option has been to manually enter the load script for the test, URL by URL. It is tedious work, so we decided we needed an alternative recorder, implemented as a true HTTP proxy. That would mean it would be able to record any and all transactions, no matter how they were generated.

The new recorder is live now. When you configure your test, you select "Record session" like before, then you will be presented with a choice of using the old "Zero-configuration", or the new "proxy recorder":

When you've chosen the new proxy recorder, a window will open that tells you how to configure your web browser to use the proxy server (which is also the recorder). At the bottom of that window, you will find a "Test proxy settings" button you can use to verify that you've managed to configure your browser correctly. If you haven't configured things correctly, this is what you will see when pressing the button:

And a correct proxy configuration in your browser looks like this:

Once the proxy is configured, you can start surfing your website. All your actions will be recorded. Note that everything you do in the browser, in all tabs and all windows, will be recorded. So make sure to close all unnecessary tabs so you don't record any transactions that happen in them.

The exception to this rule is the site loadimpact.com - nothing you load from us will be included in your load script. This is handy as it means you can simultaneously record your load script and keep a couple of tabs/windows open for loadimpact.com also, without getting any loadimpact.com URLs in your recording.

The web page analyzer has been updated with more features

Today we installed some updates for the new web page analyzer. These are the major changes we've made:

New top-of-page layout:

The new layout emphasizes the browser emulation feature of the analyzer. This feature makes our analyzer quite unique, as most analyzers have no emulation capability whatsoever, and of those few that do (we know of only one), the number of browser-specific, performance-affecting characteristics they emulate are not many.

New config parameter: client bandwidth limit

The client bandwidth parameter is configurable under the "Advanced settings" panel. Only premium users can change this parameter. The client bandwidth limit emulates a certain client bandwidth. The actual bandwidth used during the analysis will of course also be limited to the available bandwidth to (or from) the web server that is being tested - whichever of the two is lowest. The default setting is 10 Mbit/s bandwidth limitation for a test.

Load diagram:

We also made some small changes to the load diagram presentation. For compressed resources, we added information about the original, uncompressed file size in the mouseover tooltip:

Summary:

For the Summary section (below the load diagram) we added info about what browser emulation a test used, and a "Status:" line that shows whether the analysis ran to completion without any problems, or if it timed out or was aborted for some reason.