Automated Load Testing with CircleCI

Make load testing a first class citizen in your automated testing.

Getting started

Getting started

We believe in goal oriented and automated load testing. That's why we built k6 to work well in such environments, integrating nicely with services like CircleCI, the continuous integration and delivery platform.

This guide will help you get up and running with k6, CircleCI and LoadImpact Insights (with build notifications sent to Slack). This guide assumes you are familiar with k6 and CircleCI.

It also assumes you have a LoadImpact account. If not, go get one here – it’s free. Once you have your account setup the integration with CircleCI is only three more steps.

TL;DR

  • Clone this Git repository
  • Modify the k6 load test script and push it to Github which will trigger a new CircleCI build
  • Wait for the build notification to arrive in Slack!
CircleCI

The load test

It starts with code. This is the load test example script we'll be using in this guide. It puts ramped load on our example site test.loadimpact.com:

k6 load test script

To describe the load test script in words:

  1. The test will ramp up from 0 to 10 users over 60s, stay flat at 10 users for another 60s, before ramping down to 0 users again over 60s
  2. We've set a goal that the 95th percentile response time should be below 500ms. This step, k6 thresholds, is essential, it's how we define our goals which is the basis for the pass/fail mechanism to work that is built into k6. If a threshold fails, k6 will end with a non-zero exit code, which in turn indicates a failed build step to CircleCI (and other CI tools).
  3. In the load test "main" function we define a group Front page inside which, we make a request, check the response status code and sleep for 10s, before letting the user loop from the top of the function.

Configuration

Below is the barebones circle.yml config file need to run k6 and the above load test from CircleCI, nothing more!

CircleCI circle.yml config

To describe the config in words:

  1. Download a k6 release archive and extract the binary to a directory that is cached between builds
  2. Run the k6 load test main.js located in the loadtests directory in the root of the repository

Note that the k6 command includes the -o cloud option which means the results will be streamed to Insights (exclude that part of the command if that's not what you want). For that to work we'll need to make sure k6 has access to our LoadImpact API token in the environment variable K6CLOUD_TOKEN, so let's set this up next!

Environment variables

Right, so to have the results of our load tests streamed to Insights we need to add the K6CLOUD_TOKEN environment variable. Navigate to your project's settings in CircleCI and find the "Environment variables" section under "Build settings" in the menu to the left. Add a new environment variable with name K6CLOUD_TOKEN and copy/paste the API token from your account as the value:

CircleCI environment variables

Notifications

We're soon ready to run our first CI triggered load test, promise! But first, let's hook up Slack to our CircleCI project so that we get notified when our load tests finish. For this you'll need to setup an "Incoming Webhook" integration in Slack. Once that is done, copy/paste the Webhook URL into the input field in the "Chat Notifications" settings:

CircleCI notifications slack

Triggering builds

Let's trigger a build by making a small change to the load test script (lowering the threshold to 100ms perhaps). Push the change to Github, CircleCI will pick up the repo change and run the through the circle.yml, and soon enough trigger the load test run:

As we can see above a lowering of the threshold to 100ms triggered a failed build step and thus a failed build, and the notification was quick to let us know:

Result analysis

A passing build is not much fun to look at. But, when builds fail, you want to understand why. No exception when it's a load test failing the build. We've built Insights to be a perfect companion to k6 for analysing results! The analysis workflow is error driven, meaning the goal is to help you quickly finding the cause of the failed load test.

That's it, happy automated testing!

Common Questions

  • How do I run a test against a firewalled system?

    You'd need to grant CircleCI access to your system by adding the necessary IP ranges to your firewall. If you're using AWS you can temporarily grant access by adding a security group rule pre-test:

    #!/bin/bash 
    public_ip_address=$(curl -q http://checkip.amazonaws.com)
    aws ec2 authorize-security-group-ingress --region eu-west-1 --group-id <AWS Security Group ID> --ip-permissions "[{\"IpProtocol\": \"tcp\", \"FromPort\": 443, \"ToPort\": 443, \"IpRanges\": [{\"CidrIp\": \"${public_ip_address}/32\"}]}]"

    and then removing the grant post-test:

    #!/bin/bash 
    public_ip_address=$(curl -q http://checkip.amazonaws.com)
    aws ec2 revoke-security-group-ingress --region eu-west-1 --group-id <AWS Security Group ID> --ip-permissions "[{\"IpProtocol\": \"tcp\", \"FromPort\": 443, \"ToPort\": 443, \"IpRanges\": [{\"CidrIp\": \"${public_ip_address}/32\"}]}]"

    See the modified circle.yml and shell scripts to accomplish this in the example repository.

  • How often should I run my load tests?

    There's no one answer here. We recommend against running load tests on every code commit though. Load tests are different in nature from unit, functional or integration tests. They need a longer time to complete and a target environment that is closer to real life (production) than the types of tests that are more functional in nature.

    Instead, we recommend that you run your test at a frequency that makes sense for your organization. You could for example run tests when merging code into a release branch or as part of a nightly build so you get your results fresh in the morning!