Phase 1: set the stage for successful performance testing
Phase one is designed to lay the foundation for on-going testing.
The goal is to create a test program which can be followed continuously in order to benchmark performance, uncover hidden performance problems and validate performance improvements over time.
Phase one goes as follows:
- Once user scenarios have been scripted, a series of Smoke Tests will be run in order to validate the correctness of the scripts. Smoke tests flush out obvious errors so actual performance testing can begin.
- The next step is to create the tests. Setup includes specifying the number of Virtual Users (VUs) to test with and how that number should vary throughout the test. It also includes which user scenarios to include in each test, how the load will be allocated between the scenarios, where the load should be generated from and what network types the VUs should emulate.
Phase 2: create a performance baseline and find bottlenecks
The goal of phase two is to build an understanding of the current situation and iterate tests to identify hidden problems.
All types of testing will be employed when required.
Phase two goes as follows:
- Based on the goals set during the planning phase, a series of Load Tests will be run to identify bottlenecks, failures and edge conditions, as well as to establish a baseline for evaluating the impact of future system changes.
- Next, a series of Maximum Capacity Tests will be run to determine the maximum number of concurrent users the system can accommodate while continuing to perform as expected.
- Once the system has been tuned to its optimal state, the next step is to Stress Test. Stress Tests determine a breaking point and reveal what happens when the systems goes beyond the goal (i.e. when it breaks).
- Typically a lot is learned from these initial tests and, as a result, more complex scripts and configurations are created in order to dig deeper into the initial findings. This process is repeated until your goals are reached.
- Depending on the type of service being tested and the complexity of the backend (e.g. systems with complex workflows, document management, logistics controls, etc.), a series of Endurance Tests will be run.
- Depending on the type of service being test (e.g. Banks, Data Management, etc.), a series of Robustness tests will be conducted to evaluate the functionality of the service under load (e.g. failover and disaster recovery evaluations).
Phase 3: towards continuous delivery
Phase three of the test execution process is designed for proactive performance management and monitoring.
The goals of phase three are:
- Ensure system performance has been maintained after initiated changes and upgrades.
- Ensure that performance has not degraded due to uninitiated changes (e.g. network or service provider changes).
- Enable proactive, data-driven decision making regarding resource deployment.
- Move towards continuous integration and delivery by automating tests using Load Impact's REST API, SDKs (Java and Python) or CI plugins (Jenkins and TeamCity).