Performance testing - typically the job of the Non Functional Test (NFT) team - should always be completed against a stable build of the system, in an environment that resembles the final production setting as closely as possible. Extensive functional testing is usually carried out beforehand, along with various other essential actions.
On paper the above looks fine and as expected, but by the time the system reaches the NFT team the underlying code has passed through various developers over many iterations. New code gets added to existing code and existing code is refactored, as different team members work their magic to create the system.
With the code receiving so much attention in the build up to performance testing, the NFT team has a hard time determining where any code-related performance issues originate.
If the above quandary is to be solved, projects must avoid vehemently segmenting each phase of a project. To reduce any wasted time in the performance testing phase, ‘performance profiling' tests can be run during the functional test phase. Their purpose is to quickly identify any code-related performance issues. With performance profiling, a small number of virtual users is sufficient – say five or ten concurrent. These profiling tests should be run every time code is checked-in, making performance testing an essential part of the daily build process.
The key is to ensure these lightweight performance tests are run on a consistent and stable environment. Whilst this environment won't resemble the final production architecture, it will quickly highlight any degradation in performance. Each test will be executed against the code in its most recent form, making it possible to highlight the root of the problem quickly. Once diagnosed, issues can be fixed and re-tested before the code is released to the NFT team for formal and extensive load testing.
A ten-step guide to performance profiling
The above form of lightweight performance testing can have a positive impact on the speed at which you can get a product to market. And by following the ten-step guide below, adopting such an approach doesn't have to be difficult or costly.
- Source an environment
This doesn't necessarily have to be the production environment; it could even be the test environment, as long as it's not being used when the profiling is conducted. It might be possible, for example, to complete the process out of normal office hours. Be sure to log the specification of the machine used, so that it forms part of your baseline setup.
- Choose a tool
Ideally, this would be the same tool that’s used for full-scale performance testing (during the NFT phase) as the same scripts can be used but for much lower load, obviously. Otherwise, open source tools like JMeter and WebPageTest are excellent for both profiling and full-scale testing.
- Start early
Behaviour-driven development and agile vertical slice development make it pretty easy to create some client-facing functionality relatively quickly. Once at this stage, it's time to start writing the profile test - even if it's just calling a simple GET against a web page. Simply put, the earlier you test, the earlier you find bugs.
- Run a base line
With the script in place, you are ready to run a base line. Make sure you discuss the results with the team and business to ensure everything fits in with what was initially expected.
- Schedule your tests
For this, you can use a Continuous Integration (CI) server, or even a cron job. If you're using Jenkins/JMeter, it's worth using the JMeter plugin - this will not only run the tests but also report back some useful graphs. It makes sense to schedule these for every time new code is checked in.
- Monitor your results
Hooking your tests up to the CI can help with this - just be sure to implement some kind of threshold pass and failure conditions. This way, you can sound the alarm if the performance profile build goes 'RED'.
- Learn from the improvements
If it's clear that the application is starting to perform better, determine the reasons for this and see if those improvements can be implemented elsewhere in the code. Negative changes should also be picked up on and investigated promptly.
- Maintain the tests
Whenever more functionality is completed, adjust the tests accordingly. It's also important to retire tests if they become obsolete. This should be treated like any other testing or development task.
- Monitor your system
Monitoring should be installed around all parts of the system, including the database, the webserver, CPU and memory usage. New Relic may come in handy here, or there are a number of open source tools on the market as well.
- Don't forget the full load test
The goal of performance profiling is to make the process easier and more watertight - it's not designed to replace full load testing. You should still stress/soak/load your application as normal.
Why introduce performance profiling?
When performance profiling is used as part of the daily build process, it becomes easier to instantly highlight any small performance issues early on in the process – long before it even reaches the NFT team. In turn, this means that developers are able to optimize and fine-tune the code as they go, ensuring the best possible results. It can be used to lighten the load on the NFT team, who will benefit from being able to focus on serious load, performance and stress tests to identify bottlenecks that aren't necessarily code-related.
Performance profiling is definitely not a replacement for traditional load and performance testing, but its benefits as a complementary tactic are too significant to ignore. Implemented in the correct way, it will significantly cut overall costs, and help transform your releases into NFT from functionally good to operationally great.