A day in the life of a Performance Test Engineer

The pace of change is unrelenting. Technology is rapidly and continually evolving, bringing new layers of complexity and with it, greater expectations, and even bigger challenges in quality and testing.

Across the Quality Engineering practice, we have deep technical know-how, industry product and platform expertise, and innovative quality management insights to uplift any testing capability. 

With the new year in full swing, I spent the weekend taking advantage of online sales for this summer season. With the sales still on, I began thinking about the volume of online shoppers.

As a Performance Test Engineer, I instinctively assess websites during this period. Will applications have longer loading times while I wait for my order to be processed? Would a user abandon an order due to this latency? Well, my day job is to ensure this does not happen. I try to “break” the system before it goes live by simulating the expected and worst-case activities to ensure that the system is still fully functional and performant, even on eventful days like Black Friday and Cyber Monday.

Performance testing is focused on how a system behaves under various load conditions. It answers questions about the stability, availability, and accessibility of a system. For example, can a website handle a large volume of users or transactions at once? Can backend services handle multiple concurrent requests? My role is crucial to the overall delivery of any technical solution as a non-performant application can lead to financial losses, reputational losses, and poor customer satisfaction, within a matter of seconds.

Starting with the planning phase, I work closely with the key stakeholders to gather non-functional performance requirements, defining the performance testing scope, and creating the performance artefacts such as the overall test strategy.

In the implementation phase, my time is dedicated to setting up the performance test lab or infrastructure, working with stakeholders to ensure the tool can access the system under test. In addition to this, I’m also developing testing scripts using the performance testing tool to automate the in-scope scenarios. Ensuring we achieve accurate expected volumes is critical by analysing performance logs, for making informed inferences following the data, and reporting any observed performance testing bottlenecks; therefore, gaining confidence in real-life usage of the system.

The first test run is where I often hold my breath in anticipation. I keep a close eye on the metrics in the monitoring tools as I observe the resource utilisation while the throughput (i.e., amount of data being processed) gradually ramps-up. Analysing the results from the first test run will often determine the approach for subsequent tests and configurations, given that major performance issues will typically be identified during this stage. All uncovered issues will require a root cause analysis (RCA) and subsequent remediation. I’ll often work closely with the development and operational teams to identify any other factors at play, and then I rinse and repeat.

There are various performance test types that can be executed to uncover different insights, each with the shared aim of confirming that the system is performant in relation to the pre-determined non-functional requirements.

• Peak load testing – tells us whether the system can handle expected peak load volumes.
• Stress testing – tells us whether the system can remain performant under additional or extreme workloads for identifying the breaking point.
• Endurance/soak testing – tells us whether the system can remain performant under load conditions for a prolonged period.
• Spike testing – tells us whether the system can handle the sudden influence of large load spikes.

As critical as performance testing may seem to the reliability and resilience of a system, it often comes with its own set of challenges; namely, being an afterthought in planning. As a performance test engineer, it is my responsibility to guide stakeholders on the criticality of performance testing and ensure that any misconceptions are clarified. In addition to this, I also help to drive planning to materialise the non-functional test requirements and the test environment configurations (as production-like as possible), including representative test data. Often, this helps the stakeholder to consider dimensions of testing that had not been considered previously, which ultimately leads to the delivery of a better quality solution.

There is never a dull moment in the life of a performance test engineer. Each day is different; I could be writing a test plan or getting lost in script creation. I may be working on analysis and reporting or working with stakeholders to find the root cause of an issue. I may also be observing a scheduled performance test run or on calls with a Site Reliability Engineer (SRE) to restart or rebuild an environment after the performance test breaks it (hoping it does not!).

After 9 years in this field, performance testing is an area I could continue in for the remainder of my career. With ever-changing technology, there are always new tools and concepts to keep things exciting against a constant demand for performant and stable systems. There will always be a need for performance testing, which is perfect for me and all the performance testers out there!