Software Testing: Characteristics Testing

In the previous blog post we discussed load testing. Load testing is often mixed up with characteristics testing. However, although they are similar, they are two different things. In load testing we test our system under load, while in characteristics we discover different metrics of our system. In this blog post we discuss what these metrics can be.

Characteristics testing is a structured way of discovering various metrics of our system. Part of the metrics form a matrix, where on one axis we have the amount of load. For example, we can test the system under no load at all, mediocre load, high load and overload. Our second axis is the actual metrics, such as CPU utilization, memory consumption, latency etc. Another part of characteristics testing are independent of load such as product startup time.

Startup time is a characteristic which does not depend on system usage level.

As mentioned above, some of the metrics in characteristics testing require system load. If load tests are available, it might be possible to reuse them as or replace them with characteristics tests. However, in this scenario it is important to not include runs from failed runs where one or several errors or crashes were encountered. This is important, since, for example, a web server returning “403 access denied” might have done a lot less work than if it would’ve served the client.

Not all usage is equal; An erroneous usage case can be a lot faster or slower.

When characteristics tests are run as automated regression tests, regressions in performance can be easily spotted. In this scenario developers will immediately know if a refactoring, a bug fix or a new feature trashes performance. As such, characteristics testing can be a very valuable asset. However, performance regressions are not binary in nature, and therefore the test results are not binary either. In practice, the test framework needs to be fed with information of acceptable boundaries, or a human needs to evaluate the test results.

Summary

In this blog post we discussed characteristics testing. Characteristics testing give valuable key metrics about our product such as startup time, CPU usage, memory usage and so forth. When characteristics testing is run in a continuous manner, the product developers will get fast feedback if they accidentally trashed product performance.

This article is part of Omoroi’s blog series on Software Testing. You can reach us at blog@omoroi.fi or discuss in this LinkedIn thread.