Software Testing: Types of Tests
Let’s start our blog series about Software Testing by introducing a bunch of different types of tests. In this blog post we describe static analysis, compiler warnings, unit tests, feature tests and system tests.
Most types of tests can be placed on a spectrum where at one end we have tests that execute fast and give feedback early and at the other end we have tests that take a long time to execute and give feedback later. The tests that execute fast are suitable to run continuously on a developer machine. In contrast, tests that take a long time to run might be suitable to run as part of code integration, i.e. at a later stage in product development.
At the beginning of our spectrum we have compiler warnings. There are three big upsides with compiler warnings: they are fast to generate, they tell the developer exactly where the problem is and they are essentially free as little to no extra code is needed. The big downsides to compiler warnings are that they can rarely see the big picture and sometimes they can be too strict.
unused-variable.cpp:6:9: warning: unused variable ‘i’ [-Wunused-variable] 6 | int i = 0; | ^
Example of a compiler warning: unused variable in C++.
Furthermore, at the beginning of our spectrum we have also static code analysis. Depending on the product and programming language, there are various types of static analysis available. For example, golang code can be checked with golangci-li, shell scripts can be checked with shellcheck and terraform code can be checked with tflint.
Line 3: for f in $(ls "*.m3u") ^-- SC2045: Iterating over ls output is fragile. Use globs.
Example output of
shellcheckanalyzing a shell script.
As we move forward in our spectrum, next up are unit tests. At this stage we should still isolate the code and execution environment from all external things. All interfaces should be mocked. This is regardless if the interfaces are internal to the product or external such as the operating system or a cloud execution environment.
The big benefit with unit tests is that we get total control in how to test a single entity in our code base. However, this comes at a development cost, as extensive mocking might be required. Furthermore, we need to create suitable test vectors.
// code to test const fruits = ['apple', 'orange']; // test it("fruits contain apple", () => { expect(fruits).toContain('apple'); });
Example of a unit test with Jest (javascript).
Next up in our spectrum is feature testing. Feature testing is often also called acceptance testing. At this stage we want to write tests that verify the acceptance criteria set up by stakeholders are fulfilled. Isolation and mocking can be used where applicable.
At the end of our spectrum we have system testing. At this stage no internal mocking is used, but rather the whole live product is tested as it is meant to be used. At times system tests can be the most valuable type of testing as they indicate if the product works as a whole. However, the downside with system tests is that it can be hard to pinpoint where the problem is when a system test fails. Furthermore, system tests can take a substantial amount of time to run.
Summary
In this blog post we defined a spectrum of tests including compiler warnings and static analysis, unit tests, feature tests and system tests. As the upsides and downsides of the various tests vary along the spectrum, in the ideal scenario we have a balanced set of tests all along the spectrum. When this is the case, developers can catch errors quickly in compiler warnings, static analysis and unit tests, and they know exactly what to fix. Furthermore, our feature tests and system tests will give more confidence in the overall quality of the product.
This article is part of Omoroi’s blog series on Software Testing. You can reach us at blog@omoroi.fi or discuss in this LinkedIn thread.
