Software Testing: Infrastructure Testing
Today, a lot of software-defined infrastructure (SDI) is done as Infrastructure as Code (IaC). Since the infrastructure is code, it is code that can – and probably should be – tested. This blog post contains a brief overview of IaC and how it can be tested. Furthermore, it discusses some things that are special to IaC testing compared to typical software testing.
SDI stands for Software-Defined Infrastructure
To understand IaC, one needs to first understand SDI. With SDI, one can setup things such as networking and computing purely with software. This encompasses things such as network routing, firewalls, virtual machines and much more. This can lead to reduced setup times and maintenance cost.
For example, consider you need to change network routing. If you need to physically unplug and replug a lot of cables, it might be more time efficient to do it with software routing. Likewise, if new features are launched, with software you may be able to simply upgrade to a new version. If the alternative would be to buy new hardware, the software solution might be a lot cheaper and have less lead time.
IaC stand for Infrastructure as Code
Now, if your infrastructure is software, you might want to codify the infrastructure. With IaC, your SDI can be configured programmatically. For example, let’s say you need a new virtual machine. Rather than using manual operation to launch a new VM, you specify the requirements in code and let your IaC framework roll-out your desired changes.
Using IaC, it should be easier to duplicate resources as the common parts of resource definitions are written once and can then be reused. Similarly, IaC can help reduce human errors as something that has been verified to once work can be duplicated. Furthermore, IaC can provide substantial robustness, as if resources fail, they can be brought back up simply by letting the IaC framework run.
Write code to specify what kind of VM’s you would like to have and what software should be installed on them.
Since IaC is code, many of the same things apply as for conventional code. Specifically, for the context of this blog series, testing of code can be very beneficial. This applies especially if the code base is large and has much active development and maintenance.
Many of the concepts from the Types of Tests blog apply. Specifically, one of the ways to catch bugs and problems early in development and for very low cost is to utilize static code analysis. Static code analysis tools can indicate problems without requiring any real deployment of the code.
provision.yml:5: [E403] Package installs should not use latest
Example of static code analysis. Output generated by ansible-lint.
However, in contrast to testing of conventional code, unit testing of IaC is rarely used. This is due to the fact that the functionality of a resource or configuration is typically so much tied to the environment, such as a cloud provider. I.e. mocking the environment and underlying implementation would not make much sense.
While unit testing may not provide much value, module testing is widely used and can be very useful. Virtually every IaC framework is somewhat based on modules. While the modules need to be deployed for testing, these standalone modules can be tested in isolation of the rest of the IaC setup. Therefore, testing a standalone module is a lot faster than testing everything together. Furthermore, with module testing, we can verify that the module works as intended without interference from other infrastructure.
Some ideas what to test in module testing:
Deployment and tear-down of a resource.
Check that needed access points are available.
Verify that security policies deny access where applicable.
To test that several modules work together, we can introduce integration tests. By design, integration tests take a longer time to run than module tests as they need more resources to be setup (and teared down at the end of the test run). Therefore, it might be best to test as much as possible with module tests.
Deployment of software-define infrastructure can take a long time
Both module and integration tests deploy resources to a real execution environment after which they run the tests and ultimately destroy the newly created resources. This is analogous to conventional software development and testing where the program is executed and tested. However, what is special with many SDI’s is that the live deployment is a moving target and can break by itself. For example, our network setup might rely on the availability of an external DNS server etc.
Hence, it may be beneficial to run tests to verify that the live infrastructure is really operating as it should. However, there are some challenges to this as we don’t want the testing to interfere with the live product. Therefore, we may not be comfortable testing error cases, high traffic etc. Instead, simple heartbeat checks are typically used.
Summary
In this blog post introduced the concepts SDI and IaC. We discussed the similarities and differences between conventional software testing and testing of IaC. Also, we discussed the benefits and drawbacks of various types of IaC tests.
This article is part of Omoroi’s blog series on Software Testing. You can reach us at blog@omoroi.fi or discuss in this LinkedIn thread.
