How to automate testing with CMake, CTest, and gtest

In this article, we will have a look at how we can integrate unit, integration, and system tests with CMake. We will see that CMake comes with additional helper utilities, one of which is CTest. We use CMake to register all tests and then compile them into individual executables, and we use CTest to run all test executables automatically.

By the end of this article, you will have learned how simple it is to add unit, integration, and system tests to your CMake-based projects. If you combine this with a package manager such as Conan, which we explored in the previous article, you can setup your test-driven development environment in no time using your favourite testing framework. The end result is a more productive and less error-prone setup, and your software quality will improve automatically!

Download Resources

All developed code and resources in this article are available for download. If you are encountering issues running any of the scripts, please refer to the instructions for running scripts downloaded from this website.

• Download: Build-Systems-Part-6

In this series

• Part 1: How to use Makefiles to automate CFD solver compilations
• Part 2: Introduction to CMake for CFD practitioners
• Part 3: Advanced CMake features to power up your CFD development
• Part 4: How to compile, install, and use custom libraries with CMake
• Part 5: How to add external libraries into your CFD solver using CMake
• Part 6: How to automate testing with CMake, CTest, and gtest

In this article

Introduction to automated testing with CMake and CTest

One of the cornerstones of modern software engineering is software testing. It is so important that we spend an entire series looking at the different types of automated tests that we can write and how to integrate all of that into our project, which can be a CFD solver or parts of it, which typically comes in the form of a library.

This article is not about the fundamentals of software testing; we already looked at that in depth in the opening article on the software testing series. Instead, I want to look in this article at how we can use CMake to help us automate all of our testing needs. CMake does come with a few additional utilities, the most important ones being CTest and CPack. As you can imagine, CTest is used to help us with our software tests, and I want to explore how we can use it to automate our tests in this article. We will look at CPack in the next article to package our software for distribution.

Let’s first think about software testing for a second, and why we may need a CMake helper such as CTest to automate our tests. If we have a small project, with a few source files (or even just one), we may just write a single test executable, as we did, for example, with the complex number class when we first looked at software testing.

However, as our project grows, we may start to have a few tests. When we looked at how to test our linear algebra library solver, we ended up with unit, integration, and system tests. I was lazy, so I compiled all tests (manually) into a single executable, but you generally want to keep tests in their own separate executable. That is, each unit test you write for an individual class ought to go into its own executable. Each integration and system test should also go into its own executable, too, and so you can quickly build up an arsenal of tests to execute.

If your project reaches 10,000 lines of code and more, you’ll likely have a few tens, if not hundreds, of tests to execute. Clearly, you don’t want to do this manually, and ideally, you also don’t write a bash or PowerShell script to run your test executables. You would have to update that script every time you add a test, which is error-prone and in the worst case, you may write a test which would reveal a bug, but you forgot to add it to your test script, and so it is never executed.

Since CMake is all about software automation (mostly the building of the software), it does offer support for testing as well. In particular, it exposes a few functions that help us create new tests easily, and then register them with CTest. All we have to do is then execute CTest from the command line and we can see all tests running (and hopefully passing).

The best part about CTest is that it does not force any particular testing framework on you. You can either write your own test without using a testing framework or use an existing one. It is up to you. CTest integrates well with all major testing frameworks, such as google test (gtest), but if you can’t be bothered, just write a simple C++ source file with a single main() function, fill it with assert() statements and return 0 at the end of the file. CTest will interpret a return code of 0 as a test success, and anything else as a test failure.

So, you see, there is some benefit to letting CMake manage our tests automatically, and this is what we will explore in this article. In particular, we will revisit the complex number test setup we developed in a previous article. In that article, we used both bash and PowerShell scripts to build and execute our tests, and we will focus only on this part and replace it with CMake and CTest. Don’t worry if you haven’t followed the aforementioned article. I’ll provide a quick overview before we dive deep into the test automation with CMake and CTest.

Testing our complex number library with CMake, CTest, and gtest

In this section, we’ll briefy review the project that we are working with so we are all on the same page. Afterwards, we will see how simple it is to add tests written in gtest to CMake and how to execute everything with CTest. You’ll be surprised, it is actually rather straightforward! Let me show you.

Project structure

First of all, let’s review the project structure, which is given below. We have a single source file in form of a header in src/complexNumber.hpp. This contains the class definition of a complex number. We use that, to test complex addition, subtraction, etc., as well as additional functionalities exposed by the complex number class.

root
├── src/
│   └── complexNumber.hpp
├── cmake/
│   └── tests.cmake
├── tests/
│   ├── CMakeLists.txt
│   └── unit/
│       ├── CMakeLists.txt
│       └── testComplexNumbers.cpp
├── conanfile.txt
└── CMakeLists.txt

Then, we have the tests written in a form that gtest can understand, which are located in tests/unit/testComplexNumbers.cpp. All of our tests for the complex number class are in this file. We have a few additional CMakeLists.txt files, which we will look at in detail below, as well as a cmake/tests.cmake file, which we use to write a function in CMake to add tests automatically (to save us writing lengthy and repetitive test descriptions).

There is a conanfile.txt file as well, which will handle all of our dependencies through the Conan package manager. We will look at this file in more detail as well, including how to use Conan to get the dependencies into our project, but for now, let’s just have a quick look at the source and test files before we jump into the build scripts.

Quick review of the project’s source files

We have discussed the complex number class at length now, but if you just stumbled upon this article, I’ll catch you up. If you are already familiar with the complex number class and its tests written in gtest, you can safely go to the next section. You won’t miss anything.

The complex number class definition

The src/complexNumber.hpp class is shown below. It simply defines a few helper functions to work with complex numbers and then some overloaded operators so we can add, subtract, multiply, and divide complex numbers with each other. A full description of the source code was provided when we first looked at this class, which you may want to consult if something is unclear.

#pragma once

#include <iostream>
#include <cmath>
#include <stdexcept>
#include <limits>

class ComplexNumber {
public:
	ComplexNumber(double real, double imaginary) : _real(real), _imaginary(imaginary) {
    isNan(real, imaginary);
  }

	~ComplexNumber() = default;

	double Re() const { return _real; }
	double Im() const { return _imaginary; }
  void setRe(double real) { _real = real; }
  void setIm(double imaginary) { _imaginary = imaginary; }

  void conjugate() { _imaginary *= -1.0; }
  double magnitude() const { return std::sqrt(std::pow(_real, 2) + std::pow(_imaginary, 2)); }
	
	ComplexNumber operator+(const ComplexNumber &other) {
    isNan(_real, _imaginary);
    isNan(other._real, other._imaginary);
		
    ComplexNumber c(0, 0);
		c._real = _real + other._real;
		c._imaginary = _imaginary + other._imaginary;
		return c;
	}

  ComplexNumber operator-(const ComplexNumber &other) {
    isNan(_real, _imaginary);
    isNan(other._real, other._imaginary);
		
    ComplexNumber c(0, 0);
    c._real = _real - other._real;
    c._imaginary = _imaginary - other._imaginary;
    return c;
  }

  ComplexNumber operator*(const ComplexNumber &other) {
    isNan(_real, _imaginary);
    isNan(other._real, other._imaginary);
    
    ComplexNumber c(0, 0);
    c._real = _real * other._real - _imaginary * other._imaginary;
    c._imaginary = _real * other._imaginary + _imaginary * other._real;
    return c;
  }

  ComplexNumber operator/(const ComplexNumber &other) {
    isNan(_real, _imaginary);
    isNan(other._real, other._imaginary);

    double denominator = other._real * other._real + other._imaginary * other._imaginary;
    if (std::abs(denominator) < std::numeric_limits<double>::epsilon())
      throw std::runtime_error("Complex number division by zero");
    
    ComplexNumber c(0, 0);
    c._real = (_real * other._real + _imaginary * other._imaginary) / denominator;
    c._imaginary = (_imaginary * other._real - _real * other._imaginary) / denominator;
    return c;
  }

  friend std::ostream &operator<<(std::ostream &os, const ComplexNumber &c) {
    os << "(" << c._real << ", " << c._imaginary << ")";
    return os;
  }

private:
  void isNan(double real, double imaginary) const {
  if (std::isnan(real) || std::isnan(imaginary))
    throw std::runtime_error("Complex number is NaN");
}
	
private:
	double _real;
	double _imaginary;
};

The complex number unit tests

Within the tests/unit/testComplexNumbers.cpp file, we have defined all of our tests using gtest. We originally introduced a simple test written without any testing framework, which we then later rewrote to work with google test (gtest). Both versions would work with CMake and CTest, though it is usually a good idea to work with a test framework, as it has decent support for testing complex cases 9floating point numbers, exceptions, etc.). The complex number test is reproduced below, and if something is unclear, feel free to consult the above-linked article.

All tests shown below follow the AAA (arrange, act, assert) principle. This helps us to organise our tests into a predictiable format and makes them easier to read. I have written about that as well in the gtest article, where you can find a more detailed description of this as well.

#include <limits>

#include "gtest/gtest.h"
#include "src/complexNumber.hpp"

TEST(ComplexNumberTest, TestComplexMagnitude) {
  // Arrange
  ComplexNumber a(3.0, 4.0);
  
  // Act
  auto magnitude = a.magnitude();
  
  // Assert
  ASSERT_DOUBLE_EQ(magnitude, 5.0);
}

TEST(ComplexNumberTest, TestComplexConjugate) {
  // Arrange
  ComplexNumber a(3.0, 4.0);
  
  // Act
  a.conjugate();
  
  // Assert
  ASSERT_DOUBLE_EQ(a.Re(), 3.0);
  ASSERT_DOUBLE_EQ(a.Im(), -4.0);
}

TEST(ComplexNumberTest, TestComplexNumberWithNaN) {
  // Arrange
  ComplexNumber a(1.2, -0.3);
  ComplexNumber b(1.8, 5.3);
  
  // Act
  a.setRe(std::numeric_limits<double>::quiet_NaN());
  
  // Assert
  ASSERT_THROW(ComplexNumber a(1.2, std::numeric_limits<double>::quiet_NaN()), std::runtime_error);
  ASSERT_THROW(a + b, std::runtime_error);
  ASSERT_THROW(a - b, std::runtime_error);
  ASSERT_THROW(a * b, std::runtime_error);
  ASSERT_THROW(a / b, std::runtime_error);
}

TEST(ComplexNumberTest, TestComplexAddition) {
  // Arrange
  ComplexNumber a(1.2, -0.3);
  ComplexNumber b(1.8, 5.3);
  
  // Act
  ComplexNumber c = a + b;
  
  // Assert
  ASSERT_DOUBLE_EQ(c.Re(), 3.0);
  ASSERT_DOUBLE_EQ(c.Im(), 5.0);
}

TEST(ComplexNumberTest, TestComplexSubtraction) {
  // Arrange
  ComplexNumber a(1.2, -0.3);
  ComplexNumber b(1.8, 5.3);
  
  // Act
  ComplexNumber c = a - b;
  
  // Assert
  ASSERT_DOUBLE_EQ(c.Re(), -0.6);
  ASSERT_DOUBLE_EQ(c.Im(), -5.6);
}

TEST(ComplexNumberTest, TestComplexMultiplication) {
  // Arrange
  ComplexNumber a(1.2, -0.3);
  ComplexNumber b(1.8, 5.3);
  
  // Act
  ComplexNumber c = a * b;
  
  // Assert
  ASSERT_DOUBLE_EQ(c.Re(), 3.75);
  ASSERT_DOUBLE_EQ(c.Im(), 5.82);
}

TEST(ComplexNumberTest, TestComplexDivision) {
  // Arrange
  ComplexNumber a(1.0, -2.0);
  ComplexNumber b(1.0, 2.0);
  
  // Act
  ComplexNumber c = a / b;
  
  // Assert
  ASSERT_DOUBLE_EQ(c.Re(), -0.6);
  ASSERT_DOUBLE_EQ(c.Im(), -0.8);
}

TEST(ComplexNumberTest, TestComplexDivisionWithNaN) {
  // Arrange
  ComplexNumber a(1.0, -2.0);
  ComplexNumber b(0.0, 0.0);
  
  // Act
  
  // Assert
  ASSERT_THROW(a / b, std::runtime_error);
}

int main(int argc, char **argv) {
  testing::InitGoogleTest(&argc, argv);
  return RUN_ALL_TESTS();
}

Installing dependencies with Conan

The next step is to install gtest. I have provided detailed instructions on how to compile gtest manually, so if you want to build it yourself, feel free to go through my instructions. I have also provided shell scripts for Windows and UNIX to automate the download, build, and installation step.

In this article, though, I want to continue the good practice we developed in the previous article, where we have used Conan as our package manager to handle all of our dependencies. This will require a conanfile.txt, which is shown below

[requires]
gtest/1.14.0

[generators]
CMakeDeps
CMakeToolchain

We simply require gtest as a dependency, using version 1.14. We want to generate all required files for CMake, in particular, the toolchain file, which contains all required information of where to find the libraries managed by Conan so that CMake is aware of it. I talked about the toolchain file as well in greater depth in the previous article, have a look if you need a refresher.

I want to use a dev profile in Conan, which specifies that all my libraries should be compiled in Debug mode. I also want to tell CMake (through the toolchain file that Conan will generate) that I want to use Ninja as my generator. Thus, I did a copy of my C:\Users\tom\.conan2\profiles\default default profile and create a dev version, i.e. C:\Users\tom\.conan2\profile\dev. The content of this profile is shown below:

[settings]
arch=x86_64
build_type=Debug
compiler=msvc
compiler.cppstd=20
compiler.runtime=dynamic
compiler.version=193
os=Windows

[conf]
tools.cmake.cmaketoolchain:generator=Ninja
tools.env.virtualenv:powershell=True

You don’t have to do this, but I find it sensible to have a development profile (and then, later, also a release profile) so that Conan can automatically build and install libraries in Debug mode. To install all dependencies (gtest, in this case), open a console and head to the root-level directory of this project. Then, execute the following command (if you do not have a dev profile, remove the -pr dev flag from the command):

conan install . -pr=dev --output-folder=build --build=missing

This will download and build gtest and put everything into the build/ folder that we need to tell CMake where to find gtest. Our project is now set up correctly, and we can start compiling our tests. For that, let’s look at the various CMakeLists.txt files that are now scattered around our project.

CMake build scripts

There are three separate CMakeLists.txt files in our project, and an additional helper file within the cmake/ directory. We will review their content in this section and then see afterwards how to use CMake and CTest to compile and run our tests automatically.

Root-level build script

Within our root folder, we have the top-level CMakeLists.txt file, which serves as the entry point into our CMake project (i.e. this file is processed first by CMake before any other CMakeLists.txt file). In fact, CMake will only process this file unless we provide explicit instructions to CMake to look for additional files in other directories.

Let’s have a look at this entry CMake file, which is given below.

cmake_minimum_required(VERSION 3.23)

project(complexNumberTests
  VERSION 1.0
  LANGUAGES CXX
)

include(cmake/tests.cmake)
find_package(GTest REQUIRED)
enable_testing()
add_subdirectory(tests)

We do our usual calls to require a certain version. In this case, we could have used a lower version than 3.23, but this should be a pretty inconsequential requirement. We define our project complexNumberTests on lines 3-6, and then, we proceed to lines 8-11, where we deal with building all tests in our project.

Line 8 requires that we include the cmake/tests.cmake file. Typically, we want to have a clean root CMakeLists.txt file, and so everything that can be split up into either functions or macros usually goes into a file within the cmake/ directory. It is not a requirement but a convention, so if you see other projects with a cmake/ folder, chances are they have some helper functions defined here as well. We’ll look at the cmake/tests.cmake file in the next section.

Next, we require that gtest is available, otherwise, we can’t build the project. Line 10 is new for us, and enable_testing() essentially initialises CTest and tells CMake that we are going to add tests as part of our build. CMake is now aware of this and will provide a mechanism to register tests, which can then be executed with CTest after CMake has compiled all test executables. Line 11 just tells CMake to go into the tests/ directory to process the CMakeLists.txt file within that folder.

That’s it. It’s pretty straightforward, don’t you think? The other files do not get much more complicated. Let’s look at the cmake/tests.cmake helper function next to see how we can add tests automatically.

Additional CMake files in cmake/

As alluded to above, we are going to write a function. This function takes the name of the test file for which we want to build a test executable as the argument. Let’s look at the function first and then discuss the content.

# define a function that can be used to add tests automatically
function (create_test name)
  # each test creates its own executable
  add_executable(${name} ${name}.cpp)

  # we need to link each test separately against google test
  target_link_libraries(${name} gtest::gtest)

  # include the root level directory for each executable so we can find header files relative to the root
  target_include_directories(${name} PRIVATE ${CMAKE_SOURCE_DIR})

  # register the test with CTest, so we execute it automatically when running CTest later
  add_test(NAME ${name} COMMAND $<TARGET_FILE:${name}>)
endfunction()

Most of this function probably feels familiar by now if you have followed this series from the start. The function create_test() is declared on line 2 and accepts exactly one argument, which is the name of the test. We create a test executable on line 4, using the same name for the executable and source file name (to which we append the .cpp suffix), and then we link our executable with gtest on line 7. Line 10 simply tells the executable to include the root level directory so that we can easily define the relative path to src/complexNumber.hpp from within the tests.

Line 13 is the only new feature, and we call the add_test() function if we want to register a given executable with CTest. This means that once we run CTest, all tests added through the add_test() command will be executed automatically. The command seems a bit obfuscated, as we have this argument passed to the function: COMMAND $<TARGET_FILE:${name}>. This is a generator expression and it is required due to the way the add_test() functions work.

If we look up the definition in the documentation, we get the following sentence: If COMMAND specifies an executable target created by add_executable(), it will automatically be replaced by the location of the executable created at build time. Since the location at build time is not available during the configuration step, we need to use a generator expression to query for the name.

This may seem confusing, and it is. CMake still holds on to some broken design patterns and philosophies and has instead opted to patch things by introducing new features (instead of fixing broken ones). The result is generator expressions, which are injected between the configuration and build step in CMake. After the configuration is done, we have all the information ready for the build step, so we insert another step (the generator step) where we can now query for information that is only available at build time.

Since CMake replaces the test executable with the location that is created at build time (i.e. after the configuration has finished), we need the generator expression here. The good news is, however, that we pretty much always use the command as shown above on line 13 if we want to add tests (you may want to change the name of the executable, but that’s about it). Otherwise, this function is pretty robust and you can use it to add any test to your project and CTest.

Build script in tests/

The CMakeLists.txt file within the tests/ directory is pretty straightforward; we only tell CMake to go straight to the unit/ directory, without doing anything else here. For completeness, this is shown below:

# add unit tests
add_subdirectory(unit)

Build script in tests/unit/

The CMakeLists.txt file within the tests/unit/ directory is equally short, but rather effective, let’s have a look.

# add tests for complex number calculations
create_test(testComplexNumbers)

Here, we simply make a call to the create_test() function that we defined in the CMake helper file cmake/tests.cmake. The argument is the name of the source file of the test that we want to process (without the file ending). In this case, we want to process tests/unit/testComplexNumbers.cpp, so our function argument becomes simply testComplexNumbers. Now, we can add as many tests as we want and simply pass all source files to the create_test() function, to build an executable, link it against gtest, and register it with CTest.

We are now ready to compile our project and execute our tests. Let’s do that in the next section.

Compiling the tests

At this point, we have gone over the compilation step quite a few times, and hopefully, this will look very familiar to you by now. To execute the configuration step, execute the command shown below:

cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_TOOLCHAIN_FILE="conan_toolchain.cmake" -G Ninja ..

If you are using a different generator, use that instead, or omit the -G altogether, but be sure to use the same strategy in Conan, as the created toolchain file by Conan will assume a specific generator to be used, as we have discussed above at length. Once the project is configured, we can build it with the usual command

cmake --build . --config Debug

You should now see the testComplexNumbers.exe (Windows) or testComplexNumbers (UNIX) file within your build/ directory. If you used MSBuild, Ninja-multiconfig, or Xcode, you will see it in the Debug/ folder. The actual location doesn’t matter, though, as we will not execute the test ourselves. For that, we have to turn to CTest, finally.

Running CTest on the command line

Since we have already defined all tests in the tests/unit/CMakeLists.txt file, CTest already knows about all executables and can run them for us. To do so, simply invoke

ctest

on the command line, and CTest will start running all of our tests. Below is the output of CTest on my system:

Test project C:/Users/tom/code/buildSystems/Build-Systems-Part-6/build
    Start 1: testComplexNumbers
1/1 Test #1: testComplexNumbers ...............   Passed    0.01 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.01 sec

We get some high-level information, for example, we see that the total time it took to run our unit tests was 0.01 seconds (which is good, we want them to run quickly), as well as that 100% of our tests passed. This is really all there is to testing with CMake. It does not get more complicated than what we have seen in this article, and if you just keep adding tests to your create_test() function, you will be able to quickly build up your test suite of unit, integration, and system tests in no time!

Bonus: The Test-explorer plugin for VScode

If you are one of the 75% of people who use VSCode as your code development platform, then you can supercharge your testing experience even further with some extensions from the marketplace. And, if you don’t use it, don’t worry, you can ignore this section and move on to the summary, you won’t miss anything.

The power of VSCode (and, for that matter, of any integrated development environment (IDE)) is that it is extendable through plugins, and one of the available plugins in VSCode is the Test Explorer UI (which requires the Test Adapter Converter plugin to be installed as well). Once you have this installed and have opened a project folder containing a CMake file with tests, this plugin will discover your tests automatically and can run CTest for you.

You have fine-grain control and can execute individual tests instead of running all tests (which is what CTest does). This is useful if you have hundreds of unit tests and only one or two of them fail. You can then debug those tests and run them individually until your tests are passing. You can step directly through the steps with a debugger, which is very powerful for finding bugs quickly, and it provides a very neat representation of all discovered tests. You can see the tests for the complex number project developed above within the Test Explorer UI below:

As you can see, all tests are passing, as we saw with CTest as well (although, by default, CTest only reports if the executable passed, not each individual test within that executable.

Since the Test Explorer UI is very powerful in finding your tests in the background, there is not much more for you to do if you use this plugin. But, if you feel you want to read up a bit more on what this plugin can do for you, then I can only recommend the article by James Pearson aptly titled Test Explorer in Visual Studio Code.

Summary

In this article, we explored the capability of CMake to manage tests for us automatically. We saw that by having a call to enable_testing() somewhere in our CMakeLists.txt file, we were able to register tests with CMake through the add_test() command. All registered tests were then built by CMake and executed by its helper utility CTest.

We saw that CTest does not require you to follow a specific framework and integrates with manually written tests or those that are prepared with a testing framework such as gtest (which we used here in this article). CMake makes it very easy for us to include tests, and this is good. The simpler it is, the more people will use it, and I hope that you will start using unit, integration, and system tests in your projects now as well and integrate them with your CMake files. As a result, I promise your software will become better!

---

Tom-Robin Teschner is a senior lecturer in computational fluid dynamics and course director for the MSc in computational fluid dynamics and the MSc in aerospace computational engineering at Cranfield University.