In this article, we will look at testing frameworks in C++ and I provide reasons for why I am using Google’s gtest testing framework and why it covers all of our use-cases we have to test CFD solvers and related code. We will use it to rewrite our tests for the complex number class we developed in the previous article and see that the syntax is very similar, allowing us to easily switch from our own tests to gtest.
By the end of this article, you should have a good understanding of how to get started with gtest. We look at how to compile the gtest library and how we can integrate it into our project. We also look at some peculiarities that may break cross-platform compilation and how to work around that issue. After we have rewritten our tests using gtest, you will know pretty much all there is to testing with gtest and can start using it for your own projects.
While gtest is a very feature-rich library, we will mostly use only a small subset of features and these are highlighted in this article. In the following articles, we will solidify our knowledge of these core features while introducing other advanced features that will help us to clean up more complex tests, as well as use features that may be necessary for certain types of tests.
Download Resources
All developed code and resources in this article are available for download. If you are encountering issues running any of the scripts, please refer to the instructions for running scripts downloaded from this website.
- Download: Software-Testing-Part-2.zip
- Download: installgtest.zip
In this series
- Part 1: How to get started with software testing for CFD codes
- Part 2: The test-driven development for writing bug-free CFD code
- Part 3: How to get started with gtest in C++ for CFD development
- Part 4: How To Test A CGNS-based Mesh Reading Library Using gtest
- Part 5: How To Test A Linear Algebra Solver Library Using gtest
- Part 6: How to use mocking in CFD test code using gtest and gmock
- Part 7: What is test coverage and how to use LCOV/GCOV for testing
In this article
Why do we need a testing framework?
In my previous article, we looked at how we can write a few simple unit tests to check that an implementation of a complex number class was done correctly. We used this as an example and said that complex numbers may come up in our CFD solver if we want to perform some signal analysis as part of the solving stage. To be fair, we would probably use C++’s inbuilt complex number class, but it served as an easy introduction, as complex numbers are rather straightforward to implement (understanding how to use them may be a different question).
While we were able to provide a sensible suite of unit tests, we barely scratched the surface when it came to testing. We did check if a few floating point numbers were calculated correctly and used our custom-made floating point comparison function, which we also said is good but not perfect. We finished the discussion with boundary or edge cases, that we did not test for.
For example, we said that we did not check for a division by zero and for not a number (NaN) errors. If we encounter either of them, there may not be a sensible result for our complex number class to compute, so it may be a good idea to throw an error. But how would we test that?
While we could come up with now more and more helper functions, as we did with our floating point comparison function, there is no need for us to reinvent the wheel. When it comes to software testing, we are better off looking for what software testing frameworks are available, and then picking one that we like the looks of the most.
Testing frameworks will not just have functionality to test floating point numbers and detect if a function throws an error, but they have support for quite a few more things, such as automatically detecting all your tests and running them, or mocking dependencies, a topic we mentioned in our opening article for this series on how to get started with software testing, but something we have not looked into detail at this stage.
Testing frameworks are rather straightforward to learn, so if you understood all of the unit tests in the previous article, then you mostly just have the change the function signature and the assert statements. The code itself will still pretty much read in exactly the same way. So, learning how to use a testing framework is really simple, the biggest learning part on your side is to understand how testing works in general. But as I mentioned, we covered a lot of ground in the previous article, so if you feel you understood everything, then you are ready to move on to testing frameworks.
We will take a brief look at different software testing frameworks, and then I’ll introduce you to my favourite testing framework. Even if you decide not to use it (and you may have strong reasons to do so), have a look at how you would use it. What you learn from this article can be translated to most other testing frameworks as well.
Available C++ testing frameworks
When it come to available testing frameworks for C++ alone, we have more choice than time available to go through all of them to decide which one to use. Luckily, we have this handy comparison available, which shows us in a table form, which software testing frameworks exist for C++ and what features they have. At the time of writing, there are currently 84 different testing frameworks listed for C++, albeit a few being listed twice if they have different variants available. But it gives us an idea of the wealth of frameworks. And that is just C++.
Now when it comes to testing frameworks, we have to mention one particular framework, and that is xUnit. And, having dropped the x word, I can’t think of a better excuse to talk about one of my favourite programmers in history (yes, I have a list of favourite programmers, ranked by importance … )
Kent Beck – the man! (behind xUnit)
Kent Beck is a person you probably have not come across, yet, his work has probably had some of the most influence on how we develop software nowadays. It is a good name to remember for a pub quiz on software development (if such an event exists!). For no particular reason, here is a picture of the man
This picture was not taken in the 1970s, but in the 2000s, and judging by the background, this seems to have been an appropriate choice of clothing for a graduation ceremony. Anyhow, I digress.
Kent Beck gave us extreme programming, a software development approach which we have touched upon briefly at various points, but one part of this software development approach resulted in the test-driven development approach, which is a pretty standard way of developing software these days, the approach we looked at in the previous article, and the approach which is synonymous with a agile software development.
Speaking of agile software development, yes, that Beck as well. Together with 16 others, he developed the principles of agile software development that pretty much any software engineering company employs nowadays.
He gave us software design patterns, something which we have not looked at in detail, yet, but something we ought to have a look at in a future series, due to their importance. You may remember that we looked at the factory design pattern in my article on smart pointers in C++, have a look if you need a refresher, but that is Beck in action again.
And, finally, he gave us SUnit, a software testing framework for the, back then, popular programming language Smalltalk. While SUnit and Smalltalk are pretty much a thing of the past, you will find a few dinosaurs, if I may use this word with no negative connotation implied, reminiscing about Smalltalk’s glory, and probably for good reasons. Programming wasn’t always a pleasure, just look at scripting languages such as lisp, which is what people used before Python came along (all of a sudden it feels good to be a millennial, doesn’t it?). Smalltalk, in comparison, must have looked like the future!
Even though SUnit is gone now, it introduced a few common structures and rules that many other popular frameworks build on. Chances are, if you know how to use one framework, you can easily change to another one. You may have to learn a few different keywords, but that’s it, writing tests and executing them will feel pretty much the same in all of these frameworks that derive from SUnit.
The naming convention is such, that we remove the first letter and replace it with the first letter of the programming language. For example, Beck also wrote JUnit, a popular port of SUnit for Java. Thus, the naming convention xUnit was born and all unit testing frameworks that are based on SUnit are said to be xUnit frameworks.
And, if you are still wondering who else is on my list of favourite programmers, Bjarne Stroustrup, the inventor of C++ (that’s an easy pick) and Herb Sutter, but for all the wrong reasons.
Sutter is essential Microsoft’s poster child for C++. He sits on the C++ steering committee and has a large influence over the direction in which C++ is heading, probably not at all influenced by Microsoft whatsoever. Probably … We already saw how Microsoft has slaughtered C++ when it comes to implementing dynamic libraries, and we don’t need more of that. Every time I am losing my patience with Microsoft’s cl compiler, and their take on how C++ ought to be implemented, Sutter’s voodoo doll is getting it. I don’t really have a voodoo doll of Sutter. Probably …
84 testing frameworks to choose from
So, which framework for testing our code should we pick? Well, looking at the comparison I mentioned earlier, I have two requirements that I look for. The first one is that it is freely available. If there are 84 frameworks to pick from, most of which are free, why would I pick one where I have to pay for it? The second is that it should have a good number of features implemented. That means, in the comparison linked above, we want to see a lot of green tiles indicating that all of these features are available.
Based on that list, no framework has everything implemented, so it becomes a case of finding a framework that has all the features you may require. Popular frameworks that people tend to work with the most, at least from what I can see from my reading, are Boost test library, CppUnit (may I extend my congratulations for their beautiful website design), Catch2 (may I extend my congratulations for to bothering to have a website at all), and google test (at least they bothered providing a useful manual).
Boost, and that is my personal opinion, feel free to disagree, is such a complex set of libraries that I try to stay away from it as much as possible. Bringing in boost into your project guarantees to slow down compilation due to its rich set of dependencies. I know that you can be selective in what you want to use and compile, but I like a small, to the point, implementation of a problem, and boost is the exact opposite.
I haven’t used CppUnit, but people seem to be happy with it, and the same goes for Catch2. it is probably the framework I most liked at second glance, yet I have not gotten around to using it, and that’s because I never saw a reason to switch from Google test, or googletest, or gtest (no one really knows what it is called, and I bet neither do the developers). Let me explain why in the next section.
Google test (gtest): Should we use it?
gtest, as I will refer to it now, covers pretty much all use cases we may have to test CFD codes. I have not come to a point where I was limited by the framework, which is, in fact, so simple to learn that you probably will be proficient with most of its use cases and applications after you have finished reading this article.
Reasons against Google’s gtest
But let’s address the elephant in the room, it is developed by Google, is it a good or bad thing? I should say, that I have stopped using Google, as I find their data collection rather invasive. Not that I have anything to hide, but I also don’t have anything to give away for the sake of building more customised avatars for marketing purposes. I just find the whole process a bit creepy, and have not had any issues with brave search, which is pretty much up there with Google. It’s not 100% perfect, but it does get the job done most of the time.
So, with this out of the way, for me, it boils down to data collection. If Google is developing software without collecting data in the background, then I’m OK to use it. Just because it bears the name doesn’t mean it is automatically evil. We saw with Microsoft, for example, that they are collecting data wherever they can. Their C++ package manager vckpg collects data regardless, even if you explicitly revoke consent, but I have not found the same to be the case with gtest. So, in that respect, I am OK with using their software.
As I mentioned before, you may have other opinions and that is alright. I would still encourage you to finish reading this article, and then have a look at Catch2, which is probably the best alternative from what I can see. This one is driven by independent software developers so you may feel better at home there. It doesn’t do mocking, but for CFD applications, mocking is pretty much optional anyway.
Reasons for Google’s gtest
With that out of the way, why is gtest my preferred choice? Well, first of all, it does cover pretty much all use cases we ever will come across, as I have mentioned before. It is pretty intuitive, well documented, and has a lot of different assertions we can use to check if values and conditions are met for the test to pass.
I first had to use gtest when working on a larger project in a team, so it was pretty much bestowed onto me. Even if I had no idea of software testing back then, I found that it was pretty quick to pick up. I mentioned above that it is well documented, the best place to start is their gtest primer. You can probably read through it in about 10-15 minutes and it is an excellent start to get familiar with the framework.
If you went through that, then you have a more in-depth advanced version, which will cover pretty much everything there is to know, and it will take some more time to step through that document. But afterwards, you will be a gtest wizard. Since I started using it, they have added a section for mocking, typical mocking use cases in the form of a cookbook, and a cheat sheet to quickly lookup use cases for mocking. Well-documented software is always easier to pick up.
Speaking of it, gtest comes with its own mocking framework, which we may not necessarily need, but depending on your use case, mocking may be important later for specific use cases. In that case, it is great to know that you don’t have to switch testing frameworks and instead just have to do some more reading up on your current framework.
It does follow the xUnit testing framework and so if you have used testing in a different language, chances are you just have to learn new syntax but not an entirely new testing philosophy. It also integrates rather well with existing code and has a low dependency overhead, meaning the libraries are rather small, yet quite powerful.
In the end, as is so often the case, it comes down to taste. There are too many testing frameworks out there so it doesn’t really matter which one we pick. Choose one and stick with it, I’ll over you an insight into gtest in this series, but as I said above, have a look around, you may have very different or constraint requirements, in which case a different testing framework may be the right choice for you.
Compiling and installing gtest
Compiling and testing the library is rather straightforward. It can be compiled using CMake and Bazel, the latter being a build system developed by Google as well. We have looked into details of how to compile libraries with different generators, so I won’t get into details here. If you missed that part, have a look at my article on how to integrate external libraries into your project. At the end of the article, there is a walkthrough of how to compile the CGNS library and if you understand that, gtest is a piece of cake in comparison.
We will be using CMake to compile the library, so once you have downloaded gtest from their GitHub repository, go to your download location and extract the archive. I am working with version 1.14.0 here, but you may be using a later release by the time you read that. As of version 1.14.0, no external dependencies are required to make gtest work, but it has been announced that the developers will be relying on Abseil in the future, you if you use a later release, you may have to download and compile Abseil and then link the library against gtest during compilation.
If you look at the above-linked article on how to compile the CGNS library, which itself depends on the HDF5 library, which, in turn, has its own dependencies, you will see how you can compile libraries that depend on other, previously compiled libraries. So, should you read this in the future and be required to compile Abseil first, then you have a starting point on how to do that. If you can’t be bothered, just download the release 1.14.0 from gtest and you won’t have this issue.
If you are looking for an even lazier approach, I have also provided two installation scripts that will download and compile gtest, which you can download at the beginning of this article. If you want to compile it yourself, read on.
After downloading and extracting the archive, open your Developer PowerShell for VS year, where year is one of Microsoft’s releases for Visual Studio, e.g. 2016, 2019, 2022, 2025, etc. Remember that you have to tell the developer PowerShell that it should pick the 64-bit compiler, and not the 32-bit compiler, otherwise, you may have issues compiling libraries later together with your code if they mismatch. If you are on a UNIX-based operating system, then simply open your terminal and pat yourself on the shoulder for using a sensible operating system.
Create a folder called build within the extracted archive of gtest and change into that directory. If you are on Windows, type
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=C:\libraries\ ..
If you are on UNIX instead, type the following:
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=~/libraries ..
In both cases, I assume that you want to install the library into a sensible default location, here either C:\libraries\ on Windows or ~/libraries/ on UNIX, where ~ indicates the home directory, i.e. /home/<your-user-name>/. Of course, you can change this directory to anything you like, these are just the default locations I like to use.
After the library has been configured and you don’t see any error messages, continue with the next step, which is first building the library by compiling all source files and linking them into the final, static library, as well as installing the library. Installing here simply means copying the generated libraries and required header files into the install directory we specified above with the CMAKE_INSTALL_PREFIX variable. The command for this is:
cmake --build . --target install --config Release -j 4
This command is used by both Windows ad UNIX-based operating systems. After you have completed this step, you can ensure that you have lib/gtest.lib (Windows) or lib/libgtest.a available within you install directory. There will be a few more, but for the moment that is the library we are interested in (for example, you should also see lib/gmock.lib (Windows) or lib/libgmock.a (UNIX) which will be used for mocking).
Within the include/ directory, you should ensure that you have a folder called gtest and gmock, in particular, within the gtest folder, we should see a header file called gtest.h. If you can see these files, then your compilation and installation of the library has worked correctly and you are ready to move on to the next step.
Using Google test for our complex number class
In this section, we will simply transform the tests we wrote for the complex number class in the previous article, and use gtest instead. We will see that changing from the tests we already provided to gtest is rather straightforward. Very little change is required for the syntax. Since we can assert a few more things with gtest, we will also look into additional edge cases we did not test for with our own testing framework. These include division by zero and exception handling.
Thus, we have to modify our complex number class slightly, and then write additional tests. With that in place, we should have a robust implementation for complex numbers, that is very lightweight, yet it does pretty much everything we need if we wanted to use this implementation, for example, in our CFD solver for signal analysis using a fast Fourier transformation (FFT).
Project structure
The project structure for the archive that you can download at the top of this article is given below. Comparing that to our previous project with our own testing framework, we have lost the floatingPointEqual.hpp file within the tests folder, which we use to compare floating-point numbers. So the structure is a bit simplified in this project. We will have to modify the build scripts in this case to account for the fact that we now depend on an external dependency.
root
├── src
│ └── complexNumber.hpp
├── tests
│ └── unit
│ └── testComplexNumbers.cpp
├── runTests.ps1
└── runTests.shBuild scripts
Right, build script time. Brace yourself, it is one of those times where we have to be patient with Microsoft. It wants to play with the big boys and we can’t expect miracles. Anyways, we can get our code to work across platforms if we do some unnecessary coding, but at least we will create cross-platform code. That’s just how it is, if you want to target different platforms, you have to be ready to compromise. I’ll make sure to point out where we are compromising, so you can test this for yourself if you want to.
Windows
For Windows, we follow the same pattern we used in previous build scripts as well. We can see that we first want to remove the build directory on line 2 and then recreate it so that we get rid of any old files we have built. In general, this is a bad idea, but for smaller projects, the overhead to recompile all files is small and so we won’t notice this overhead.
The reason I do that is simple, I build the code using both Windows and Ubuntu on the same platform, so I want to make sure all files are gone before I switch platforms, otherwise I may get nasty linker messages if I am trying to include files compiled from a different platform in my current executable. It just preserves my sanity, slightly.
We compile the file containing our tests (now using gtest) on line 6, which means we need to include the folder in which to find our gtest header files. In my case, this is in C:\libraries\include\. On line 9, we create an executable called testComplexNumbers.exe, and we need to link our object file against gtest.lib here, for which we also specify the file location, i.e. C:\libraries\lib\. Finally, we also execute the tests after the compilation is finished, which can be seen on line 12.
# clean up before building
Remove-Item .\build -Force -Recurse
New-Item -Name "build" -ItemType "directory"
# compile source files into object files
cl.exe /nologo /EHsc /std:c++20 /Zi /I. /I"C:\libraries\include\" /c .\tests\unit\testComplexNumbers.cpp /Fo".\build\testComplexNumbers.obj"
# create test executable
cl.exe /nologo /EHsc /std:c++20 /Zi .\build\testComplexNumbers.obj /Fe".\build\testComplexNumbers.exe" /link /MACHINE:x64 /LIBPATH:"C:\libraries\lib" gtest.lib
# run tests
.\build\testComplexNumbers.exeNow, every C++ code needs a main() function, and gtest provides that for us. If we don’t want to provide one ourselves, that we could, in theory, also lin against gtest_main.lib, which will provide us with a main() function. If we don’t link against this library, we have to provide the main function ourselves, which we will do later.
The reason we opt to write our own main() function, is the issue I mentioned above. Microsoft’s cl compiler and its linker are unable to link against gtest_main.lib correctly, and you will get a linker error saying that no entry point is defined. I have tried compiling gtest in a few ways, but all seem to produce the same error. Providing the main() function ourselves resolves this issue.
But, try it yourself. You may have a slightly different installation and it may be that my environment just isn’t set up properly. But even if it does work, take this as a warning; even if it does work for you, it may not for others, so it is worth considering sticking to the same approach as I am so that you avoid nasty bug-reports from users who are using Windows.
UNIX
We do pretty much the same thing on UNIX using bash. First, we remove and recreate the build folder on lines 4-5, compile the new test function on line 8, this time using gtest, hence we need to include the gtest include folder, which in my case is at ~/libraries/include. On line 11, we create the executable and link against gtest, which is located in ~/libraries/lib, which we tell the linker through the -L flag. We then link against gtest using the -l flag.
#!/bin/bash
# clean up before building
rm -rf build
mkdir -p build
# compile source files into object files
g++ -std=c++20 -I. -I ~/libraries/include/ -c -g ./tests/unit/testComplexNumbers.cpp -o ./build/testComplexNumbers.o
# create test executable
g++ -std=c++20 ./build/testComplexNumbers.o -o ./build/testComplexNumbers -L ~/libraries/lib -lgtest
# run tests
./build/testComplexNumbersOn UNIX, you can now link against gtest_main, i.e., you can add -lgtest_main at the end of line 11, and then you will get a main() file provided. In this case, you will need to delete the main() function from your test file, which we will look at below. Again, try it yourself, play around and see the results.
Rewriting the unit tests
Below are the new tests using gtest. These are located in the tests/unit/testComplexNumbers.cpp file. You’ll see that we gtest’s header file on line 3 to have access to the library. The remaining tests are almost the same, with a few additions to test for edge cases, i.e. floating point numbers which store NaN (not a number) and checking for division by zero. Some of these cases will throw an exception and so we catch these exceptions when they are thrown. Have a look through the file, and we will discuss it below in more detail.
#include <limits>
#include "gtest/gtest.h"
#include "src/complexNumber.hpp"
TEST(ComplexNumberTest, TestComplexMagnitude) {
// Arrange
ComplexNumber a(3.0, 4.0);
// Act
auto magnitude = a.magnitude();
// Assert
ASSERT_DOUBLE_EQ(magnitude, 5.0);
}
TEST(ComplexNumberTest, TestComplexConjugate) {
// Arrange
ComplexNumber a(3.0, 4.0);
// Act
a.conjugate();
// Assert
ASSERT_DOUBLE_EQ(a.Re(), 3.0);
ASSERT_DOUBLE_EQ(a.Im(), -4.0);
}
TEST(ComplexNumberTest, TestComplexNumberWithNaN) {
// Arrange
ComplexNumber a(1.2, -0.3);
ComplexNumber b(1.8, 5.3);
// Act
a.setRe(std::numeric_limits<double>::quiet_NaN());
// Assert
ASSERT_THROW(ComplexNumber a(1.2, std::numeric_limits<double>::quiet_NaN()), std::runtime_error);
ASSERT_THROW(a + b, std::runtime_error);
ASSERT_THROW(a - b, std::runtime_error);
ASSERT_THROW(a * b, std::runtime_error);
ASSERT_THROW(a / b, std::runtime_error);
}
TEST(ComplexNumberTest, TestComplexAddition) {
// Arrange
ComplexNumber a(1.2, -0.3);
ComplexNumber b(1.8, 5.3);
// Act
ComplexNumber c = a + b;
// Assert
ASSERT_DOUBLE_EQ(c.Re(), 3.0);
ASSERT_DOUBLE_EQ(c.Im(), 5.0);
}
TEST(ComplexNumberTest, TestComplexSubtraction) {
// Arrange
ComplexNumber a(1.2, -0.3);
ComplexNumber b(1.8, 5.3);
// Act
ComplexNumber c = a - b;
// Assert
ASSERT_DOUBLE_EQ(c.Re(), -0.6);
ASSERT_DOUBLE_EQ(c.Im(), -5.6);
}
TEST(ComplexNumberTest, TestComplexMultiplication) {
// Arrange
ComplexNumber a(1.2, -0.3);
ComplexNumber b(1.8, 5.3);
// Act
ComplexNumber c = a * b;
// Assert
ASSERT_DOUBLE_EQ(c.Re(), 3.75);
ASSERT_DOUBLE_EQ(c.Im(), 5.82);
}
TEST(ComplexNumberTest, TestComplexDivision) {
// Arrange
ComplexNumber a(1.0, -2.0);
ComplexNumber b(1.0, 2.0);
// Act
ComplexNumber c = a / b;
// Assert
ASSERT_DOUBLE_EQ(c.Re(), -0.6);
ASSERT_DOUBLE_EQ(c.Im(), -0.8);
}
TEST(ComplexNumberTest, TestComplexDivisionWithNaN) {
// Arrange
ComplexNumber a(1.0, -2.0);
ComplexNumber b(0.0, 0.0);
// Act
// Assert
ASSERT_THROW(a / b, std::runtime_error);
}
int main(int argc, char** argv)
{
testing::InitGoogleTest(&argc, argv);
RUN_ALL_TESTS();
}First of all, let’s start at the bottom, on lines 108-112 we see the main() function that we have been talking about above. If we remove this function, then we have to link against gtest_main. As I said, this will work on UNIX, but not on Windows, at least for me. But for completeness, you now know which function I was referring to. And, just to finish this thought, if we had more than one test file, we would only have to define the main() function once, so you may want to stick it into its own source file.
To kick off the discussion on the tests using gtest, I thought it would be useful to compare two tests, i.e. how we wrote tests with the previous test framework we developed ourselves (framework is probably a strong word here, we added some code to compare floating point numbers, that’s it) and how we do that now using gtest. We looked at the complex addition case previously, so let’s contrast them here and discuss their similarities.
Comparing our custom-made test with gtest for complex addition
void testComplexAddition() {
// Arrange
ComplexNumber a(1.2, -0.3);
ComplexNumber b(1.8, 5.3);
// Act
ComplexNumber c = a + b;
// Assert
assert(floatingPointEqual(c.Re(), 3.0));
assert(floatingPointEqual(c.Im(), 5.0));
}TEST(ComplexNumberTest, TestComplexAddition) {
// Arrange
ComplexNumber a(1.2, -0.3);
ComplexNumber b(1.8, 5.3);
// Act
ComplexNumber c = a + b;
// Assert
ASSERT_DOUBLE_EQ(c.Re(), 3.0);
ASSERT_DOUBLE_EQ(c.Im(), 5.0);
}On the left, we have the test code we wrote in our previous article using our own testing approach, and on the right, we have the same implementation using gtest. You see, the structure is the same, i.e. we are using in both cases the Arrange, Act, Assert (AAA) structure. The only differences are how we structure the code on the first line and the assertions in the assert section.
Each test in gtest is assigned a group, and then within that group, we can have different test cases. The group we are currently in is the ComplexNumberTest, which contains essentially all unit tests for the complex number class. Then, the test we are looking at is the TestComplexAddition. These are the first and second arguments on line 1, respectively.
Then, on lines 10-11, we make sure that the real and imaginary parts are equal to what we expect them to be, and we are using the ASSERT_DOUBLE_EQ macro, which will do the same job as the floatingPointEqual() function we used in the previous article, shown on the left above.
Some more thoughts on floating-point number comparisons
There are additional gtest functions available for floating point number comparisons, for example, ASSERT_FLOAT_EQ and ASSERT_NEAR. The former is the same floating point comparison for floats, while the latter allows us to compare values within a certain tolerance. This may be useful when working with rational numbers, e.g.
ASSERT_NEAR(1.0/7.0, 0.142857, 1e-6);If you look up the documentation for either ASSERT_DOUBLE_EQ or ASSERT_FLOAT_EQ, you will see that it states that we check if the floating point values are within 4 ULPs. Trying to explain that goes a bit beyond what I want to cover here, but have a look again at the write-up on how to compare floating point numbers by Bruce Dawson. We looked at his write-up in the previous article when we developed our floating point comparison algorithm.
I mentioned that we won’t go as deep into floating-point number comparison as he did, and if you read beyond the comparison we implemented, then you will see that he starts talking about ULPs, which stands for units in the last place. In a nutshell, even floating point numbers are represented as integer values, which means their precision is finite. If we say that our values are allowed to differ by 4 ULPs for floating-point comparison, what we are saying is that the integer representation is allowed to be plus minus 2 units away from the value we are comparing against.
If you understand floating-point numbers, then this will make sense, if you don’t, then the article linked above is an excellent resource to learn about it. The C++ standard template library (STL) has you covered as well, you can ask it to tell you what the next floating point number is past a given floating point number. In other words, given a floating point number f, what is the next value after f, which is exactly 1 ULP ahead. We use the std::nextafter function for that.
But, chances are that you simply don’t care about going into that much level of detail about floating-point numbers, and that’s OK too. In which case, let’s continue.
What about testing exception handling?
I mentioned above that we also want to test for exception handling, i.e. what happens if we have to deal with NaN? Let’s say that if we detect that we have not a number given for whatever reason, we want to throw an exception. This means we will have to change our code, but this also means that we need to test for its correct usage. Below is an implementation that checks that any computation with a NaN should catch an exception that was thrown. This is copied from the test code above.
TEST(ComplexNumberTest, TestComplexNumberWithNaN) {
// Arrange
ComplexNumber a(1.2, -0.3);
ComplexNumber b(1.8, 5.3);
// Act
a.setRe(std::numeric_limits<double>::quiet_NaN());
// Assert
ASSERT_THROW(ComplexNumber a(1.2, std::numeric_limits<double>::quiet_NaN()), std::runtime_error);
ASSERT_THROW(a + b, std::runtime_error);
ASSERT_THROW(a - b, std::runtime_error);
ASSERT_THROW(a * b, std::runtime_error);
ASSERT_THROW(a / b, std::runtime_error);
}In this case, we set up our complex numbers as we did before on lines 3-4, but within the act section, we are setting now the real part to NaN, which requires the limits header file. Then, with one of the components set to NaN, we check in the assert section that any calculation results in a std::runtime_error exception being thrown. We did not implement that earlier and will need to do so to make it work.
But remember, we follow the test-driven development approach, so we write the tests first, and only then do we start to implement the core functionality.
Catching division by zero errors
One last thing I want to look at is the division by zero case. This test is also copied from above and reproduced here:
TEST(ComplexNumberTest, TestComplexDivisionWithNaN) {
// Arrange
ComplexNumber a(1.0, -2.0);
ComplexNumber b(0.0, 0.0);
// Act
// Assert
ASSERT_THROW(a / b, std::runtime_error);
}I did not mention that for the test before, but we can see here again on line 1 that we are still in the ComplexNumberTest group, and the new test here is called TestComplexDivisionWithNaN.
We see that the act section now is empty. The reason is that we cannot store an exception that was thrown in a variable and then check that this variable stores an exception, that just isn’t how C++ works. But, it shows an important point; we deviate here from here from the AAA pattern. It is more important to have a clean code design than trying to awkward production code just to fit the test. It is better to deviate in the test code for a cleaner design than the other way around.
Thus, we can see that the actual act section has been moved into the assert section and we are testing that a complex division by zero will throw an exception immediately.
These are all changes. By now, hopefully, the switch from our bespoke tests that we wrote in the previous article to gtest is clear and you get an idea of how to use gtest. We will see it in more action in the next articles, but this article serves as a gentle introduction. Testing with gtest does not get much more complicated unless you want it to be. What we have covered in this article covers most, if not all, of the use cases you will have for it.
Updated complex number class
We saw that we added some additional functionality to our complex number class, so I have provided the updated implementation below. I will point out underneath the code what has changed, but these changes are minimal.
#pragma once
#include <iostream>
#include <cmath>
#include <stdexcept>
#include <limits>
class ComplexNumber {
public:
ComplexNumber(double real, double imaginary) : _real(real), _imaginary(imaginary) {
isNan(real, imaginary);
}
~ComplexNumber() = default;
double Re() const { return _real; }
double Im() const { return _imaginary; }
void setRe(double real) { _real = real; }
void setIm(double imaginary) { _imaginary = imaginary; }
void conjugate() { _imaginary *= -1.0; }
double magnitude() const { return std::sqrt(std::pow(_real, 2) + std::pow(_imaginary, 2)); }
ComplexNumber operator+(const ComplexNumber &other) {
isNan(_real, _imaginary);
isNan(other._real, other._imaginary);
ComplexNumber c(0, 0);
c._real = _real + other._real;
c._imaginary = _imaginary + other._imaginary;
return c;
}
ComplexNumber operator-(const ComplexNumber &other) {
isNan(_real, _imaginary);
isNan(other._real, other._imaginary);
ComplexNumber c(0, 0);
c._real = _real - other._real;
c._imaginary = _imaginary - other._imaginary;
return c;
}
ComplexNumber operator*(const ComplexNumber &other) {
isNan(_real, _imaginary);
isNan(other._real, other._imaginary);
ComplexNumber c(0, 0);
c._real = _real * other._real - _imaginary * other._imaginary;
c._imaginary = _real * other._imaginary + _imaginary * other._real;
return c;
}
ComplexNumber operator/(const ComplexNumber &other) {
isNan(_real, _imaginary);
isNan(other._real, other._imaginary);
double denominator = other._real * other._real + other._imaginary * other._imaginary;
if (std::abs(denominator) < std::numeric_limits<double>::epsilon())
throw std::runtime_error("Complex number division by zero");
ComplexNumber c(0, 0);
c._real = (_real * other._real + _imaginary * other._imaginary) / denominator;
c._imaginary = (_imaginary * other._real - _real * other._imaginary) / denominator;
return c;
}
friend std::ostream &operator<<(std::ostream &os, const ComplexNumber &c) {
os << "(" << c._real << ", " << c._imaginary << ")";
return os;
}
private:
void isNan(double real, double imaginary) const {
if (std::isnan(real) || std::isnan(imaginary))
throw std::runtime_error("Complex number is NaN");
}
private:
double _real;
double _imaginary;
};Checking for NaN
The first and most influential change that we are going to implement is the checking against NaN (not a number). We have provided a new private function for that on lines 74-77, which checks that both real and imaginary parts are real numbers. The function std::isnan() is located within the cmath header which we already included before.
With this function available, we can now add it in front of all arithmetic operators where we add, subtract, multiply and divide numbers to ensure that we only continue if numbers are real. We also put a check in the constructor so that we are not even constructing a complex number object if the real or imaginary parts are corrupted to begin with.
New setRe() and setIm() functions
The new tests require us to change the real and imaginary parts before we continue. This is to bypass the check in the constructor for NaN. If we construct an object with NaN, then we always throw an error, so we would never be able to test the error checking for the operators for addition, subtraction, multiplication, and division. So we construct a complex number with real values and only then set one of them to NaN with either the setRe() or setIm() setter, which are given on lines 18 and 19, respectively.
Catching division by zero
The new operator/() function on lines 54-66 has now been extended to not only check for NaNs (lines 55-56) but also to catch a division by zero on lines 58-60. We calculate the value for the denominator of the fraction first, and if that is less than the smallest number we can represent (epsilon, a value close to zero), then we say that the denominator is zero and we have a division by zero situation and throw and throw a runtime error.
If you have read up on ULPs above, when we talked about floating point number comparisons, then we can also say that epsilon is the distance between zero and the next positive floating point value we can represent, or one ULP to the right. If you can’t be bothered about ULPs, then just see epsilon as the smallest number we can represent with a double before we lose precision due to round-off errors.
Inspecting the output
Now that we have generated all of these tests, it is time to inspect the test runner. The test runner is part of any xUnit testing framework and it is responsible for collecting all of your tests and then executing them. Running the tests is as simple as running the executable we have generated. If you do that, you will get the output as shown below, with some nicer colours.
[==========] Running 8 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 8 tests from ComplexNumberTest
[ RUN ] ComplexNumberTest.TestComplexMagnitude
[ OK ] ComplexNumberTest.TestComplexMagnitude (0 ms)
[ RUN ] ComplexNumberTest.TestComplexConjugate
[ OK ] ComplexNumberTest.TestComplexConjugate (0 ms)
[ RUN ] ComplexNumberTest.TestComplexNumberWithNaN
[ OK ] ComplexNumberTest.TestComplexNumberWithNaN (0 ms)
[ RUN ] ComplexNumberTest.TestComplexAddition
[ OK ] ComplexNumberTest.TestComplexAddition (0 ms)
[ RUN ] ComplexNumberTest.TestComplexSubtraction
[ OK ] ComplexNumberTest.TestComplexSubtraction (0 ms)
[ RUN ] ComplexNumberTest.TestComplexMultiplication
[ OK ] ComplexNumberTest.TestComplexMultiplication (0 ms)
[ RUN ] ComplexNumberTest.TestComplexDivision
[ OK ] ComplexNumberTest.TestComplexDivision (0 ms)
[ RUN ] ComplexNumberTest.TestComplexDivisionWithNaN
[ OK ] ComplexNumberTest.TestComplexDivisionWithNaN (0 ms)
[----------] 8 tests from ComplexNumberTest (8 ms total)
[----------] Global test environment tear-down
[==========] 8 tests from 1 test suite ran. (12 ms total)
[ PASSED ] 8 tests.Let’s start from the bottom, we see on line 24 that a total of 8 tests were performed and the overall status is that all tests passed. We can see on the line before that all tests were performed in 12ms. This is good, as we want to add hundreds, if not thousands of unit tests to larger projects and we want to be able to catch any possible regression (bug) early on. If we just have to wait a few seconds, at most, then we can run them often. If they take minutes to run, we run them less frequently and regressions will slip through and need to be fixed later.
On lines 4-19, we see that we are running all the 8 different tests. We get two lines per test, the first one will indicate that we are running a test, which is identified by the test group and name that we specified for each test (the two strings that go into the TEST() function, and the second one will return either OK or fail, depending on the test outcome. It will also print the time it took to perform each test, and we can see that they are close to 0ms.
Making the tests fail
In our case, all tests pass, because I made sure they do, but let’s inspect what happens if tests are not passing. After all, this is why we want to use gtest. We want to write tests that fail initially and then start to provide an implementation that will make the test pass. That is the heart of test-driven development and so we need to understand what a failing test looks like as well.
For that, let’s go back to our complex addition test and make the following modifications
TEST(ComplexNumberTest, TestComplexAddition) {
// Arrange
ComplexNumber a(1.2, -0.3);
ComplexNumber b(1.8, 5.3);
// Act
ComplexNumber c = a + b;
// Assert
ASSERT_DOUBLE_EQ(c.Re(), 3.1);
ASSERT_DOUBLE_EQ(c.Im(), 5.0);
}I have changed line 10 here, where we are expecting the real part to be equal to 3.1, but in reality it is 3.0. So this test should now fail, let’s see what the output is from the test runner in this case.
[==========] Running 8 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 8 tests from ComplexNumberTest
[ RUN ] ComplexNumberTest.TestComplexMagnitude
[ OK ] ComplexNumberTest.TestComplexMagnitude (0 ms)
[ RUN ] ComplexNumberTest.TestComplexConjugate
[ OK ] ComplexNumberTest.TestComplexConjugate (0 ms)
[ RUN ] ComplexNumberTest.TestComplexNumberWithNaN
[ OK ] ComplexNumberTest.TestComplexNumberWithNaN (0 ms)
[ RUN ] ComplexNumberTest.TestComplexAddition
.\tests\unit\testComplexNumbers.cpp(54): error: Expected equality of these values:
c.Re()
Which is: 3
3.1
Which is: 3.1000000000000001
[ FAILED ] ComplexNumberTest.TestComplexAddition (0 ms)
[ RUN ] ComplexNumberTest.TestComplexSubtraction
[ OK ] ComplexNumberTest.TestComplexSubtraction (0 ms)
[ RUN ] ComplexNumberTest.TestComplexMultiplication
[ OK ] ComplexNumberTest.TestComplexMultiplication (0 ms)
[ RUN ] ComplexNumberTest.TestComplexDivision
[ OK ] ComplexNumberTest.TestComplexDivision (0 ms)
[ RUN ] ComplexNumberTest.TestComplexDivisionWithNaN
[ OK ] ComplexNumberTest.TestComplexDivisionWithNaN (0 ms)
[----------] 8 tests from ComplexNumberTest (16 ms total)
[----------] Global test environment tear-down
[==========] 8 tests from 1 test suite ran. (20 ms total)
[ PASSED ] 7 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] ComplexNumberTest.TestComplexAdditionWe see at the bottom of our console output that now only 7 tests pass while 1 test failed. So overall, our test suite failed. But gtest is very verbose and tells us exactly what went wrong. We run the complex addition test on line 10 and get a message that on line 54 within the testComplexNumbers.cpp file, we expect c.Re() and 3.1 to be equal. We are also getting the values of these two expressions, for c.Re() the value is 3.0.
We can use this information to now check which value is unexpected. In our case, we now the second argument is wrong but typically it is the other way around, i.e. the expected value is correct but the computed one is wrong. This tells us that our implementation is wrong and so we go back to our implementation and debug it until we have found the error and the test is passing. Then we move on to the next failing test, or, if everything is working, the next feature we want to implement.
Summary
In this article, we departed from our own testing implementation and we saw that Google’s gtest is a pretty well-rounded testing framework to help us catch regressions (bugs) early on in the development process. Even if we just scratched the surface of what gtest can do, what we saw is what you will use most of the time. If you want to start using gtest to test your own CFD solver code, then you have already most of the knowledge you need. It doesn’t get much more complicated than that.
In the next articles, we will look at a few more examples of how to integrate gtest into more complex test scenarios. This will flush out some more use cases we may have and show how to use some of the more advanced features of gtest. These will help us to clean up large tests that require a lot of preparation and repeated code set-up.
From now on, I would encourage you to start using gtest (or any other testing framework) for any projects that go beyond a few hundred lines of code. You will lose some time for the test setup and getting gtest compiled, but you will save time in the long run. Give it a try and see it for yourself, you will feel a lot more empowered and confident to make changes to code sections that you have developed earlier. If you break the code, you know it will be caught by your tests, so cleaning up older code sections becomes a rewarding task.
Tom-Robin Teschner is a senior lecturer in computational fluid dynamics and course director for the MSc in computational fluid dynamics and the MSc in aerospace computational engineering at Cranfield University.