In this article, we look at the last remaining topic in our series on software testing; Mocking. This aspect plays a a very big role in the testing of web and mobile applications, but not so much for console applications such as CFD solvers. However, there are specific use cases where mocking can help us avoid writing brittle and slow tests, and we will look at some specific cases where we may want to use mocking instead of using actual production code.
We exemplify mocking by writing a quick project in which we read some input parameter files but instead of reading actual files from disk during the test, which can be rather slow, we replace the function to read a file from disk with a mock. Since we have already used Google Test (gtest) in previous examples, we stick with this testing framework, which happens to have a mocking framework included (called gmock). We will see how simple and quickly we can write a mocked interface and create expected behaviours for these mocks with our example.
By the end of this article, we will have gone through a real mocking example and have seen how to integrate that into our unit testing workflow. We also discuss where mocking may be useful in terms of testing CFD applications and you should be sensibilised as to when you may want to use mocking in your own projects.
Download Resources
All developed code and resources in this article are available for download. If you are encountering issues running any of the scripts, please refer to the instructions for running scripts downloaded from this website.
- Download: Software-Testing-Part-5.zip
In this series
- Part 1: How to get started with software testing for CFD codes
- Part 2: The test-driven development for writing bug-free CFD code
- Part 3: How to get started with gtest in C++ for CFD development
- Part 4: How To Test A CGNS-based Mesh Reading Library Using gtest
- Part 5: How To Test A Linear Algebra Solver Library Using gtest
- Part 6: How to use mocking in CFD test code using gtest and gmock
- Part 7: What is test coverage and how to use LCOV/GCOV for testing
In this articles
Introduction to mocking
Up until this point, we have taken a deep dive into testing and used Google’s gtest testing framework to write our unit, integration, and system tests. We have applied this to our linear algebra solver and mesh reading library to write these tests with and without test fixtures, all while following the test-driven development approach.
What we have learned thus far is sufficient to test pretty much any project and there will be rarely any cases that will be difficult to test. However, when we first started to talk about testing, I mentioned that mocking is a way of replacing external dependencies. Classical examples are interaction with a graphical user interface or external dependencies such as web requests and accessing a database.
I also mentioned that there are two different kinds of testing approaches, the classical school and the London school of unit testing. In the classical approach, we test our code without replacing dependencies with mocks, whereas in the London school approach, we replace all external dependencies which are mutable by mocks. Mutable here means any dependencies whose internal state can change. We used the example of the matrix-vector multiplication where we may want to mock the vector class. The values (internal state) of the vector can change, and thus it is mutable.
However, even if we want to stick with the classical approach and don’t want to replace dependencies (be it internal or external) with mocks, there are still good examples where we may want to consider mocking. A list of possible examples is given below:
- File reading: Depending on the file type and the complexity of reading from it, input and output operations may become excessively slow. If we have to go through a few of these files in our test base, then executing tests will become rather slow. If we compare the time it took to run our mesh reading unit tests and our linear algebra solver unit, integration, and system test combined, the mesh reading tests took 243 milliseconds, vs. the 1 millisecond of the linear algebra solver library tests. Thus, the mesh file reading has a clear overhead in this test.
- Random number generators: If we want to include code that relies on random numbers, such as a synthetic turbulence generator for inlet boundary conditions for large eddy simulations (LES), then we probably want to have some way of influencing the way the random numbers are generated. We need this to be able to repeatedly test our code that does not involve any random behaviour.
- Hardware dependencies: CFD applications are notoriously CPU-intensive and as a result, parallel computing is a must for any serious 3D solver. Testing and parallel computing bring their own challenges. For example, we may want to include GPU computing using CUDA, but what if the test is run on a computer that doesn’t have a GPU? If a test fails because the hardware is not available, our tests become brittle and don’t test the code but rather the hardware that is available. Mocking hardware dependencies is a good idea in this case.
- Solver components: Related to the file reading example provided above, mocking entire solver components may become necessary to save time. Let’s assume we do mesh generation as part of our solver. Even generating a small mesh may take a few seconds, depending on the complexity of our mesh generator. If we want to write an integration test that calculates mesh quality metrics based on a generated mesh, we may not want to wait for a few seconds just to perform a calculation that takes milliseconds. Mocking components can be lucrative in this case.
So we can see that there are good examples where we want to mock dependencies. So let’s have a look at the prerequisite of mocking, how do we prepare our code to make use of mocking?
Preparing code for mocking
There is pretty much just one prerequisite and that is that you have to have a class hierarchy, i.e. you need to have a base class (sometimes we also refer to it as an interface), from which other classes inherit. Take our CGNS library as an example, we had the mesh reading base class, from which we inherited both a structured and unstructured mesh reading class. Or, in the case of our linear algebra solver library, we create a solver base class from which we inherited to create the conjugate gradient class.
The idea here is that we inherit from this base class and then tell our mocking framework which functions exist and which ones we want to potentially override, and then later in the test code, we create an object from this mocked class where we then specify what these mocked classes ought to return. This sounds overly complicated, so let’s jump straight in and look at an example where we can discuss the nuances of mocking more easily.
Example: Mocking the reading of parameter files
I want to look at a proper example that is not too complicated to follow and look at something new and fresh. We have been looking at the mesh reading and linear algebra solver library extensively and while there is a good potential to mock something in either case, I wanted to come up with a fresh example and look at something we may want to implement into our solver later: parameter reading from an input file.
Prerequisites: The JSON file format
When it comes to parameter reading, there are countless file types and bespoke solutions available that allow us to read data from disk into our solver. I want to look at a somewhat non-native C++ solution, which is however widely used in pretty much every other programming language. I am talking about the Javascript Object Notation, os JSON file format.
JSON files follow the notation we would use in Javascript to define classes. They just turn out to be very useful and straightforward to understand and so the file type itself has been widely adopted for parameter files. Other popular choices are YAML and TOML, which look more or less similar to JSON and are used in other parts of programming. We have come across TOML files already when we looked at Conan – the C++ package manager – and YAML files will cross our path later when we discuss continuous integration and delivery.
You won’t find many CFD solvers using either JSON, TAML, or TOML files, and I think the reason is that CFD developers quite often seem to be disconnected from the rest of the software engineering world. Pretty much everyone else working in the broad field of software engineering uses JSON, YAML, and TOML on a daily basis, so I think we need to move on as well and learn at least one of these file formats.
If we wanted to store a few inputs in a JSON file, we would have to define the file first, so let’s look at an example:
{
"meshing": {
"file": "structured2D.cgns"
},
"solver": {
"inner_iterations": 20,
"outer_iterations": 2000,
"CFL": 0.5,
"residual_threshold": {
"pressure": 1e-10,
"velocity": 1e-10
},
"underrelaxation": [0.3, 0.7, 0.7],
"turbulence": {
"model": "k-epsilon",
"sigma": 1.0
}
},
"write_output": true
}Each JSON file starts with opening and closing curly braces (i.e. {}). Then, we simply define key/value pairs, where the key is given as a string, and the value can be anything between a string (line 15), a boolean (line 19), a sub-dictionary (lines 2-4, 5-18, 9-12, and 14-17), an integer (line 6), a floating point (line 8), or an array of values (line 13). Each key/value pair has to be ended with a comma unless it is the last entry within the JSON file or sub-dictionary (e.g. see lines 11, 16, and 19). And this is a JSON file explained in one paragraph, congratulations, you are a JSON expert now.
So we see how we can easily write a JSON file, but how about reading it? Well, if it has this nice structure, it would be easy to develop a class that parses a JSON file and returns the key/value pairs to us, but luckily, we don’t have to reinvent the wheel and can use off-the-shelve solutions.
Reading JSON files in C++ with the nlohmann::json library
When it comes to reading JSON files in C++, Niels Lohmann’s JSON library is pretty much what you want to be using, which is among the most popular libraries used for C++. The beauty of it is that it is a single header file that you need to throw into your project and you have access to the library, i.e. it is a single-file header-only library.
So, if you head over to the latest release, and scroll down to the Assets section (at the bottom of the release notes), you will find a file called json.hpp. If you can’t be bothered, then use this link to download it directly, but this will be for a specific library version (but to be fair, the core of the library doesn’t change).
I have put all my libraries into C:\libraries\ and ~/libs on Windows and UNIX, respectively, and all header files live inside the include directory. If you look at example codes using the nlohmann::json library, it is common that the header file is located in a directory called nlohmann, and so I will follow that convention. This means that my json.hpp file is located in C:\libraries\include\nlohmann\json.hpp on Windows and ~/libs/include/nlohmann/json.hpp on UNIX. If you want to use my build scripts, then you will need the same file structure.
Project structure and build files
I don’t want this to be a very long section, so I have decided to discuss the project structure together with the build scripts for Windows and UNIX.
The project structure for this discussion is shown below, where we have a parameter reading base class in the src/ directory, from which we derive a variant that implements a certain interface using JSON files as the parameter files (we could have also interfaces for YAML, TOML, and even txt files if we wanted).
Then, we provide our usual tests/unit directory to unit test the parameter file reading. You see that we have a file called testParameterReadingJson.cpp which will be the actual test code we compile, but also a file called parameterReadingJsonMock.hpp, where we mock the JSON file reading interface and we will see below how this is done.
root
├── src
│ ├── parameterReadingBase.hpp
│ └── parameterReadingJson.hpp
├── tests
│ ├── unit
│ │ ├── parameterReadingJsonMock.hpp
│ │ └── testParameterReadingJson.cpp
│ └── mainTest.cpp
├── buildAndRun.ps1
└── buildAndRun.shIn order to build our project, which essentially consists now of only the mainTest.cpp and testParamterReadingJson.cpp files (all other files are header files), we would build this simple project on Windows using the following build script:
# clean up before building
Remove-Item .\build -Force -Recurse
New-Item -Name "build" -ItemType "directory"
# compile source files into object files
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c .\tests\unit\testParameterReadingJson.cpp /Fo".\build\testParameterReadingJson.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c .\tests\mainTest.cpp /Fo".\build\mainTest.obj"
# link object files
cl.exe /nologo /EHsc /std:c++20 .\build\mainTest.obj .\build\testParameterReadingJson.obj /Fe".\build\parameterReadingTest.exe" /link /MACHINE:x64 /LIBPATH:"C:\libraries\lib" gtest.lib gmock.lib
# execute the test
.\build\parameterReadingTest.exeAnd on UNIX we would use the following bash script:
#!/bin/bash
rm -rf build
mkdir -p build
# compile
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. -I ~/libs/include ./tests/unit/testParameterReadingJson.cpp -o ./build/testParameterReadingJson.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. -I ~/libs/include ./tests/mainTest.cpp -o ./build/mainTest.o
# link
g++ -o build/parameterReadingTest ./build/mainTest.o ./build/testParameterReadingJson.o -L ~/libs/lib -lgtest -lgmock
# run
./build/parameterReadingTestBy now, these probably look very familiar, i.e. we first compile both source files and then link them against the gtest library, but you will also notice that we have to link against gmock in both cases (line 10 for the PowerShell script and line 11 for the bash script). The remaining part of the build scripts follow very similar structures compared to what we saw in previous examples.
Parameter reading files
OK, so let’s discuss the parameter file reading then. In this example, I really want to focus on mocking, so we will not write a fully-fledged parameter file reading implementation. Instead, we concentrate on the essentials and only provide an interface for file reading, but, for example, we don’t implement any error checking, in case there is a typo in the input file, for example.
Parameter file reading base class
The interface for the file reading class is given below. It consists of a default constructor and destructor, as well as a read() function, which is a pure virtual function which needs to be implemented by any class deriving from this interface (including a mocked class). We also use a template parameter on line 5, which we first store on line 8 and we use this on line 11 as the return type of the read() function. Since we potentially want to support different file types like JSON, YAML, or TOML, we need to allow to return different types from the read() function, hence the template parameter.
#pragma once
#include <filesystem>
template <typename StorageType>
class ParameterReaderBase {
public:
using FileType = StorageType;
ParameterReaderBase() = default;
virtual ~ParameterReaderBase() = default;
virtual FileType read(std::filesystem::path file) = 0;
};JSON-based parameter file reading
For the JSON-based parameter reading class implementation, we first have to include the library for the JSON library which we do on line 6. We also include the parameter reading base definition on line 8 and a few C++ standard headers on lines 3-4 that we will need here.
On line 10, we define our JSON-based parameter reading class, which inherits from the base class. We can see that we specify here the template parameter to be of type nlohmann::json, i.e. this will be a JSON file. We then provide an implementation for the read() function, and you also see that we define everything in header files, rather than separating code into header and source files. When working with templates, it is easier to stick everything into header files, but we will look at that issue later when dealing with more templates and how to get around this issue.
The read() function implementation is rather straightforward (5 lines in total), i.e. we first create a JSON variable on line 13, and then prepare to read a JSON file on line 14. We then stream all of the content from the input file into the JSON variable and close the file afterwards. That’s all, we return the JSON file on line 17 and can now proceed to use it in our code. This is likely all the JSON code you ever need to interact with the nlohmann::json library, see, it wasn’t that difficult.
#pragma once
#include <filesystem>
#include <fstream>
#include "nlohmann/json.hpp"
#include "src/parameterReaderBase.hpp"
class ParameterReaderJson : public ParameterReaderBase<nlohmann::json> {
public:
ParameterReaderBase::FileType read(std::filesystem::path file) override {
nlohmann::json jsonFile;
std::ifstream inputFile(file);
inputFile >> jsonFile;
inputFile.close();
return jsonFile;
}
};Testing parameter file reading with gmock
OK, so we have seen how we can create a simple parameter reading interface using the JSON file format. But let’s assume that we have written a test for this file reading and realised that we spent a lot of time in our unit test on file input and output. Let’s also assume that our application is writing a lot of these JSON files and unit testing now becomes a very painful task as executing our tests takes tens of seconds and thus we may feel discouraged running our tests frequently.
To avoid the costly file reading, we decided to mock this using gmock, so let’s see how mocking is here to save the day. I would recommend looking through the next example first to get an idea, and then afterwards to to read (or at least browse) through the mocking documentation. It is not a long read and it will give you an idea of what else you can do with mocking. What we look at below will cover most cases, but there are a few more things you can do with mocking, so it is worthwhile to look into the documentation.
Mocking the parameter file reading interface
Our mocking implementation starts similar to our JSON parameter reading class definition, with the appropriate includes at the top. We also include the gmock/gmock.h header file here to have access to Google’s mocking framework. We derive from the interface again on line 11 and again specify that the template parameter here should be nlohmann::json. Since we are storing that template parameter in the base class as we saw previously, we can later use that template parameter using the ParameterReaderBase::FileType syntax.
The mocking happens on line 13 and essentially just tells the framework which function needs to be mocked. We don’t specify here what happens if we actually call this method, that is something we have to set up during the test. When we declare the class, we just have to provide information on which function of the class should be mocked.
Since the read() function as declared as a pure virtual function, we need to provide an implementation for it (or, in this case, a mock), and we see this on line 13. The syntax is as follows: each function can be decomposed into returnType functionName(functionArgument1, functionArgument2, ...) trailingArguments. What you do now, is take this function signature, put commas between each of these arguments (keeping the parenthesis for the function arguments, and also introducing parenthesis for the trailing arguments), and put that into the MOCK_METHOD().
This would result in MOCK_METHOD(returnType, functionName, (functionArgument1, functionArgument2, ...), (trailingArguments)); And that’s it. We see this executed on line 13. If we had more arguments as the trailing arguments, for example, const final override, then we would specify this as (const, final, override). The reason we have to chop this up using commas and parenthesis is that gmock uses C++ macro syntax in the background to extract information here, which needs to be separated by commas.
#pragma once
#include <filesystem>
#include <fstream>
#include "nlohmann/json.hpp"
#include "gmock/gmock.h"
#include "src/parameterReaderBase.hpp"
class ParameterReaderJsonMock : public ParameterReaderBase<nlohmann::json> {
public:
MOCK_METHOD(ParameterReaderBase::FileType, read, (std::filesystem::path), (override));
};Once we have provided the mocked interface, we can now create a mocked object in tests and then specify what the expected behaviour is when we call the read() function. Instead of reading a file from disk now, we can simply specify what we would expect this operation to return and use that instead. Let’s look at the test code then to see this in action.
Writing a unit test using the mocked parameter file reading implementation
We will only look at a single test here. We could also test the JSON-based parameter reading class, but I wanted to look at the mocking specifically and there would be nothing interesting (or new) about testing the JSON parameter reading.
Below, then, is the test code within the tests/unit/testParameterReadingJson.cpp file. It imports the mocked interface on line 6 and then uses that in the test starting from line 8.
We structure our test into the standard Arrange, Act, Assert section, although we differentiate between variable and mock set-up in the arrange section. We first create a parameter reading object on line 10, which derives from the mocked class implementation we provided above. We also set up a JSON object on lines 11-14, that we would expect our parameter reading to return and we use this on line 17, where we set up our mock.
Line 17 is where mocking starts, this is where we specify what we expect a call to the read() function would produce. The syntax is as follows: first, we specify which variable we want to set up the mocking behaviour for (in this case, the parameterReadingMock variable). Secondly, we specify the function and all of its arguments to which we expect a call will be made (we can see this call on line 20 in the act section). This EXPECT_CALL() function is then followed by what is expected to happen.
In our case, we expect that a call to the read() function will return only once (this is most often the case and indicated by the WillOnce() function name) and, we specifically say what it will return, in this case, the expectedJsonFile we set up on lines 11-14. So, if we make a call to read() now on line 20, gmock will now check what this function should do or return and based on what we have provided on line 17, it will do exactly that.
On line 23, we check that the JSON parameter we receive within the inputFile is now equivalent to the expectedJsonFile. I also provide some additional statements on lines 24-25, these are not technically required but show you how you would use a JSON file now. For example, the get the number of iterations specified within the JSON file, we would use the inputFile["iterations"] syntax and we can see that JSOn files are stored as std::maps in C++. So if you feel comfortable with std::maps, then working with JSON files will be straightforward.
#include <filesystem>
#include "gtest/gtest.h"
#include "gmock/gmock.h"
#include "tests/unit/parameterReadingJsonMock.hpp"
TEST(ParameterReadingJson, ReadJsonFileMock) {
// Arrange inputs
ParameterReaderJsonMock parameterReadingMock;
nlohmann::json expectedJsonFile = {
{"iterations", 100},
{"turbulence_model", "Spalart-Allmaras"}
};
// Arrange mocks
EXPECT_CALL(parameterReadingMock, read(std::filesystem::path("input.json"))).WillOnce(::testing::Return(expectedJsonFile));
// Act
auto inputFile = parameterReadingMock.read(std::filesystem::path("input.json"));
// Assert
ASSERT_EQ(inputFile, expectedJsonFile);
ASSERT_EQ(inputFile["iterations"], 100);
ASSERT_EQ(inputFile["turbulence_model"], "Spalart-Allmaras");
}One thing which may look a bit odd is the ::testing::Return statement on line 17. All of gtests’s and gmock’s code live inside the testing namespace. The leading :: in front of ::testing means that this is referring to the global namespace. You can remove the leading ::, i.e. testing::Return but the convention with gtest is that we refer to the global namespace, just in case you have defined a namespace somewhere else called testing as well. This avoids namespace clashes.
Suffice to say, the ::testing::Return or testing::Return statement is used to indicate what should be returned by the function. I just mention it here because, in most of the documentation, you will see simply a Return written instead of ::testing::Return. To get around this, the documentation uses a using ::testing::Return statement at the top of the file, but I like my code to be verbose and readable, so I usually include the namespaces where I access a variable or function from that namespace.
Test output
The testing output is perhaps not very interesting, but for completeness, it is given below. We just run a single test and we see that the test is passing, so our mocking has worked correctly. We never read any file from disk but still have a test passing, and since all variables were defined in main memory, rather than anything needing to be read from the hard disk (i.e. a JSON file), we see that the actual unit test took less than a millisecond.
You can try to implement the JSON-based parameter reading test and see how it compares, how much time do we save using mocking? In this case, probably not a lot, but in the case of the mesh reading library, we saw that there was a quite large overhead for reading files from disk, so it may be a good idea to consider mocking even for these simple types of grids we were dealing with.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from ParameterReadingJson
[ RUN ] ParameterReadingJson.ReadJsonFile
[ OK ] ParameterReadingJson.ReadJsonFile (0 ms)
[----------] 1 test from ParameterReadingJson (0 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (2 ms total)
[ PASSED ] 1 test.Summary
So then, this completes our brief look at mocking as well. As I have mentioned in the opening paragraph, mocking is not of high importance when we write CFD applications, it can be useful but apart from some very specific cases. We can get away without having to resort to mocking, unless we want to speed up some unit tests, in which case, we would use mocking for convenience but not out of necessity.
Mocking plays a much bigger role in web development (mocking web requests and testing websites, yes, websites are also unit tested, see the popular Django python framework, for example) and mobile development, and less so for classical console applications. But we have seen how to use it now and can recognise use cases where mocking may be beneficial.
At this point, you have seen pretty much all there is to testing with gtest and gmock and in the last remaining article of this series, we will look at test coverage, an area that allows us to check how much of our code was actually tested by our unit, integration, and system tests. We’ll leave that for the next article.
Tom-Robin Teschner is a senior lecturer in computational fluid dynamics and course director for the MSc in computational fluid dynamics and the MSc in aerospace computational engineering at Cranfield University.