How To Test A Linear Algebra Solver Library Using gtest

In this article, we look at how we can test the linear algebra solver library we developed previously. In particular, we will look at how we can write unit, integration, and system tests that ensure our code is working using both white-box and black-box testing. This will provide our code with additional resistance against regressions (bugs) and generally forms the basis of any software testing approach.

By the end of this article, you should have solidified your knowledge of how to use gtest as a testing framework to provide unit, integration, and system test in your project, and how you compile your tests separately from your library code and run all tests during development.

Download Resources

All developed code and resources in this article are available for download. If you are encountering issues running any of the scripts, please refer to the instructions for running scripts downloaded from this website.

• Download: Software-Testing-Part-4.zip

In this Series

• Part 1: How to get started with software testing for CFD codes
• Part 2: The test-driven development for writing bug-free CFD code
• Part 3: How to get started with gtest in C++ for CFD development
• Part 4: How To Test A CGNS-based Mesh Reading Library Using gtest
• Part 5: How To Test A Linear Algebra Solver Library Using gtest
• Part 6: How to use mocking in CFD test code using gtest and gmock
• Part 7: What is test coverage and how to use LCOV/GCOV for testing

In this series

Recap of our linear algebra solver library and outlook

Some time ago, we developed a library to help us solve the linear system of equations that can be classically expressed by Ax=b. We did perform some tests on it, which, if you remember, was to solve the 1D, steady-state heat diffusion equation. We picked this example because there is an analytic solution we can construct if we set the boundary conditions appropriately. Based on our previous discussions, we would classify this test as a system test, but we did not provide any unit or integration tests.

In contrast, in the last article, we wrote tests for our structured and unstructured mesh reading library, and we said that these were classified as unit tests, as we did test each component (function) of our library individually and in isolation. So the goal of this article is to bridge the gap and provide unit, integration, and system tests so that we can get a feeling for how to write these and what they would look like in a real project.

For this, we have to supplement the system test we have already created with some unit and integration tests, which is what we will focus on in this article. We will also dick a little deeper into some additional pain points that we may come across in cross-platform development, especially on Windows, and look at possible issues and solutions to compile dynamic-linked libraries (DLLs) and how to use them in our test code.

If you have followed along in this series and read through the previous articles, you will see that the project structure, writing the build scripts and test code will follow a very predictive manner. My goal is not to bore you to death with code that looks very similar to something you have already looked at previously, but rather to provide you with an overview of how to integrate unit, integration, and system tests into a single test framework.

As a result, I will skip over some sections a bit quicker than in previous discussions and if you are lost with some of these code sections, I recommend looking at articles published previously in this series, which all build up to today’s discussion. Instead, I want to look at some of the challenges that you may face when building and testing cross-compiled library and test code and we have a more in-depth discussion of that towards the end of this article. With that out of the way, let’s get started then!

Adding Tests to the linear algebra solver library

So then, let’s look at some test code. We are working with the final version of the linear algebra solver library that we developed towards the end of the series on how to write a linear algebra solver library. However, we are going to make some modifications before we get started with writing tests.

The final version of the library contained build scripts for all sorts of build systems. We looked at CMake, Make, Autotools (oh no), MSBuild, ninja, and Meson. We won’t be rewriting the build scripts for all of them but instead, stick with our custom-made build scripts for now (i.e. PowerShell and Bash scripts).

While I wouldn’t recommend writing custom-made build scripts in general for managing larger (or even smaller) projects, they help us understand exactly what compiler commands to use. Once we start to look at build systems, this knowledge will help us to debug build files to pinpoint the location of the compilation errors and this will allow us to debug build scripts quicker. So, we use them here purely for educational reasons but eventually, they will make way for proper build systems.

The second change I want to make is to retire the batch script in favour of a PowerShell script. The original build script for Windows used a batch script, i.e. a file ending in *.bat, and we will replace that with a PowerShell script, i.e. a file ending in *.ps1. This is in line with build scripts we have previously developed for the mesh reader library and just in general it is a better idea to work with PowerShell scripts than batch files, as these are rather outdated on Windows by now. The world has moved on, and so should we.

And, finally, there is no more version of the header-only library. We discussed how we could turn our library code into a header-only library if we want to (and if you wanted to use that in our tests instead, then you would simply include the header include file in your test code). But the reason we are dropping it here is just for simplicity; we remove a lot of code that we do not need to discuss. I trust that you can replace the header include file for the static/dynamic library for the header-only version in your test code if you want to use that instead.

Project Structure

The code structure of the project we will be developing in this article is shown below. Lines 1-12 are essentially the same as our original linear algebra-solving library. Similarly, lines 23-28 are the same and show the build scripts we will be modifying for this project, with the exception that we have changed from batch to PowerShell scripts as discussed above and also changed the names of the build scripts slightly to be in line with how we have named things previously.

The new bits are added on lines 13-22, and show the build structure that we will use. We see that we will add unit, integration, and system tests separately and they all live in their own directory. We are going to write 3 separate unit tests for the conjugate gradient, vector, and matrix class, while there is just one integration test for the matrix-vector multiplication and one system test for the 1D heat diffusion equation.

We also provide the same mainTest.cpp file as shown on line 22 which is just the default main() function required by gtest to locate all tests.

root
├── linearAlgebraLib
│   ├── src
│   │   ├── conjugateGradient.hpp
│   │   ├── conjugateGradient.cpp
│   │   ├── linearAlgebraSolverBase.hpp
│   │   ├── linearAlgebraSolverBase.cpp
│   │   ├── sparseMatrixCSR.hpp
│   │   ├── sparseMatrixCSR.cpp
│   │   ├── vector.hpp
│   │   └── vector.cpp
│   └── linearAlgebraLib.hpp
├── tests
│   ├── unit
│   │   ├── testConjugateGradient.cpp
│   │   ├── testMatrix.cpp
│   │   └── testVector.cpp
│   ├── integration
│   │   └── testMatrixVectorMultiplication.cpp
│   ├── system
│   │   └── testHeatEquation.cpp
│   └── mainTest.cpp
├── buildAndRun.ps1
├── buildAndRun.sh
├── buildStaticAndRun.ps1
├── buildStaticAndRun.sh
├── buildDynamicAndRun.ps1
└── buildDynamicAndRun.sh

Build scripts

As discussed above, we will have a look at all of the build scripts first, but these should be fairly straightforward to understand at this point. If you require some additional discussions, have a look at my discussion on build scripts for the mesh reading library, and article referenced therein, which will provide you with some additional insights.

Windows

We will first transform all build scripts into PowerShell files and then create versions to build the test executable in one go, or separately where the libraries are either compiled as static or dynamic libraries first and then linked into the test executable.

Building everything into a single executable

Building all code in one go and then linking all of it into the test executable follows the same pattern as for the previous article. We compile all source code files first (lines 6-15) and then link them into the test executable on line 18, where we have to ling against the gtest library as well to execute the test code.

We can see that all test code is compiled on lines 10-15, which is in addition to the code compiled on lines 6-9, which is essentially the code for the linear algebra solver library, with the exception that we do not link these object files into static or dynamic libraries first but rather inject them straight away into the executable on line 18. This should feel familiar if you read the previous articles.

# clean up before building
Remove-Item .\build -Force -Recurse
New-Item -Name "build" -ItemType "directory"

# compile source files into object files
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c /DCOMPILELIB .\linearAlgebraLib\src\sparseMatrixCSR.cpp /Fo".\build\sparseMatrixCSR.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c /DCOMPILELIB .\linearAlgebraLib\src\vector.cpp /Fo".\build\vector.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c /DCOMPILELIB .\linearAlgebraLib\src\linearAlgebraSolverBase.cpp /Fo".\build\linearAlgebraSolverBase.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c /DCOMPILELIB .\linearAlgebraLib\src\conjugateGradient.cpp /Fo".\build\conjugateGradient.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c /DCOMPILELIB .\tests\unit\testVector.cpp /Fo".\build\testVector.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c /DCOMPILELIB .\tests\unit\testMatrix.cpp /Fo".\build\testMatrix.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c /DCOMPILELIB .\tests\unit\testConjugateGradient.cpp /Fo".\build\testConjugateGradient.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c /DCOMPILELIB .\tests\integration\testMatrixVectorMultiplication.cpp /Fo".\build\testMatrixVectorMultiplication.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c /DCOMPILELIB .\tests\system\testHeatEquation.cpp /Fo".\build\testHeatEquation.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c /DCOMPILELIB .\tests\mainTest.cpp /Fo".\build\mainTest.obj"

# link object files
cl.exe /nologo /EHsc /std:c++20 .\build\mainTest.obj .\build\testHeatEquation.obj .\build\conjugateGradient.obj .\build\linearAlgebraSolverBase.obj .\build\vector.obj .\build\sparseMatrixCSR.obj .\build\testVector.obj .\build\testMatrix.obj .\build\testConjugateGradient.obj .\build\testMatrixVectorMultiplication.obj /Fe".\build\linearAlgebraTest.exe" /link /MACHINE:x64 /LIBPATH:"C:\libraries\lib" gtest.lib

# execute the test
.\build\linearAlgebraTest.exe

Building a separate static library and test executable

Moving on then to the static library version, i.e. where we compile the library code into a static library first, we see that the code we compiled in lines 6-9 before is now compiled the same way but then linked into a static library on line 12. It resides with the build directory and after we compile the test code on lines 14-19, we link all compiled tests against gtest and our linear algebra library on line 22.

# clean up before building
Remove-Item .\build -Force -Recurse
New-Item -Name "build" -ItemType "directory"

# compile source files into object files
cl.exe /nologo /EHsc /std:c++20 /I. /c /DCOMPILELIB .\linearAlgebraLib\src\sparseMatrixCSR.cpp /Fo".\build\sparseMatrixCSR.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /c /DCOMPILELIB .\linearAlgebraLib\src\vector.cpp /Fo".\build\vector.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /c /DCOMPILELIB .\linearAlgebraLib\src\linearAlgebraSolverBase.cpp /Fo".\build\linearAlgebraSolverBase.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /c /DCOMPILELIB .\linearAlgebraLib\src\conjugateGradient.cpp /Fo".\build\conjugateGradient.obj"

# link static library
lib.exe /nologo /OUT:build\linearAlgebra.lib .\build\sparseMatrixCSR.obj .\build\vector.obj .\build\linearAlgebraSolverBase.obj .\build\conjugateGradient.obj

cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c /DCOMPILELIB .\tests\unit\testVector.cpp /Fo".\build\testVector.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c /DCOMPILELIB .\tests\unit\testMatrix.cpp /Fo".\build\testMatrix.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c /DCOMPILELIB .\tests\unit\testConjugateGradient.cpp /Fo".\build\testConjugateGradient.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c /DCOMPILELIB .\tests\integration\testMatrixVectorMultiplication.cpp /Fo".\build\testMatrixVectorMultiplication.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c /DCOMPILELIB .\tests\system\testHeatEquation.cpp /Fo".\build\testHeatEquation.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c /DCOMPILELIB .\tests\mainTest.cpp /Fo".\build\mainTest.obj"

# link object files
cl.exe /nologo /EHsc /std:c++20 .\build\mainTest.obj .\build\testVector.obj .\build\testMatrix.obj .\build\testConjugateGradient.obj .\build\testMatrixVectorMultiplication.obj .\build\testHeatEquation.obj /Fe".\build\linearAlgebraTest.exe" /link /MACHINE:x64 /LIBPATH:"C:\libraries\lib" /LIBPATH:.\build linearAlgebra.lib gtest.lib

# execute the test
.\build\linearAlgebraTest.exe

Building a separate dynamic library and test executable

The dynamic library version of our build script is essentially the same as the static library version, with the exception of course that we are linking our library code into a dynamic library on line 12, rather than a static library. We do that by using link.exe instead of lib.exe and by providing the /DLL linker flag that will generate our build/linearAlgebra.dll and build/linearAlgebra.lib library and wrapper for the library, as discussed in previous articles.

# clean up before building
Remove-Item .\build -Force -Recurse
New-Item -Name "build" -ItemType "directory"

# compile source files into object files
cl.exe /nologo /EHsc /std:c++20 /I. /c /DCOMPILEDLL .\linearAlgebraLib\src\sparseMatrixCSR.cpp /Fo".\build\sparseMatrixCSR.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /c /DCOMPILEDLL .\linearAlgebraLib\src\vector.cpp /Fo".\build\vector.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /c /DCOMPILEDLL .\linearAlgebraLib\src\linearAlgebraSolverBase.cpp /Fo".\build\linearAlgebraSolverBase.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /c /DCOMPILEDLL .\linearAlgebraLib\src\conjugateGradient.cpp /Fo".\build\conjugateGradient.obj"

# link object files into dynamic library
link.exe /nologo /DLL /OUT:build\linearAlgebra.dll .\build\sparseMatrixCSR.obj .\build\vector.obj .\build\linearAlgebraSolverBase.obj .\build\conjugateGradient.obj

# compile test code
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c /DCOMPILEDLL .\tests\unit\testVector.cpp /Fo".\build\testVector.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c /DCOMPILEDLL .\tests\unit\testMatrix.cpp /Fo".\build\testMatrix.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c /DCOMPILEDLL .\tests\unit\testConjugateGradient.cpp /Fo".\build\testConjugateGradient.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c /DCOMPILEDLL .\tests\integration\testMatrixVectorMultiplication.cpp /Fo".\build\testMatrixVectorMultiplication.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c /DCOMPILEDLL .\tests\system\testHeatEquation.cpp /Fo".\build\testHeatEquation.obj"
cl.exe /nologo /EHsc /std:c++20 /I. /I "C:\libraries\include" /c /DCOMPILEDLL .\tests\mainTest.cpp /Fo".\build\mainTest.obj"

# link object files
cl.exe /nologo /EHsc /std:c++20 .\build\mainTest.obj .\build\testVector.obj .\build\testMatrix.obj .\build\testConjugateGradient.obj .\build\testMatrixVectorMultiplication.obj .\build\testHeatEquation.obj /Fe".\build\linearAlgebraTest.exe" /link /MACHINE:x64 /LIBPATH:"C:\libraries\lib" /LIBPATH:.\build linearAlgebra.lib gtest.lib

# execute the test
.\build\linearAlgebraTest.exe

Unix

On UNIX, not much changed. We have renamed the build scripts as well compared to the original library version we are basing this work on but otherwise, the build scripts remained the same and we are simply adding the test code compilation to the process. Instead of building the main.cpp file, which housed the 1D heat diffusion equation test before, we are now linking everything against tests/mainTest.cpp, as we did with our Windows build scripts as well.

Building everything into a single executable

Similarly to our Windows build scripts, if we want to compile everything in one go, we compile all code into object files as shown on lines 7-16 and then link them on line 19 into a single executable. We use slightly different compiler flags here compared to Windows, i.e. we use here -g and -O0 to indicate that this is a debug build, as well as the -Wall and -Wextra flags to show all warnings (and then a few extra warnings as well, excellent naming convention …).

We could have brought these in line with the Windows compiler flags, or rather, we could have used the same on Windows, but in my experience turning on warnings on Windows can sometimes be quite frustrating. It will warn you about all sorts of things you don’t have control over (for example, issues in library code you haven’t written) and this can quickly become too much so that you are losing an overview of what warnings are relevant and which aren’t.

Thus, you will see me using sometimes inconsistent build flags when we switch from Windows to UNIX. While this can be confusing, it is just optimised for the platform we are building on. In my opinion, it is simpler to develop on UNIX and then test the code on Windows after all tests are passed, but having said that, I usually develop the other way around now, just because I know there are more issues on Windows and if I can catch all of them, then usually there are no problems compiling the code on UNIX.

You can, of course, embrace cross-compilation completely and just use a compiler that works on all platforms such as the Clang C++ compiler. With that, you have a single build script that works across all platforms (well, pretty much, library creation will still be slightly different but the compiler flags for source file compilations will be the same). We use g++ (UNIX) and cl (Windows) here because I would consider these the native compilers for these platforms.

#!/bin/bash

rm -rf build
mkdir -p build

# compile
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. ./linearAlgebraLib/src/sparseMatrixCSR.cpp -o ./build/sparseMatrixCSR.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. ./linearAlgebraLib/src/vector.cpp -o ./build/vector.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. ./linearAlgebraLib/src/linearAlgebraSolverBase.cpp -o ./build/linearAlgebraSolverBase.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. ./linearAlgebraLib/src/conjugateGradient.cpp -o ./build/conjugateGradient.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. -I ~/libs/include ./tests/unit/testVector.cpp -o ./build/testVector.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. -I ~/libs/include ./tests/unit/testMatrix.cpp -o ./build/testMatrix.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. -I ~/libs/include ./tests/unit/testConjugateGradient.cpp -o ./build/testConjugateGradient.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. -I ~/libs/include ./tests/integration/testMatrixVectorMultiplication.cpp -o ./build/testMatrixVectorMultiplication.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. -I ~/libs/include ./tests/system/testHeatEquation.cpp -o ./build/testHeatEquation.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. -I ~/libs/include ./tests/mainTest.cpp -o ./build/mainTest.o

# link
g++ -o build/linearAlgebraTest ./build/mainTest.o ./build/testHeatEquation.o ./build/conjugateGradient.o ./build/linearAlgebraSolverBase.o ./build/vector.o ./build/sparseMatrixCSR.o ./build/testVector.o ./build/testMatrix.o ./build/testConjugateGradient.o ./build/testMatrixVectorMultiplication.o  -L ~/libs/lib -lgtest

# run
./build/linearAlgebraTest

Building a separate static library and test executable

The static library creation is similar to the way we build them on Windows. First, we compile the library code on lines 7-10, then we link it to a static library on line 13 using ar, followed by test code compilation on lines 16-21 which is used on line 24 to link all test code object files into an executable, while also linking against the static version of our linear algebra library, as well as gtest.

#!/bin/bash

rm -rf build
mkdir -p build

# compile source files into object code
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. linearAlgebraLib/src/sparseMatrixCSR.cpp -o build/sparseMatrixCSR.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. linearAlgebraLib/src/vector.cpp -o build/vector.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. linearAlgebraLib/src/linearAlgebraSolverBase.cpp -o build/linearAlgebraSolver.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. linearAlgebraLib/src/conjugateGradient.cpp -o build/conjugateGradient.o

# link object files into dynamic library
ar rcs build/libLinearAlgebra.a build/sparseMatrixCSR.o build/vector.o build/linearAlgebraSolver.o build/conjugateGradient.o

# compile test code
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. -I ~/libs/include ./tests/unit/testVector.cpp -o ./build/testVector.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. -I ~/libs/include ./tests/unit/testMatrix.cpp -o ./build/testMatrix.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. -I ~/libs/include ./tests/unit/testConjugateGradient.cpp -o ./build/testConjugateGradient.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. -I ~/libs/include ./tests/integration/testMatrixVectorMultiplication.cpp -o ./build/testMatrixVectorMultiplication.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. -I ~/libs/include ./tests/system/testHeatEquation.cpp -o ./build/testHeatEquation.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. -I ~/libs/include ./tests/mainTest.cpp -o ./build/mainTest.o

# compile main function and link library into executable
g++ -o build/linearAlgebraTest ./build/mainTest.o ./build/testHeatEquation.o ./build/testVector.o ./build/testMatrix.o ./build/testConjugateGradient.o ./build/testMatrixVectorMultiplication.o  -Lbuild -L ~/libs/lib -lLinearAlgebra -lgtest

# remove object files
rm build/*.o

# run
echo "Running dynamic library example"
./build/linearAlgebraTest

Building a separate dynamic library and test executable

The dynamic library version is again similar, with the only difference here that we need to compile all library code on lines 7-10 as position-independent code (using the -fPIC compiler flag) which is then used on line 13 to create the dynamic library, indicated by the -shared keyword. Again, we discussed previously that we use g++ here which will simply forward the instruction to the linker, and this saves us from writing some boilerplate instructions that are rather long and repetitive. g++ can do that for us, so let’s make use of that offer.

On line 24, we link all compiled test code into an executable and now link against the dynamic version of the library. We also pass some additional arguments to the linker using -Wl, specifically the runtime path with the rpath variable. Technically speaking, this isn’t required. As long as the dynamic library is in the same folder as the executable that depends on it (which is the case here since both the executable and library are located in the build/ folder), the executable will find the dynamic library.

The reason we are verbose here and provide this additional instruction is more for documentation. We explicitly state where we expect the library to be. Later, if we try to execute the library and it can’t be found at runtime, then we would see from the build script where it ought to be so we can check if it is there. If it isn’t, then it is a quick fix, and we simply copy the library into the folder. But if it is, it gives us immediately some different ideas, i.e. it may be compiled for the wrong architecture (32-bit vs. 64-bit) or for a different platform.

#!/bin/bash

rm -rf build
mkdir -p build

# compile source files into object code
g++ -c -fPIC -g -O0 -Wall -Wextra -I. linearAlgebraLib/src/sparseMatrixCSR.cpp -o build/sparseMatrixCSR.o
g++ -c -fPIC -g -O0 -Wall -Wextra -I. linearAlgebraLib/src/vector.cpp -o build/vector.o
g++ -c -fPIC -g -O0 -Wall -Wextra -I. linearAlgebraLib/src/linearAlgebraSolverBase.cpp -o build/linearAlgebraSolver.o
g++ -c -fPIC -g -O0 -Wall -Wextra -I. linearAlgebraLib/src/conjugateGradient.cpp -o build/conjugateGradient.o

# link object files into dynamic library
g++ -shared -g -O0 -Wall -Wextra -I. -o build/libLinearAlgebra.so build/sparseMatrixCSR.o build/vector.o build/linearAlgebraSolver.o build/conjugateGradient.o

# compile test code
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. -I ~/libs/include ./tests/unit/testVector.cpp -o ./build/testVector.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. -I ~/libs/include ./tests/unit/testMatrix.cpp -o ./build/testMatrix.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. -I ~/libs/include ./tests/unit/testConjugateGradient.cpp -o ./build/testConjugateGradient.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. -I ~/libs/include ./tests/integration/testMatrixVectorMultiplication.cpp -o ./build/testMatrixVectorMultiplication.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. -I ~/libs/include ./tests/system/testHeatEquation.cpp -o ./build/testHeatEquation.o
g++ -c -g -O0 -std=c++20 -Wall -Wextra -I. -I ~/libs/include ./tests/mainTest.cpp -o ./build/mainTest.o

# compile main function and link library into executable
g++ -o build/linearAlgebraTest ./build/mainTest.o ./build/testHeatEquation.o ./build/testVector.o ./build/testMatrix.o ./build/testConjugateGradient.o ./build/testMatrixVectorMultiplication.o  -Lbuild -L ~/libs/lib -Wl,-rpath=build -lLinearAlgebra -lgtest

# remove object files
rm build/*.o

# run
echo "Running dynamic library example"
./build/linearAlgebraTest

Testing the linear algebra solver library

So now that we have all the files and build scripts in place, we can write the test code in the various test subdirectories and execute our test code. As we have seen before, to do so, we need to first compile a simple main() function, that will locate all test code automatically, as long as it is defined using the TEST(TestGroup, TestName) or TEST_F(TestFixture, TestName) syntax. We have looked at that code before and it is given below for completeness. This main function is taken directly from the gtest documentation.

#include "gtest/gtest.h"

int main(int argc, char **argv) {
  testing::InitGoogleTest(&argc, argv);
  return RUN_ALL_TESTS();
}

Unit tests

Let’s start with the unit tests then. The original mesh reading library did not have anything in the way of unit testing the library, but we wrote a system test to ensure that all components were working together in harmony. Thus, we would expect the library to work as expected, but we will add some small unit tests here to ensure that the core logic of the library is working as expected.

Testing the vector class

The first class we will test is the vector class. It primarily houses overloaded operators that provide us with the functionality to add, subtract, or multiply vectors together. For the multiplication, we differentiate between the scalar product, which produces a scalar value, and the scalar multiplication, which produces a vector of the same size as the original vector.

For all tests, we use the same Arrange, Act, Assert pattern that we have seen before. You could argue if we need a test fixture here or not, and even though a few vectors will be defined in the same way, I have decided against using a test fixture to keep each test self-contained. They are not complicated, so a test fixture may just add bloat, rather than simplifying it.

Have a look at the tests and then we will discuss them briefly below.

#include <cmath>

#include "gtest/gtest.h"

#include "linearAlgebraLib/linearAlgebraLib.hpp"

TEST(VectorTest, Size) {
  // Arrange
  linearAlgebraLib::Vector vector(3);

  // Act
  auto vectorSize = vector.size();

  // Assert
  ASSERT_EQ(vectorSize, 3);
}

TEST(VectorTest, Access) {
  // Arrange
  linearAlgebraLib::Vector vector(3);
  vector[0] = 1.0;
  vector[1] = 1.7;
  vector[2] = -3.3;

  // Act

  // Assert
  ASSERT_DOUBLE_EQ(vector[0], 1.0);
  ASSERT_DOUBLE_EQ(vector[1], 1.7);
  ASSERT_DOUBLE_EQ(vector[2], -3.3);
}

TEST(VectorTest, L2Norm) {
  // Arrange
  linearAlgebraLib::Vector vector(3);
  vector[0] = 1.0;
  vector[1] = 1.0;
  vector[2] = 1.0;

  // Act
  auto L2Norm = vector.getL2Norm();

  // Assert
  ASSERT_DOUBLE_EQ(L2Norm, std::sqrt(3.0));
}

TEST(VectorTest, VectorAddition) {
  // Arrange
  linearAlgebraLib::Vector vector(3);
  vector[0] = 1.0;
  vector[1] = 2.3;
  vector[2] = -0.7;

  // Act
  auto result = vector + vector;

  // Assert
  ASSERT_DOUBLE_EQ(result[0], 2.0);
  ASSERT_DOUBLE_EQ(result[1], 4.6);
  ASSERT_DOUBLE_EQ(result[2], -1.4);
}

TEST(VectorTest, VectorSubtraction) {
  // Arrange
  linearAlgebraLib::Vector vector(3);
  vector[0] = 1.0;
  vector[1] = 2.3;
  vector[2] = -0.7;

  // Act
  auto result = vector - vector;

  // Assert
  ASSERT_DOUBLE_EQ(result[0], 0.0);
  ASSERT_DOUBLE_EQ(result[1], 0.0);
  ASSERT_DOUBLE_EQ(result[2], 0.0);
}

TEST(VectorTest, VectorDotProduct) {
  // Arrange
  linearAlgebraLib::Vector vector(3);
  vector[0] = 1.0;
  vector[1] = 2.0;
  vector[2] = -3.0;

  // Act
  auto result = vector.transpose() * vector;

  // Assert
  ASSERT_DOUBLE_EQ(result, 14.0);
}

TEST(VectorTest, VectorMultiplication) {
  // Arrange
  linearAlgebraLib::Vector vector(3);
  vector[0] = 1.0;
  vector[1] = 2.0;
  vector[2] = -3.0;

  // Act
  auto result = 2.0 * vector;

  // Assert
  ASSERT_DOUBLE_EQ(result[0], 2.0);
  ASSERT_DOUBLE_EQ(result[1], 4.0);
  ASSERT_DOUBLE_EQ(result[2], -6.0);
}

Lines 7-16 simply test that once we define a vector, we get the right size back. Not a complicated test, but this just helps us to ensure that the vector is operated as expected, in case we mess up the codebase later. Lines 18-31 test that we can access the entries of the vector using the square bracket operator, i.e. [], and again, this is just to check that this is working as expected, as all subsequent tests will make use of that.

There is no testing for boundary cases, i.e. what happens if we access variables out of bounds? This is currently a limitation of the vector class and we may want to think of adding this in the future, but for simplicity, we assume that out-of-bounds access does not occur (which in itself is a dangerous assumption).

We test that the calculation of the L2 norm is correct on lines 33-45, which, again, will be required later in the system test to ensure that the error has been reduced to a sufficiently small level.

On lines 47-61, we test that the operator+() was implemented correctly and that we can add vectors together correctly, which is similar to the test given on lines 63-77, where we ensure that the operator-() was implemented correctly, i.e. we can subtract vectors from each other.

For vector multiplication, as mentioned above, there are two possible use cases, either the scalar product or scalar multiplication. We test the scalar product (or dot product) on lines 79-91, where we test that taking the scalar product of a vector with itself (which requires a row and column vector, hence the need to call transpose() on the vector during the dot product operation).

The scalar multiplication, i.e. where we multiply a vector with a scalar is given on lines 93-107. This test looks fairly straightforward but will make our compilation crash. This is not clear from this simple test code and we will explore this issue further down below. But let’s just say, debugging this code on Windows was again a major headache and in true Microsoft fashion, the compiler errors were not at all helpful. Debugging this code took a week. and just left a further stain on Microsoft’s reputation and inability to write a decent C++ compiler.

But I am getting ahead of myself, there is an entire section on this later in the article, so we will look at this in more detail. For now, let’s move on and write some more tests for the matrix class.

Testing the matrix class

The matrix class test follows the vector class test very closely. We are not using any test fixtures and simply test the essential parts of the matrix class. It doesn’t expose a lot of functionality so its unit test is rather short. The code is given below and we will look at the different tests underneath the code block.

#include "gtest/gtest.h"

#include "linearAlgebraLib/linearAlgebraLib.hpp"

TEST(MatrixTest, RowsAndColumns) {
  // Arrange
  linearAlgebraLib::SparseMatrixCSR matrix(3, 5);

  // Act

  // Assert
  ASSERT_EQ(matrix.getNumberOfRows(), 3);
  ASSERT_EQ(matrix.getNumberOfColumns(), 5);
}

TEST(MatrixTest, SetAndGetValue) {
  // Arrange
  linearAlgebraLib::SparseMatrixCSR matrix(3, 3);
  matrix.set(2, 1, 8.0);
  matrix.set(0, 1, 2.0);
  matrix.set(1, 0, 4.0);
  matrix.set(2, 2, 9.0);
  matrix.set(1, 1, 5.0);
  matrix.set(0, 0, 1.0);
  matrix.set(2, 0, 7.0);
  matrix.set(0, 2, 3.0);
  matrix.set(1, 2, 6.0);

  // Act

  // Assert
  ASSERT_DOUBLE_EQ(matrix.get(0, 0), 1.0);
  ASSERT_DOUBLE_EQ(matrix.get(0, 1), 2.0);
  ASSERT_DOUBLE_EQ(matrix.get(0, 2), 3.0);
  ASSERT_DOUBLE_EQ(matrix.get(1, 0), 4.0);
  ASSERT_DOUBLE_EQ(matrix.get(1, 1), 5.0);
  ASSERT_DOUBLE_EQ(matrix.get(1, 2), 6.0);
  ASSERT_DOUBLE_EQ(matrix.get(2, 0), 7.0);
  ASSERT_DOUBLE_EQ(matrix.get(2, 1), 8.0);
  ASSERT_DOUBLE_EQ(matrix.get(2, 2), 9.0);
}

TEST(MatrixTest, ScalarMultiplication) {
  // Arrange
  linearAlgebraLib::SparseMatrixCSR matrix(3, 3);
  matrix.set(0, 0, 1.0);
  matrix.set(1, 1, 2.0);
  matrix.set(2, 2, -3.0);

  // Act
  auto result = 2.0 * matrix;

  // Assert
  ASSERT_DOUBLE_EQ(result.get(0, 0), 2.0);
  ASSERT_DOUBLE_EQ(result.get(1, 1), 4.0);
  ASSERT_DOUBLE_EQ(result.get(2, 2), -6.0);
}

The first test on lines 5-14 simply tests that we can generate a matrix of a certain size, not necessarily square in shape (although these are the only matrices we will be interested in), and then it asserts that the size is returned correctly.

Lines 16-41 test the ability to set a certain value and then retrieve it correctly. If the matrix class would use a 2D std::vector to store its data entries, then this test would be somewhat pointless, but since we have implemented the compressed sparse row (CSR) data structure for our matrix, setting and getting values from the matrix is somewhat more complicated.

In particular, you may have noticed that we insert values in a random order on lines 19-27. This is done as the insertion needs to be able to handle random insertion, which is somewhat more complicated to implement than a sequential insertion (which, in turn, is a rather restrictive limitation). When I originally developed the CSR-based matrix class, I tested it with sequential insertion and that was working fine until I randomly inserted values in another test only to wonder why the code was not working anymore.

There is a saying that each regression (bug) in your code should result in a test that should be written to capture the same regression again in the future. If you introduce code somewhere else that triggers the same unwanted behaviour, you will have a test now to protect you against the same error. Thus, I opted to insert the values randomly to ensure that this results in the same ordered matrix that we test in the assert section on lines 32-40.

Lines 43-57, then, test a scalar multiplication with a matrix, which again looks straightforward but introduces the same compilation errors as alluded to above in the vector class test. We will leave this discussion to the end of the article.

Testing the Conjugate Gradient class

Whenever you want to test a linear algebra solver, it is a good idea to come up with an example that uses only whole numbers. This makes the assertions much easier and you don’t have to deal with round-off errors. We will use the following example here:

Ax=b\rightarrow \begin{bmatrix}1 & 2 & 0 \\2 & 1 & 2 \\ 0 & 2 & 1\end{bmatrix}\begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}=\begin{bmatrix} 1 \\ 4 \\ 1 \end{bmatrix}

Notice that the coefficient matrix on the left-hand side is symmetrical; this is one of the assumptions of the Conjugate Gradient method. The other assumption is that the matrix is positive definite, though that is usually not a limiting factor. Symmetric coefficient matrices arise when we use a numerical scheme with symmetrical stencils, such as the central scheme.

Non-linear terms in the Navier-Stokes equations are best approximated using some form of upwind-based numerical scheme (Upwind, MUSCL, ENO, WENO, Flux-Vector splitting, etc.). However, upwind-based schemes result in non-symmetric matrices and we would need to extend our linear algebra solver library with some non-symmetrical matrix solvers, such as the Bi Conjugate Gradient Stabilised (BiCGStab) algorithms.

While central schemes produce a symmetric matrix, these schemes produce unconditionally unstable results in the approximation process and are best avoided. However, if we wanted to make use of a central scheme and thus still be able to use the Conjugate Gradient algorithm as we have implemented it, then we can turn central schemes into stable and converging schemes by considering a stabilised version of the central scheme.

Be that as it may, the test for our Conjugate Gradient algorithm is shown below:

#include "gtest/gtest.h"

#include "linearAlgebraLib/linearAlgebraLib.hpp"

TEST(ConjugateGradientTest, ConjugateGradient) {
  // Arrange
  linearAlgebraLib::SparseMatrixCSR matrix(3, 3);
  matrix.set(0, 0, 1.0);
  matrix.set(0, 1, 2.0);
  matrix.set(0, 2, 0.0);
  matrix.set(1, 0, 2.0);
  matrix.set(1, 1, 1.0);
  matrix.set(1, 2, 2.0);
  matrix.set(2, 0, 0.0);
  matrix.set(2, 1, 2.0);
  matrix.set(2, 2, 1.0);

  linearAlgebraLib::Vector rightHandSide(3);
  rightHandSide[0] = 1.0;
  rightHandSide[1] = 4.0;
  rightHandSide[2] = 1.0;

  linearAlgebraLib::ConjugateGradient solver(3);  
  solver.setRightHandSide(rightHandSide);
  solver.setCoefficientMatrix(matrix);

  // Act
  linearAlgebraLib::Vector solution = solver.solve(100, 1e-12);

  // Assert
  ASSERT_NEAR(solution[0], 1.0, 1e-8);
  ASSERT_NEAR(solution[1], 0.0, 1e-8);
  ASSERT_NEAR(solution[2], 1.0, 1e-8);
}

We set the coefficient matrix as per the above equation on lines 7-16, as well as the right-hand side vector on lines 18-21. We construct and solve for the vector x on lines 23-28 and then assert that we have obtained the correct values for our vector x.

Integration tests

We haven’t looked at integration tests before, so let’s have a quick recap of what an integration test is. We said before that a unit test is testing a function of a class, typically in isolation. This is the definition of a unit. As soon as we test more than one function, especially from two different classes, we have an integration test.

Sometimes we use more than one function from the same class, though we should exercise common sense here. If we test a function that makes use of a few getter and setter functions, each only one line in length, then it is probably more prudent to assume that these are simple helper functions rather than integrated components that are tested together.

If, however, we make use of a function from another class that implements some logic, then we are testing how different units work together. This is the point where we go from a unit to an integration test, and there is one integration test that we can write for the linear algebra solver library, which is the matrix-vector multiplication case.

Testing Matrix-Vector multiplication

In this test, we need both a matrix and a vector object. We make use of the same matrix and vector that we used before in the Conjugate Gradient test. If we now multiply A and x together, we should get the right-hand side. Before, we provided the right-hand side and wanted to obtain x. So, we should be able to reuse the same values and get the same results. This is shown in the test code below:

#include "gtest/gtest.h"

#include "linearAlgebraLib/linearAlgebraLib.hpp"

TEST(MatrixVectorMultiplicationTest, MatrixVectorMultiplication) {
  // Arrange
  linearAlgebraLib::SparseMatrixCSR matrix(3, 3);
  matrix.set(0, 0, 1.0);
  matrix.set(0, 1, 2.0);
  matrix.set(0, 2, 0.0);
  matrix.set(1, 0, 2.0);
  matrix.set(1, 1, 1.0);
  matrix.set(1, 2, 2.0);
  matrix.set(2, 0, 0.0);
  matrix.set(2, 1, 2.0);
  matrix.set(2, 2, 1.0);

  linearAlgebraLib::Vector vector(3);
  vector[0] = 1.0;
  vector[1] = 0.0;
  vector[2] = 1.0;

  // Act
  auto result = matrix * vector;

  // Assert
  ASSERT_NEAR(result[0], 1.0, 1e-8);
  ASSERT_NEAR(result[1], 4.0, 1e-8);
  ASSERT_NEAR(result[2], 1.0, 1e-8);
}

System tests

While unit tests are an excellent way to check that every component works correctly through white-box testing, system tests provide us with the best possible protection against regressions through black-box testing. We looked at white-box and black-box testing at the beginning of the series, give it a read if you need a refresher.

System tests exercise all code together and check that each component still works correctly when working together with other code. They may require longer to run, but we should always have a few system tests in place that are quick and easy to run, just to ensure that all code behaves nicely when executed together.

Testing the 1D heat equation solver

The system test that we use in this case is the same test we provided in the original write-up for the linear algebra solver library. You may wish to read up on the original test to remind yourself what the code is doing.

The only difference here is that we are now using gtest’s TEST() statement on line 9 and we have replaced the assertion on line 67 with an ASSERT_NEAR() statement. Otherwise, this test code, which now has become our system test, remains exactly the same as it was before.

#include "gtest/gtest.h"

#include "linearAlgebraLib/linearAlgebraLib.hpp"

// solve the heat equation implicitly of the form dT/dt = gamma * d^2 T/ dx^2 over a domain L using the conjugate
// gradient methdod
// initial condition: 0 everywhere
// boundary condition: T(0) = 0, T(L) = 1
TEST(SystemTest, HeatEquation1DImplicit) {
  // input variables
  const double gamma = 1.0;
  const unsigned numberOfCells = 100;
  const double domainLength = 1.0;
  const double boundaryValueLeft = 0.0;
  const double boundaryValueRight = 1.0;
  const double dx = domainLength / (numberOfCells);

  // vectors and matrices
  linearAlgebraLib::Vector coordinateX(numberOfCells);
  linearAlgebraLib::Vector temperature(numberOfCells);
  linearAlgebraLib::Vector boundaryConditions(numberOfCells);

  linearAlgebraLib::SparseMatrixCSR coefficientMatrix(numberOfCells, numberOfCells);

  // initialise arrays and set-up 1D mesh
  for (unsigned i = 0; i < numberOfCells; ++i) {
    coordinateX[i] = i * dx + dx / 2.0;
    temperature[i] = 0.0;
    boundaryConditions[i] = 0.0;
  }

  // calculate individual matrix coefficients
  const double aE = gamma / dx;
  const double aW = gamma / dx;
  const double aP = -1.0 * (aE + aW);

  // set individual matrix coefficients
  for (unsigned i = 1; i < numberOfCells - 1; ++i) {
    coefficientMatrix.set(i, i, aP);
    coefficientMatrix.set(i, i + 1, aE);
    coefficientMatrix.set(i, i - 1, aW);
  }

  coefficientMatrix.set(0, 0, -1.0 * (aE + 2.0 * aW));
  coefficientMatrix.set(0, 1, aE);
  coefficientMatrix.set(numberOfCells - 1, numberOfCells - 2, aW);
  coefficientMatrix.set(numberOfCells - 1, numberOfCells - 1, -1.0 * (2.0 * aE + aW));

  // set boundary conditions
  boundaryConditions[0] = -2.0 * aW * boundaryValueLeft;
  boundaryConditions[numberOfCells - 1] = -2.0 * aE * boundaryValueRight;

  // solve the linear system using the conjugate gradient method
  linearAlgebraLib::ConjugateGradient CGSolver(numberOfCells);
  CGSolver.setCoefficientMatrix(coefficientMatrix);
  CGSolver.setRightHandSide(boundaryConditions);
  temperature = CGSolver.solve(100, 1e-10);

  // the obtain temperature profile is a linear one of the form T(x) = x. Thus, we can compare it directly against
  // the coordinate vector (which in this case acts as an analytic solution)
  linearAlgebraLib::Vector difference(numberOfCells);
  for (unsigned i = 0; i < numberOfCells; ++i) {
    difference[i] += std::fabs(temperature[i] - coordinateX[i]);
  }

  // ensure that temperature has converged to at least single precision
  ASSERT_NEAR(difference.getL2Norm(), 0.0, 1e-8);
}

Test execution

There are two possible scenarios at this point. Either, you downloaded the code above and you are going through the code to check what I have been doing. You have followed along with this discussion, tried to compile the code and everything was working. What you should see is the output that is shown below, which by now should look somewhat familiar.

[==========] Running 13 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 1 test from SystemTest
[ RUN      ] SystemTest.HeatEquation1DImplicit
[       OK ] SystemTest.HeatEquation1DImplicit (1 ms)
[----------] 1 test from SystemTest (1 ms total)

[----------] 7 tests from VectorTest
[ RUN      ] VectorTest.Size
[       OK ] VectorTest.Size (0 ms)
[ RUN      ] VectorTest.Access
[       OK ] VectorTest.Access (0 ms)
[ RUN      ] VectorTest.L2Norm
[       OK ] VectorTest.L2Norm (0 ms)
[ RUN      ] VectorTest.VectorAddition
[       OK ] VectorTest.VectorAddition (0 ms)
[ RUN      ] VectorTest.VectorSubtraction
[       OK ] VectorTest.VectorSubtraction (0 ms)
[ RUN      ] VectorTest.VectorDotProduct
[       OK ] VectorTest.VectorDotProduct (0 ms)
[ RUN      ] VectorTest.VectorMultiplication
[       OK ] VectorTest.VectorMultiplication (0 ms)
[----------] 7 tests from VectorTest (0 ms total)

[----------] 3 tests from MatrixTest
[ RUN      ] MatrixTest.RowsAndColumns
[       OK ] MatrixTest.RowsAndColumns (0 ms)
[ RUN      ] MatrixTest.SetAndGetValue
[       OK ] MatrixTest.SetAndGetValue (0 ms)
[ RUN      ] MatrixTest.ScalarMultiplication
[       OK ] MatrixTest.ScalarMultiplication (0 ms)
[----------] 3 tests from MatrixTest (0 ms total)

[----------] 1 test from ConjugateGradientTest
[ RUN      ] ConjugateGradientTest.ConjugateGradient
[       OK ] ConjugateGradientTest.ConjugateGradient (0 ms)
[----------] 1 test from ConjugateGradientTest (0 ms total)

[----------] 1 test from MatrixVectorMultiplicationTest
[ RUN      ] MatrixVectorMultiplicationTest.MatrixVectorMultiplication
[       OK ] MatrixVectorMultiplicationTest.MatrixVectorMultiplication (0 ms)
[----------] 1 test from MatrixVectorMultiplicationTest (0 ms total)

[----------] Global test environment tear-down
[==========] 13 tests from 5 test suites ran. (1 ms total)
[  PASSED  ] 13 tests.

The second option is that you have actually implemented all of the tests above into your own local version of your linear algebra solver library and tried to compile it. If you are on UNIX, congratulations, it should have worked out of the box. If you have tried to compile it on Windows, it should work for the first two build scripts provided, i.e. the single build step (no libraries involved), as well as for the static library version.

If you now try to compile the code on Windows as a dynamic library, you will get the following error message:

testVector.obj : error LNK2019: unresolved external symbol "class linearAlgebraLib::Vector __cdecl linearAlgebraLib::operator*(double const &,class linearAlgebraLib::Vector)" (??DlinearAlgebraLib@@YA?AVVector@0@AEBNV10@@Z) referenced in function "private: virtual void __cdecl VectorTest_VectorMultiplication_Test::TestBody(void)" (?TestBody@VectorTest_VectorMultiplication_Test@@EEAAXXZ)
testMatrix.obj : error LNK2019: unresolved external symbol "class linearAlgebraLib::SparseMatrixCSR __cdecl linearAlgebraLib::operator*(double const &,class linearAlgebraLib::SparseMatrixCSR const &)" (??DlinearAlgebraLib@@YA?AVSparseMatrixCSR@0@AEBNAEBV10@@Z) referenced in function "private: virtual void __cdecl MatrixTest_ScalarMultiplication_Test::TestBody(void)" (?TestBody@MatrixTest_ScalarMultiplication_Test@@EEAAXXZ)
.\build\linearAlgebraTest.exe : fatal error LNK1120: 2 unresolved externals

This is the point I mentioned above. If you turn off the TEST(MatrixTest, ScalarMultiplication) and TEST(VectorTest, VectorMultiplication) tests, then your code will compile fine. It would have been easy for me to simply delete these tests and try to ignore this issue, but it would have eventually come back to us and it would have shown itself in a different test or library that we may develop in the future.

So, we need to understand what the issue is, and this is what the final part of this article is about. It is not that straightforward and will require some additional discussion, but it will make us as programmers more robust for cross-platform development. Let’s dive into it.

Fixing Microsoft’s broken C++ mentality

Oh the joy of cross-compiling code on Windows and UNIX. Yet again, Windows requires some additional code modification to work correctly. To be fair, this time the fix we have to do makes sense and you could argue it’s not Microsoft’s fault; however, as you will see, we have to fix code that was originally written to make code compile on Windows in the first place. The code modification we have to do will not affect the compilation on UNIX so it is again a Windows-only work-around.

We first look at the issue at hand and then discuss afterwards which classes are affected and how we can fix them.

Exporting symbols for dynamic libraries

Let’s dissect the utterly useless error message that we are getting from Microsoft’s cl compiler. It states that the operator*(doube const &, class linearAlgebra::Vector) is an unresolved external reference, and this is referenced in the TEST(VectorTest, VectorMultiplication) unit test.

testVector.obj : error LNK2019: unresolved external symbol "class linearAlgebraLib::Vector __cdecl linearAlgebraLib::operator*(double const &,class linearAlgebraLib::Vector)" (??DlinearAlgebraLib@@YA?AVVector@0@AEBNV10@@Z) referenced in function "private: virtual void __cdecl VectorTest_VectorMultiplication_Test::TestBody(void)" (?TestBody@VectorTest_VectorMultiplication_Test@@EEAAXXZ)

Let’s recap quickly how compilation works. When we compile code that depends on other code, be it some other classes we have defined somewhere in our project or code defined in an external dependency (library), the compiler checks that this code exists. We use relative imports in our header files to ensure that we can find all code in our project and then, during compilation, we specify where additional dependencies can be located using either the -I (UNIX) or /I (Windows) flag.

These compiler flags allow us then to write #include "gtest/gtest.h", for example, instead of providing the absolute path such as #include "C:/libraries/include/gtest/gtest.h". The latter approach works fine but is not portable. If I write code like this and you try to import it on your machine, then you need to have gtest installed at exactly the same location, so we tend to use relative imports instead.

The compiler is happy, it can see the dependencies and it can compile your code. Once the compilation is done, we go to the linking stage. The linker now has to resolve all of these external references that we said are there. It will look for them in your compiled object files and in any library that we have given as a linker argument (i.e. on UNIX using the -l flag and on Windows using the /link flag).

If it can’t find any of these references, we get a linker error saying that we have an unresolved reference. This is what the error message above tells us. In this case, however, Microsoft is trying to tell us that the operator*(doube const &, class linearAlgebra::Vector) definition could not be found. If you look into your library, i.e. vector.hpp, you will see that the definition is there (on line 41).

All the other functions were resolved and didn’t lead to a compiler error, so why did this one then? Furthermore, the code compiled just fine as a static library and we were able to get the code to run, why is the dynamic library all of a sudden giving up? The error message is not particularly helpful, and some googling (or rather braving, is that a word?) ultimately resolved the issue, but slowly.

The issue is with the export definition of the linear algebra solver library. Take the simplified example below:

#pragma once

#if defined(WIN32) || defined(_WIN32) || defined(__WIN32__) || defined(__NT__)
  #if defined(COMPILEDLL)
    #define EXPORT __declspec(dllexport)
  #elif defined(COMPILELIB)
    #define EXPORT
  #else
    #define EXPORT __declspec(dllimport)
  #endif
#else
  #define EXPORT
#endif

#include <vector>

class EXPORT MyClass {
public:
  MyClass() { _vector.resize(3); }
  const std::vector<int> &getVector() const { return _vector; }
private:
  std::vector<int> _vector;
};

When we introduced the dynamic library for the linear algebra solver library, we discussed how we had to modify the library in order to work as a dynamic library. This resulted in the rat tail we see in the code above, i.e. lines 3-13, which ensures that we are exporting the library during compilation and importing it during linking if we are on Windows and compiling a dynamic library. In all other cases, the EXPORT macro is an empty string and so won’t modify our code at all.

If we look at the class definition on line 17, we took the lazy (and perhaps pragmatic?!) approach of exporting the entire class. This means our dynamic library will now know about the constructor, the getVector() function, as well as the _vector itself. The class, incidentally, doesn’t do anything useful, I just wanted to construct the smallest possible example for us to look at.

This approach may work fine as long as you are not getting into trouble with the exported std::vector. Typically, you will not see an issue during your development, but exporting the std::vector is actually rather restrictive. For some reason, code compiled on Windows, where an STL container is exported, requires the user to have the same version of the STL (or, at least, a version in which the STL container is the same). Otherwise, it may break. So the code may compile fine for you during development, but it may not for someone else using a different STL version.

However, it can trigger an upset if used in certain ways. As we have seen in the vector test example, we got the error message shown above, which is related to the STL container being exported which triggered some internal compilation error. I am unsure about the exact cause, and so does seem to be everyone else on the internet. It is not an easy problem to debug but ultimately, the consensus among the various sites that I visited was that you should not export STL containers on Windows. So, let’s try to do that instead then:

class MyClass {
public:
  EXPORT MyClass() { _vector.resize(3); }
  EXPORT const std::vector<int> &getVector() const { return _vector; }
private:
  std::vector<int> _vector;
};

We ignored all the preprocessor directives, but they would still be here, I just wanted to look at some clean code. We removed the EXPORT macro from the class definition and now explicitly state which function we want to export. We export here the constructor and the getVector() function, but crucially not the STL container std::vector. Let’s see how we can adjust our vector.hpp and sparseMatrixCSR.hpp file to make our code compile again for dynamic libraries.

Redefining the vector class

We start with the vector class, located at linearAlgebraLib/src/vector.hpp. We do exactly what we have discussed above, and remove the export statement from the class and, instead, export all functions individually. This is shown below.

#pragma once

#if defined(WIN32) || defined(_WIN32) || defined(__WIN32__) || defined(__NT__)
  #if defined(COMPILEDLL)
    #define LINEARALGEBRALIB_EXPORT __declspec(dllexport)
  #elif defined(COMPILELIB)
    #define LINEARALGEBRALIB_EXPORT
  #else
    #define LINEARALGEBRALIB_EXPORT __declspec(dllimport)
  #endif
#else
  #define LINEARALGEBRALIB_EXPORT
#endif

#include <iostream>
#include <vector>
#include <cassert>
#include <cmath>

namespace linearAlgebraLib {

class Vector {
public:
  using VectorType = std::vector<double>;

public:
  LINEARALGEBRALIB_EXPORT Vector(const unsigned &size);

  LINEARALGEBRALIB_EXPORT unsigned size() const;

  LINEARALGEBRALIB_EXPORT Vector transpose();
  LINEARALGEBRALIB_EXPORT double getL2Norm() const;

  LINEARALGEBRALIB_EXPORT const double &operator[](unsigned index) const;
  LINEARALGEBRALIB_EXPORT double &operator[](unsigned index);

  LINEARALGEBRALIB_EXPORT Vector operator+(const Vector &other);
  LINEARALGEBRALIB_EXPORT Vector operator-(const Vector &other);

  LINEARALGEBRALIB_EXPORT Vector &operator*(const double &scaleFactor);
  LINEARALGEBRALIB_EXPORT friend Vector operator*(const double &scaleFactor, Vector vector);  
  LINEARALGEBRALIB_EXPORT double operator*(const Vector &other);
  
  LINEARALGEBRALIB_EXPORT friend std::ostream &operator<<(std::ostream &out, const Vector &vector);

private:
  VectorType _vector;
  bool _isRowVector = false;
};

} // namespace linearAlgebraLib

The VectorType on line 47 is not exported, and we can see from line 24 that this is just an alias for a std::vector. In general, the rule is that we should only export functions or variables that we want to access from outside this class, i.e. the export mechanism on Windows is used to construct the interface that is visible to anyone wanted to interact with the class. If we export a function, it becomes part of the interface.

In a sense, this allows you fine-grained control over what is visible in the interface. In my view, this is completely useless as we already have a mechanism to deal with that in C++ using access modifiers (i.e. private, protected, and public). In any case, we already know that privately declared variables should not be part of the interface, so not exporting them makes sense. This is essentially what we achieve here, by just having to type a bit more on Windows.

Redefining the sparse matrix class

The matrix class, then, follows exactly the same format. We export all the functions in the public interface but keep all private variables outside the interface by not exporting them. This is shown below.

#pragma once

#if defined(WIN32) || defined(_WIN32) || defined(__WIN32__) || defined(__NT__)
  #if defined(COMPILEDLL)
    #define LINEARALGEBRALIB_EXPORT __declspec(dllexport)
  #elif defined(COMPILELIB)
    #define LINEARALGEBRALIB_EXPORT
  #else
    #define LINEARALGEBRALIB_EXPORT __declspec(dllimport)
  #endif
#else
  #define LINEARALGEBRALIB_EXPORT
#endif

#include <iostream>
#include <algorithm>
#include <vector>
#include <cassert>

#include "linearAlgebraLib/src/vector.hpp"

namespace linearAlgebraLib {

class SparseMatrixCSR {
public:
  LINEARALGEBRALIB_EXPORT SparseMatrixCSR(unsigned rows, unsigned _columns);

  LINEARALGEBRALIB_EXPORT void set(unsigned row, unsigned column, double value);
  LINEARALGEBRALIB_EXPORT double get(unsigned row, unsigned column) const;

  LINEARALGEBRALIB_EXPORT unsigned getNumberOfRows() const;
  LINEARALGEBRALIB_EXPORT unsigned getNumberOfColumns() const;

  LINEARALGEBRALIB_EXPORT Vector operator*(const Vector& rhs);
  LINEARALGEBRALIB_EXPORT friend SparseMatrixCSR operator*(const double &scaleFactor, const SparseMatrixCSR &matrix);
  LINEARALGEBRALIB_EXPORT friend std::ostream &operator<<(std::ostream &os, const SparseMatrixCSR &rhs);

private:
  std::vector<double> _values;
  std::vector<unsigned> _columns;
  std::vector<unsigned> _rows;
  unsigned _numberOfRows;
  unsigned _numberOfColumns;
};

} // namespace linearAlgebraLib

With both fixes applied to the vector and matrix class, we are again in a position to compile the code for dynamic libraries on Windows and it should work now. Even if this solution is more verbose, I have to say that I am in favour of writing out the export statements explicitly. It gives us control over what gets exported and what gets left in the class. Even if it replicates what the access modifiers are already doing, I am always of the opinion that your code should document itself, and extra code here and there can help with that.

BONUS: How to lose your sanity with more undebuggable code

While debugging the above issue, I came across another one, which I found rather peculiar. Take a look at the following two statements, which are taken from the matrix class’ header file:

Statement 1:

friend std::ostream LINEARALGEBRALIB_EXPORT &operator<<(std::ostream &os, const SparseMatrixCSR &rhs);

Statement 2:

friend std::ostream& LINEARALGEBRALIB_EXPORT operator<<(std::ostream &os, const SparseMatrixCSR &rhs);

First of all, you have to find the difference between the two. Did you find it? You’ll notice that the export statement is now between the type that the function returns and the function declaration itself. This is how I had the class defined originally when testing, but you can put the export statement at the beginning of the line, which I just find more readable and also, as it turns out, safer.

The difference is in the reference operator (&), in statement 1, it is attached to the function declaration, i.e. &operator<<(), while in statement 2, it is attached to the return type, i.e. std::ostream&. Now usually that doesn’t make any difference if you don’t have any export statements, but, placing the export placement between the return type and function declaration can spectacularly break the code.

I won’t go into the error message here (simply because the error message is rather large). There is a good hint in there as to why all of a sudden the code is not working anymore, but there are also a lot of misleading error messages that will take your attention away from the real problem. If you want to experience that yourself, change the code to what is shown in statement 2 and experience that for yourself.

Summary

So then, at this point, we have looked not just at how to write test fixtures in the previous article, but also how to integrate unit, integration, and system tests into your tests. With this knowledge, you are now ready to write tests for all of your own projects and there isn’t really much more to it than this.

If you prefer to use a different testing framework, one which ideally is also based on xUnit, then you have seen all the important steps that we have to take during testing, i.e. using unit, integration, and system tests, using a test-driven development approach, and writing our tests using the Arrange, Act, Assert pattern.

In this article, we highlighted how we can test our linear algebra solver library and what changes were required to make it compile under Windows. Cross-compilation will always throw a few surprises our way but as we have seen in this article, the solutions are usually not that complicated and we should make an effort to make cross-compilation part of our projects.

---

Tom-Robin Teschner is a senior lecturer in computational fluid dynamics and course director for the MSc in computational fluid dynamics and the MSc in aerospace computational engineering at Cranfield University.