After having reviewed how to compile your own library with CMake in the previous article, including what steps to take so that other people can consume your library in their project, we reverse the role in this article and look at how we can use someone else’s library in our project. In the process, we will look at three different ways how we can bring dependencies into our project. These include manually compiling and installing libraries, asking CMake to handle dependencies for us, and finally, using a package manager.
By the end of this article, you will have a comprehensive understanding of how to handle dependency management with CMake and how to bring in any library you want to use in your own project with ease. Spare yourself long hours debugging compiler instructions, linker errors, and painful hacks to get dependencies into your project and use the tools provided in this article. Your future self will thank you for it in the long run, honestly!
Download Resources
All developed code and resources in this article are available for download. If you are encountering issues running any of the scripts, please refer to the instructions for running scripts downloaded from this website.
- Download: Build-Systems-Part-5
In this series
- Part 1: How to use Makefiles to automate CFD solver compilations
- Part 2: Introduction to CMake for CFD practitioners
- Part 3: Advanced CMake features to power up your CFD development
- Part 4: How to compile, install, and use custom libraries with CMake
- Part 5: How to add external libraries into your CFD solver using CMake
- Part 6: How to automate testing with CMake, CTest, and gtest
In this article
Introduction
In our previous article, we looked at how to compile either a static or dynamic library and then what steps we need to take to install a library so that it exposes important information to other projects if they want to use this compiled library. We also looked at the small changes we need to make to get header-only libraries to work (i.e. installing header files only, no compilation required).
We then used that library in another project to show how we can consume our generated libraries as an end-user. If you followed all the steps in this series up until now, you are in a perfect position to modularise your C++ software projects, which essentially means breaking up your code into smaller units. If we write a CFD solver, for example, we may want to separate the mesh reading from the discretisation, have another module on reading input files, one for turbulence modelling, and so on, you get the idea.
So, in this article, I want to look at CMake’s approach to handling external dependencies. We have looked briefly at find_package(), which allowed us to find our compiled and installed library. You may remember from the previous article that we had to provide the location for CMake where to find the package first (at least if it is in a non-standard path), which can result in an error if not specified. But even if you do specify the path, not all libraries may expose the necessary files for CMake to find all required files (headers and libraries).
Thus, we look at increasingly more self-contained options, starting with a mechanism where CMake is responsible for downloading an external project, compiling it, and making it available for our current project. This is great, but comes with some additional brittle features. This will then bring us to what I would consider the gold standard of dependency management with CMake: using a package manager to download and compile libraries and passing that information to CMake so we are guaranteed to find any dependency.
We’ll explore how to integrate an external library, not one written by ourselves, to showcase how to work with real dependencies (and what potential pitfalls you can come across using either of these methods). So, with these introductory thoughts out of the way, let us jump straight in and develop some code and build scripts to quench that CMake thirst!
Working with Imath: A library for vector-matrix manipulations
I want to start by looking at the library that we will work with, Imath. I have chosen a particularly simple library that only takes a few minutes to learn, but it is powerful enough to write a somewhat meaningful, CFD-related example to showcase why this may be a library that we would consider in a practical situation.
Sidenote: In reality, I would use Eigen for the example that I am about to introduce, which offers all the same functionality as Imath, and then a few (quite a few) more. Eigen is a header-only library and thus not really suitable if we want to show how to work with compiled libraries (more challenging than header-only libraries).
So what can Imath do for us? It is a specialised library to perform vector matrix modifications, most notably, performing vector-matrix multiplications for geometric transformations. If you have ever implemented 3D graphics, you know what I mean. And for those who have never touched 3D graphics, let’s just say that going from an object to what is presented on screen requires a lot of matrix-matrix multiplication to transform the geometry into different view spaces. If you are intrigued, then get lost in this rabbit hole. (Honestly, I love OpenGL and 3D programming!)
But let’s imagine a more realistic application. Let’s say we want to implement moving mesh capabilities into our CFD solver. Then, we may want to be able to rotate some mesh elements around a specific axis. A simple program that would achieve that is given below, using Imath as the underlying library:
#include <Imath/ImathMatrix.h>
#include <Imath/ImathVec.h>
#include <iostream>
#include <numbers>
int
main() {
// Vertices of a triangle
Imath::V3f v1(1.0, 0.0, 0.0);
Imath::V3f v2(1.0, 1.0, 0.0);
Imath::V3f v3(2.0, 0.0, 0.0);
// Rotation matrix
Imath::M44f m;
m.makeIdentity();
m.rotate(Imath::V3f(0.0, 0.0, std::numbers::pi/2.0));
// Rotated vertices
Imath::V3f v1Rotated, v2Rotated, v3Rotated;
m.multVecMatrix(v1, v1Rotated);
m.multVecMatrix(v2, v2Rotated);
m.multVecMatrix(v3, v3Rotated);
// Print rotated vertices
std::cout << "v1Rotated: " << v1Rotated << std::endl;
std::cout << "v2Rotated: " << v2Rotated << std::endl;
std::cout << "v3Rotated: " << v3Rotated << std::endl;
return 0;
}Here, we define some dummy vertices to form a triangle on lines 9-11 using a float-based 3D vector (V3f), and then we construct a 4×4 transformation matrix on line 14, which we set to the identity matrix on line 15. Line 16 then applies the rotation we want to achieve. In this case, I want to rotate everything around the z-axis (third entry to the vector) by \pi/2 (radians), which is equivalent to 90^\circ.
Our transformation matrix (rotation only, but we could have also applied scaling and translation) is now set, and I have shown below what we want to achieve using a small animation:
We can see here that we may have some form of circular mesh, and we are rotating around the origin (by 90^\circ). We can see vertices 1-3 defined with a red, green, and blue vertex, respectively, which correspond to the vertices we defined in the C++ file above, i.e. on lines 9-11.
The rotation is carried out on lines 19-22 and the results are printed to the console on lines 25-27. That’s it, no rocket science, just a simple rotational transformation. We will see now how we can can use the various methods in CMake to find the Imath library and make it available to the above-shown code.
Project structure
Let’s quickly review how the project is structured before we look at the various methods of finding and managing dependencies. The project that we will work with looks like the following:
root
├── build/
├── CMakeLists.txt
├── main.cpp
└── (conanfile.txt) // only for conan projectsWe have our root-level build/ folder as is customary with CMake. Then, we have the CMakeLists.txt file, which will find the Imath library for us and then compile our main.cpp file, which will link against Imath. The main.cpp file is the one we looked at in the previous section. There is one additional file called conanfile.txt, which is required by Conan if we want to use this package manager. We will come back to this file at the end when looking at package manager integration with CMake.
The classic way: find_package()
The first approach I want to look at is the one I would label the classical approach. I say classic, by which I mean providing dependencies ourselves. This means we have to download, compile, and install libraries ourselves and then tell CMake where to find them. We have looked at this manual process in detail when we compiled the CGNS library from scratch. We saw that the CGNS library requires some additional dependencies (zlib and HDF5) before we can compile the CGNS library itself, and the process can quickly become time-consuming.
But let’s say that is what we want to do. From a learning perspective, this is certainly the best approach, as you likely get errors eventually with CMake and have to troubleshoot them. Case in point: try to do it with the CGNS library, and you won’t succeed. The issue is that the CGNS library does not provide the required *.cmake files, so CMake won’t be able to find the library easily. Of course, we could always hack our CMake file and provide hardcoded paths to libraries and header files, but that’s a bad solution.
Returning to our CMakeLists.txt file, the content is provided below:
cmake_minimum_required(VERSION 3.23)
project(
ImathTest
VERSION 1.0
LANGUAGES CXX
)
# Set the C++ standard to C++20 (required to get pi, honestly, C++ added pi only in the year 2020 ...)
set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
# Find packages will search for a specific library. CMake will fail if the package is REQUIRED but not found
find_package(Imath REQUIRED)
# Add an executable and provide the source file directly.
add_executable(${CMAKE_PROJECT_NAME} main.cpp)
# Make sure you link your executable with your library. Notice the namespace that we defined earlier.
target_link_libraries(${CMAKE_PROJECT_NAME} PRIVATE Imath::Imath)Since we rotate by \pi/2 (radians), we need to enable the C++ 2020 standard, as Pi was only added then to the C++ standard! (there were always ways to get it into your code by either approximating it, hardcoding the first digits, or using PI from C-headers). We require that Imath be available on line 14; otherwise, the compilation will be aborted. Then, we first create our test program based on the main.cpp file on line 17, which gets linked against Imath on line 20.
Compiling and installing Imath manually
We have looked at the steps in detail how to compile and install a library by now, i.e. see, for example, how to build the CGNS library from scratch or the previous article where we compiled and installed our own small test library. Look at these articles if you need a refresher, we will quickly walk through how to do this for the Imath library in this section.
First, head over the the Imath github repository. I’ll be working with version 3.1.11, which you can download from their release page if you want to use the exact same version as me (though as long as you use any version start with 3.x.x, you should be fine).
Download the source code as either a *.zip or *.tar.gz format and extract it somewhere on your PC (it doesn’t matter where, as we will specify an install directory later). Open a terminal and change into the Imath root-level directory. Create a build/ folder, and change into it. Then, we can compile the library with the following commands:
cmake -DCMAKE_BUILD_TYPE=Debug -G Ninja ..You can, of course, use any other generator than Ninja or simply omit the -G Ninja option to use the native build system of your platform. With our project configured, we can build it:
cmake --build . --config DebugHopefully, we didn’t encounter any issues. If that is the case, we should now have the library available to us. If you want to verify that, change into the bin/ folder, (build/bin/) and execute the ImathTest.exe file (ImathTest on UNIX). It will run some tests, and if no errors are found, we should be confident that the library is correctly built.
The next step is to make it available by installing it on our system. For testing, I like to use the C:\temp directory on Windows, or ~/temp on Unix, which will be in my user’s home directory. To do so, write either
cmake --install . --prefix C:\temp --config Debugon Windows, or
cmake --install . --prefix ~/temp --config Debugon UNIX. You are allowed to change the folders where you want to install the library to, you have my blessing! Make sure that the folder you have specified does now indeed contain the lib/ and include/ folders, which contain the required library and header files. It is also a good idea to check that you have the lib/cmake/<lib-name> folder available (here <lib-name>=Imath), which itself should contain at least the <lib-name>Config.cmake and <lib-name>Targets.cmake files. This indicates that we ought to find this library later, with find_package().
If you do the same for the CGNS library, you’ll notice that the cgnsConfig.cmake file is missing, indicating that the library developers did not provide this for us during the installation step. This means we do not stand any chance of using find_package() to locate the CGNS library, without having to do some modifications (or hacks) to our CMakeLists.txt file.
Also note that I am building in debug mode here, as I am planning to link this later with my project with will be compiled in debug mode as well. For UNIX, it doesn’t matter if the library is compiled as either debug or release, but on Windows, we do have to differentiate between the two, otherwise we get linker errors, as explored in the previous article.
We now have Imath available and can continue to consume it in our own library. Let’s see how we can do that.
Using Imath in our project
First, let’s change into the root-level directory of our project on the console. Then, go into the build folder as indicated in the project structure above. To compile our current project, simply run the following command:
cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=C:\temp -G Ninja ..Notice that we have to specify where Imath is located. We do that by specifying the root level directory with the CMAKE_INSTALL_PREFIX variable. In my case, I have the library installed in the C:\temp\lib directory, and the header files located in C:\temp\include, as we specified C:\temp as the installation prefix earlier. On UNIX, you would have to change this directory to something more UNIX-like, for example:
cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=~/temp -G Ninja ..where the library is now located in the temp/ directory within the user’s home directory. In either case I use Ninja, simply because it is likely the fasted build tool out there, but also because it works cross-platform and I like to use the same tools across different platform if I have a choice. But feel free to use your favourite generator.
Once the configuration is done and CMake hasn’t complained about Imath not being available, then we can go ahead and build our project with the following command:
cmake --build . --config DebugTo test the executable that we have generated, run the executable directly within the build/ directory, i.e.
.\ImathTest.exeon Windows and
./ImathTeston UNIX. If you used MSBuild on Windows or Xcode on macOS, these executables will be located within build/Debug/, so change directories before running the executable.
If you do, you will see the following output printed to the console:
v1Rotated: (-4.37114e-08 1 0)
v2Rotated: (-1 1 0)
v3Rotated: (-8.74228e-08 2 0)Looking at the animation above, we can indeed verify that the coordinates are correct (the x value of the first and third vertex being essentially zero).
You may run into trouble if you are on Windows, as the library is built as a shared (dynamic) library. If that is the case, you may need to copy the *.dll file from C:\temp\bin into your build/ directory, or wherever the ImathTest.exe file is located. In my case it isn’t located in the same folder as the executable, and it should not work. However, we are talking about Windows here, so anything is possible.
Wow, hang on, a dynamic library on Windows without any pre-processor definitions?
As much as cfd.university is a place about learning how to code specific CFD aspects, it also doubles as a personal journal where I document how Microsoft is slowly shipping away at my sanity (in the quest of supporting cross-platform compilations). Especially when it comes to working with shared libraries on Windows. The astute reader will remember the dreaded article #17, where it all started.
In the previous article, we got the shared (dynamic) library to work by specifying the compiler pre-processor directive COMPILEDLL (i.e. see the CMakeLists.txt file, line 27 in the linked article). This ensured that each function was prepended with __declspec(dllexport). We then also defined the same pre-processor directive in the test application, which resulted in each function being correctly prepended with the __declspec(dllimport) statement. We got our shared library to work this way.
However, when we linked against Imath, it was also building a shared (dynamic) library on Windows, and if you look in its source code, specifically into <Imath-root>/src/Imath/ImathExport.h, you’ll find this wonderful code:
#if defined(IMATH_DLL)
// when building Imath as a DLL for Windows, we have to control the
// typical DLL export / import things. Luckily, the typeinfo is all
// automatic there, so only have to deal with symbols, except Windows
// has some weirdness with DLLs and extern const, so we have to
// provide a macro to handle that.
# if defined(IMATH_EXPORTS)
# define IMATH_EXPORT __declspec(dllexport)
# define IMATH_EXPORT_CONST extern __declspec(dllexport)
# else
# define IMATH_EXPORT __declspec(dllimport)
# define IMATH_EXPORT_CONST extern __declspec(dllimport)
# endif
#else
// ...
#endif // IMATH_DLLIt’s good to see that it is not just me who finds it weird. You see that if the IMATH_DLL is defined, then we set the IMATH_EXPORT macro to either __declspec(dllexport) or __declspec(dllimport). The only thing we need to do is to define the IMATH_EXPORTS (notice the s at the end) when building the library to get the dllexport behaviour otherwise, we’ll default to dllimport, which is the correct behaviour. But how was this correctly called? We never specified IMATH_DLL, so how did the library know?
Well, this is where a bit of CMake magic comes in. If we have a look into the <Imath-root>/config/LibraryDefine.cmake file, we can see the following code (I’ve shortened it here to concentrate on the important bits):
function(IMATH_DEFINE_LIBRARY libname)
# ...
set(objlib ${libname})
add_library(${objlib}
${IMATH_CURLIB_HEADERS}
${IMATH_CURLIB_SOURCES})
if(IMATH_CURLIB_PRIV_EXPORT AND BUILD_SHARED_LIBS)
target_compile_definitions(${objlib} PRIVATE ${IMATH_CURLIB_PRIV_EXPORT})
if(WIN32)
target_compile_definitions(${objlib} PUBLIC IMATH_DLL)
endif()
endif()
# ...
endfunction()First, we create the library on lines 6-8, and then we check if we are building a shared library on line 10. If so, we further check if we are on Windows on line 12, and if this is also the case, we add some target_compile_definitions() to our library. We did something very similar with our complex number library in the previous article, where we had a similar statement, which was
if(${ENABLE_SHARED})
add_library(${CMAKE_PROJECT_NAME} SHARED)
if (MSVC)
target_compile_definitions(${CMAKE_PROJECT_NAME} PRIVATE COMPILEDLL)
endif()
else()
add_library(${CMAKE_PROJECT_NAME} STATIC)
if (MSVC)
target_compile_definitions(${CMAKE_PROJECT_NAME} PRIVATE COMPILELIB)
endif()
endif()First, we check if the library is shared on the first line and then create the shared library on line 2. We check if we are using Microsoft’s cl.exe compiler (which, of course, is named MSVC, why wouldn’t it be), and then add the COMPILEDLL statement. If it is not a shared library, then we are building a static library and add the COMPILELIB statement instead on line 12 for the static library defined on line 9.
The difference between our complex number example and Imath is that we used target_compile_definitions(name PRIVATE ...), whereas Imath used target_compile_definitions(name PUBLIC ...). You may remember an earlier discussion we had on PRIVATE and PUBLIC, where we said that everything set to PRIVATE will only be visible to our current target. In this case, we define COMPILEDLL as PRIVATE, and so only our complex number library will know about it. If we use this library, we need to set it up again.
PUBLIC, on the other hand, will pass on any definitions to other projects that use a specific target. So, in the case of Imath, it defines IMATH_DLL as PUBLIC, and so when we are using Imath in our project, we will also get the IMATH_DLL definition, without ever having to set it. If you are shipping your library for other people to use, then this is what you want to do. Define all pre-processor directives they need as PUBLIC, people will thank you later for it.
So why did I not do it? Well, the honest answer is, it was an oversight on my side. I set it to PRIVATE, because I usually only export headers, but overlooked that we can do that with pre-processor directives as well. However, I think this is not really an issue (otherwise, I would have simply changed the previous article), and, in fact, there is educational value in setting this twice. CMake hides all of this complexity from us, if we instruct it to do so, but having to specify this manually in the consuming project means that we don’t lose sight of this Windows-only issue.
The modern way: FetchContent
Let’s step back for a moment and think about what we have done in the previous section. We manually went to Imath’s website to download the source code (in some form of a zip archive), extract it, open a console, change into the build/ folder, compile the library and make it available to other projects by installing it into a specific location.
All we really need is the URL from where to retrieve the source code. The rest of the steps are pretty much the same for all CMake projects (apart from setting some project-specific options). So why not automate this entire process? Well, that is what the developers of CMake thought as well, and they gave us two options, not just one, to deal with this. These are ExternalProject and FetchContent, both of which offer functionality that we can implement directly into our CMake scripts. The differences between the two are detailed below:
- ExternalProject: This allows you to download and compile a third-party project (library, executable) which can have any build script (it does not have to be CMake). After it is compiled, it will be available for the current project to use. It is downloaded and compiled during the configuration step, meaning that your external project and current project can have different configurations (debug or release), which may cause issues on Windows.
- FetchContent: This is very similar to ExternalProject, but it assumes that the build script is CMake (though there are ways around it). If that is the case, it can be integrated much better with the current project. It still downloads and compiles the project alongside your own project. However, it does all of that during the build step, not the configuration. Project settings that are common are shared among your project and the fetched one, meaning that if you specify a debug build, the fetched project will also compile in debug mode, and the same is true for release builds.
So FetchContent allows us to bring in dependencies and delegate the task of resolving all internal linkages to libraries, as well as finding header include directories to CMake. On paper, this is great, in practice, I have lost a lot of time with trying to get FetchContent to work, but I am jumping ahead of myself. I’ll get to some of the shortcomings in a bit. First, let’s inspect our CMakeLists.txt file first!
Modifying our CMakeLists.txt file
Compared to the previous example, we have to make some modifications in our build script. Have a look through it and then we discuss the changes necessary below.
cmake_minimum_required(VERSION 3.23)
project(
ImathTest
VERSION 1.0
LANGUAGES CXX
)
# Set the C++ standard to C++20 (required to get pi, honestly, C++ added pi only in the year 2020 ...)
set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
# First, require that FetchContent is available
include(FetchContent)
# Declare the location from where to download Imath
FetchContent_Declare(Imath
GIT_REPOSITORY https://github.com/AcademySoftwareFoundation/Imath.git
GIT_TAG v3.1.11
)
# Compile Imath and make it available for current project
FetchContent_MakeAvailable(Imath)
# Get some information about Imath, such as its compilation directory
FetchContent_GetProperties(Imath)
# Find packages will search for a specific library. CMake will fail if the package is REQUIRED but not found
find_package(Imath REQUIRED)
# Add an executable and provide the source file directly.
add_executable(${CMAKE_PROJECT_NAME} main.cpp)
# Make sure you link your executable with your library. Notice the namespace that we defined earlier.
target_link_libraries(${CMAKE_PROJECT_NAME} PRIVATE Imath::Imath)
# Fix: CMake can't find Imath headers, as not properly set in IMath's CMake file. Include it manually here ...
target_include_directories(${CMAKE_PROJECT_NAME} PRIVATE "${imath_SOURCE_DIR}/src")The bulk of the CMakeLists.txt is the same, but lines 13-26 were added to deal with finding the Imath library online and making it available to the current project. If we want to make FetchContent available in our project, we first need to include it, as done on line 14. This will make calls to the FetchContent API available.
We then use the FetchContent_Declare on line 17 (you have to admire CMake’s inconsistent usage of naming conventions, FetchContent_Declare uses both Pascal case and (capitalised) snake case, while other calls like add_executable() on line 32, use pure snake case. Software engineering at its best!). The arguments are a target name, so we can refer to it later (line 17, here Imath), and then we simply specify the GitHub repository, along with the version we want to download.
The call to FetchContent_MakeAvailable() (seriously, I feel compelled to write a strongly worded letter to the makers of CMake. Is it just me or is this naming abomination inducing anxiety for you, too?) is responsible (for anxiety, but also) for compiling the project during the build stage. And in theory, that’s it. All you need is a call to FetchContent_Declare() and FetchContent_MakeAvailable() (and perhaps to your doctor), to get dependencies into your project. But you see, life isn’t that simple.
If you tried to only use these two statements, which the CMake documentation promises you is all you need, then you run into trouble. Depending on which version of the library you use, you get different error messages, but they are all related to the same underlying issue. The problem is that CMake can’t find the path to the header files (i.e. the include path). As a result, the project won’t be able to be compiled.
Thus, we have to manually hack this into the CMake build script, and this is what we do on line 26. By calling FetchContent_GetProperties(), we expose a few variables from the fetched project that we can use in our own project. One of these variables is imath_SOURCE_DIR, which will point to the Imath root folder, which was downloaded and extracted as part of the FetchContent_MakeAvailable() call. We use this information to include the directory with all header files on line 38 for the current project, i.e. imath_SOURCE_DIR/src/Imath.
With this call, our project can be compiled again. Let’s see how we can do that. If you have downloaded the source code at the top of this article, change into the FetchContent/build/ folder. Then, simply execute this following line (and use your favourite build tool if Ninja is not for you):
cmake -DCMAKE_BUILD_TYPE=Debug -G Ninja ..Then, we go ahead and build the project as usual:
cmake --build .You will notice that the build now requires slightly longer compared to the previous section, since we are building the Imath library now along with our own project. Once this is done, we can execute the tests again using
.\ImathTest.exeon Windows, and
./ImathTeston UNIX. Happy days, it is working (hopefully), and we can now automatically handle all dependencies, without any issues, right? Well, I mentioned earlier that there are some issues with the FetchContent API (and for that matter, with ExternalProject as well), let me elaborate.
I have used FetchContent extensively on a project where I was developing a graphical user interface to do some geometry clean-up tasks. This software was reading in a *.stl file and my goal was to split it into separate sections so that I could apply separate boundary conditions to separate collection of triangles in OpenFOAM’s mesh generator snappyHexMesh.
The project required a graphical user interface library (I used wxWidgets), as well as OpenGL, to display 3D graphics on the screen, which I could embed into a wxWidgets window. OpenGL was already on my system, but interfacing it required a few more libraries like GLM (for mathematical operations, very similar to what Imath is doing) and GLEW (communicating with the OpenGL layer itself), which I attempted to include, as best as possible, with FetchContent and ExternalProject.
Long story short, for simple libraries, FetchContent works like a charm; for anything more complicated, you’ll start hacking solutions into your CMake files that are brittle and likely break regardless. I was able to come up with a solution that took a good two weeks to test and validate, only to find out six months later that it did not work anymore, even though I didn’t change my CMake version nor any version of the dependencies I was downloading. It’s one of many Heisenbugs I produced over the years.
In the end, I replaced that with a custom python script which was handling the downloading, compiling, and installing of the dependencies. it is working to this day without issues.
The other issue is complex dependencies. Take the CGNS library, for example. If you want to be a CFD engineer of the 21st century, you’ll likely want to use it with HDF5 as the underlying compression algorithm. HDF5 requires zlib, in return, and so if you want to built the CGNS library, you first need to build HDF5, and to build HDF5, you first need to build zlib. This rat tail of dependencies is something FetchContent probably can handle, but I wasn’t able to get it to work, and I have tried it a few times years ago (perhaps it has changed now).
While the idea of having someone else handle your dependencies is great, ExternalProject and FetchContent do not live up to that expectation for me. I am not saying that these tools aren’t mature enough; they are quite decent, but they are brittle and difficult to use for a wide range of dependencies. As a result, I would not recommend using it, but it is good to be aware on a conceptual level, at least, of what these two methods can do. Instead, I would recommend using a package manager, and this is what we will look into next.
The gold standard: Using a package manager (Conan, not its mentally insane brother, vcpkg!)
We did look at package managers a while ago, and we had a brief look at how to use them to build dependencies. In our example, we looked at the CGNS library. At that point, though, I have not talked about CMake yet, so some of the details may have gone over your head. I’ll summarise the steps here again.
There are two reasons why I prefer using a package manager for C++, and Conan in particular. Conan is one of the larger package managers and has receipts for pretty much any library out there. Given that users can contribute Conan recipes to build their own library, chances are that the library you want to build already has a Conan recipe available.
Some libraries, though, just can’t be built with Conan. I call that the PETSc test. PETSc is a very powerful and capable library to solve linear systems of the form \mathbf{Ax}=\mathbf{b}. It has tons of linear algebra solvers, preconditioners, and native support for parallel processing using MPI, which makes it a great tool to drop into your CFD solver. The only issue is, it compiles on UNIX platforms only, there isn’t any support for Windows.
Before you jump in my face, yes, I know, you can use it on Windows with mingw. But then I would argue, you can also use it on Windows using the Windows Subsystem for Linux (WSL). Mingw is based on UNIX tools and not a native Windows solution, so when I say there is no support for Windows, I mean native support using Microsoft’s compilers.
Consequently, Conan does not list PETSc, as it is difficult to build on different platforms. This also means, however, that if something is available on Conan, then chances are high that you can use it on any operating system, and you don’t lock yourself into a UNIX-only situation, as many CFD developers do.
Of course, dependency management also becomes as simple as specifying which library you want Conan to download for you and then use it. The best part, though, about Conan is that it will check your system and hardware architecture. If the library has already been built for this platform and hardware, Conan will simply download the compiled version instead of downloading the source code and compiling it from scratch. This saves on compilation time and is a huge advantage.
Once Conan is done and has collected all dependencies, it exposes those to CMake through a toolchain file. What is a toolchain in CMake? Loosely speaking, a toolchain contains information about all the tools we need to develop a project. At a minimum, we need a compiler and a linker. However, when we depend on other libraries, these become part of the toolchain, and Conan will write the path, the header files, the library itself, and the name of the library, as well as any other necessary information, into a toolchain file.
If we then use the toolchain during the compilation, CMake knows where to find all of this information. In general, we always need a toolchain when using CMake, but usually, we just use the default one provided by CMake, i.e. CMake will guess where to find the compiler and linker (e.g. /usr/bin/ on most UNIX platforms) and as long as they are available, we don’t have to provide a toolchain explicitly.
If you think about it, this is very similar to what we did two sections ago, where we provided an explicit directory to CMake to search for header files and libraries, i.e. by specifying -DCMAKE_PREFIX_PATH=C:\temp or -DCMAKE_PREFIX_PATH=~/temp during the configuration stage. CMake was then looking by itself for the required compiler and linker, and combined with the provided directory, had all the informaton required to build our project. A toolchain just collects all of that information and we just have tell CMake to use that toolchain file. Let’s see how we can achieve this.
The (unmodified?) CMake build script
Yes, that’s right, we don’t have to change anything about our CMake build script. Since Conan version 2 and onwards, we have used the aforementioned toolchain technique. In version 1, we needed to include a few additional settings in our CMake file, meaning that Conan-specific instructions were included in our build script, even if we didn’t plan on using Conan. Using the toolchain option allows us to write a clean build script. For completeness, the CMakeLists.txt file is given below.
cmake_minimum_required(VERSION 3.23)
project(
ImathTest
VERSION 1.0
LANGUAGES CXX
)
# Set the C++ standard to C++20 (required to get pi, honestly, C++ added pi only in the year 2020 ...)
set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
# Find packages will search for a specific library. CMake will fail if the package is REQUIRED but not found
find_package(Imath REQUIRED)
# Add an executable and provide the source file directly.
add_executable(${CMAKE_PROJECT_NAME} main.cpp)
# Make sure you link your executable with your library. Notice the namespace that we defined earlier.
target_link_libraries(${CMAKE_PROJECT_NAME} PRIVATE Imath::Imath)As you can see for yourself, it is exactly the same as in the section where we manually provided the location to our library and so we can treat both Conan and CMake as either separate entities or combine them, without having to modify any code/instructions.
Input files for Conan
What we have to do, though, is to provide a so-called conanfile.txt. This file collects all the libraries, including their version, that we want to build and have made available for our current project. In our example, this file will look like the following:
[requires]
imath/3.1.11
[generators]
CMakeDeps
CMakeToolchainFirst, we say that the library Imath, version 3.1.11, is required. This means Conan will download and build (if necessary) this library and then create the files specified by the generators, where the CMakeToolchain in particular, is of interest. This is all; we just dealt with our dependency! But how do you know the library is available? Well, Conan comes with a corresponding website called Conan Center. It is like GitHub, where you can upload Conan recipes to build libraries (or, well, download them if you want to use them in your own project).
If you search for Imath on Conan Center, you will get some instructions, including what to put into your conanfile.txt. It is pretty self-explanatory, and Conan gives you additional instructions for how to use Conan from the command line to get this dependency, as well as how to change your CMake file and what headers to include in your project! Give it a try and see the results for yourself. Conan Center provides a decent search algorithm as well, so you can search for features as well, and Conan will return libraries that have these implemented.
For example, searching for the keywords linear library solver preconditioning returns both Eigen and Armadillo as available libraries, and we can see that both of them have the features we need, and both of them work on Windows. Happy days! We no longer need PETSc! (Eigen is a header-only library, as these don’t get compiled, they’ll always work on all platforms, including Windows).
Command line instruction to marry CMake with Conan (the most beautiful day of your and my life (don’t tell my wife …))
The first thing we need to do is to make Conan available. It is written in Python, and thus, we need Python available on our system. Python has its own package manager, called pip, and we use pip to install Conan on our system. Pip may not be installed by default if you install Python, but you can instruct Python to install it with
py -m ensurepip --upgradeon Windows, and
python3 -m ensurepip --upgradeon UNIX. For additional information, see the pip documentation (it is not rocket science; I trust you can handle it). Once pip is available, use it to get Conan:
py -m pip install conanThis command is for Windows, again, for UNIX, use the following:
python3 -m pip install conanNow, we need to detect our default profile (which will be sufficient for now). The profile details which operating system you are running, whether you want to build it in release or debug mode, your hardware architecture, and so on. You can find details in the Conan documentation. This information is then used by Conan to check if a compiled library for your build type, hardware, and operating system already exists. If that is the case, Conan can simply download it as mentioned earlier, there is no need to compile t first.
To create a default profile, run the following command
conan profile detect --forceThis will give us some sensible default settings. On Windows, for example, this is what I am getting as my default profile:
[settings]
arch=x86_64
build_type=Release
compiler=msvc
compiler.cppstd=20
compiler.runtime=dynamic
compiler.version=193
os=WindowsConan will now use that information whenever it needs it. For completeness, running conan on WSL (i.e. Ubuntu on Windows) gives me the following information:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu20
compiler.libcxx=libstdc++11
compiler.version=11
os=LinuxSlightly different, but again, these are just default values Conan will use during compilation if need be. We can now start to think about different profiles, i.e. we may want to create a developer profile, where we build everything in debug mode, and a release profile to optimise our build for performance.
Let’s make one change to the default profile to see how we can modify it. All profiles are located in your home directory (i.e. C:\Users\<username>\ on Windows and /home/<username>/ on UNIX). Here, you have a hidden folder called .conan2/, where you have a profiles/ folder. All profiles that you want to use are stored here. In my case, the default profile I just created is at C:\Users\tom\.conan2\profiles\default on Windows, and /home/tom/.conan2/profiles/default on UNIX.
If we open that file, we see the same content as above. I want to tell Conan that my preferred generator in CMake is Ninja. Otherwise, it will assume I use the default generator for my platform (which is MSBuild on Windows and Makefiles on Linux). We can see from Conan’s documentation that we need to add the following section at the end of the file.
[conf]
tools.cmake.cmaketoolchain:generator=NinjaIf you don’t have Ninja installed or you are happy with the default generator, then you can skip this step. If you want to create a new profile, simply copy the default profile and make changes as you need. You can see a full list of supported sections and options in the documentation that you can set in your profiles.
With a profile in place (you only have to do this once after installing Conan, not every time you plan on running Conan), we can open a console, change into the Conan folder provided in the download at the beginning of this article, and then install all required dependencies (here just Imath). We use the following command for that:
conan install . --output-folder=build --build=missing --settings=build_type=DebugWe instruct Conan to install our dependencies (which will include a built step if required) and to look for the conanfile.txt in the current directory (indicated by the dot (.)) We tell it to store all dependencies in the build/ folder, and only to build dependencies if they are not available as precompiled binaries. We are also overwritting here some settings from our default profile, just to show that we can. Alternatively, you can copy the default profile, change the build_type to Debug, and then use that profile instead.
Let’s assume that we have such a profile, which we have called dev, then we can use it during the Conan install step as
conan install . -pr=dev --output-folder=build --build=missing --settings=build_type=DebugRunning conan install -h, by the way, shows all the options we can specify, and this will provide us with all the options that we can pass on to the install step. This works for other sections as well, i.e. we can also do conan profile -h, which will show us the relevant help section for the profile command.
Once all dependencies are installed and put into our build/ folder, we can change into the build/ folder in our console, and then execute the CMake configuration step using the following command:
cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_TOOLCHAIN_FILE="conan_toolchain.cmake" -G Ninja ..Notice that we now specify the toolchain file conan_toolchain.cmake, which was generated by Conan as an output and which is located in the build/ folder. If you look through this file, you’ll notice that it essentially provides the path to where to find the library, with some additional compiler and linker flags that are hardware and platform-specific (potentially).
If you specified the CMake generator to be Ninja in your Conan profile, then you must specify it here as well with the -G Ninja flag, otherwise you likely run into trouble, as Conan has written its toolchain file assuming Ninja will be used. On Windows, for example, the default build system (MSBuild) has a different build layout compared to Ninja, i.e. these are incompatible, and if you omit -G Ninja here and MSBuild is used instead, you will get an error.
If you are not sure, then simply remove the [conf] section from your Conan profile (if you have added that as shown in the step above). Do not provide a generator during the CMake configurations tep, i.e. remove the -G Ninja instruction, and then, both Conan and CMake will correctly guess the right build tool on your system and use it correctly.
To build our test project, run
cmake --build . --config Debugwhich should make the test executables available. On Windows, verify this by running
.\ImathTest.exeor, if you are on UNIX, execute
./ImathTestYou should be seeing the same output as with the previous two approaches using the native find_package() and FetchContent_Declare|MakeAvailable() commands.
Summary
We have taken your skills on how to handle dependencies with CMake to the next level by first considering how to use the primitive approach of managing dependencies yourself (i.e. manually downloading, compiling and installing dependencies) and then looking at more advanced approaches by letting CMake (using FetchContent) or Conan manage all dependencies for you.
In this day and age, we have technologies available to handle all of our dependency management needs, and I would strongly recommend that you at least try to use Conan for one of your projects. Dependency management has never felt that great, it is the same feeling switching from a normal keyboard to a mechanical one, the typing just feels that much different (and better!).
This article, together with the previous one, really exposes how to efficiently and effectively deal with dependencies, be it writing your own libraries and making these available to others (previous article), or, using someone elses libraries in your project (current article). From today, you have no good excuse not to use someone elses library and isntead of reinventing the wheel, see if someone has not already provided a library for what you want to achieve, and then just use it, your productivity will increase significantly as a result, I guarantee it!
Tom-Robin Teschner is a senior lecturer in computational fluid dynamics and course director for the MSc in computational fluid dynamics and the MSc in aerospace computational engineering at Cranfield University.