How to use Conan to manage your own CMake-based libraries

In this final article in our series on build automation, we look at how we can provide our own Conan recipe to compile and package our libraries with Conan so that we can store them in our local Conan cache for easy reuse in other projects. We will be able to put our own libraries into a conanfile.txt as a dependency, and Conan will not only make these libraries available to us, but also resolve any dependencies our libraries depend on in the background, e.g. the CGNS library.

By the end of this article, you will be a Conan master and will know how to use it to bring in external and internal dependencies. Never before has developing CFD libraries been so painless and we should use the tools available to use to make our life, and that of our users, as simple as possible, so that all the hard work we put into our library will pay off and users will start to use our work.

If you have followed this series, you will also have become a black belt CMake practitioner, and you will be able to automate all of your software projects, from configuring to packaging. Knowing CMake, especially in conjunction with Conan, will make you more productive, and you will find that building software has never been that simple before, especially if you have to resolve dependencies.

Download Resources

All developed code and resources in this article are available for download. If you are encountering issues running any of the scripts, please refer to the instructions for running scripts downloaded from this website.

• Download: Build-Systems-Part-10-meshReaderLib
• Download: Build-Systems-Part-10-linearAlgebraSolverLib
• Download: Build-Systems-Part-10-testing

In this series

• Part 1: How to use Makefiles to automate CFD solver compilations
• Part 2: Introduction to CMake for CFD practitioners
• Part 3: Advanced CMake features to power up your CFD development
• Part 4: How to compile, install, and use custom libraries with CMake
• Part 5: How to add external libraries into your CFD solver using CMake
• Part 6: How to automate testing with CMake, CTest, and gtest
• Part 7: Never worry about Docs again! Automate it with CMake instead
• Part 8: How to use CMake/CPack to create beautiful GUI-installers
• Part 9: Complete walkthrough: Adding CMake to an existing library
• Part 10: How to use Conan to manage your own CMake-based libraries

In this article

Introduction: How to impress your users

In my previous article on how to manage dependencies with CMake, we looked at three separate ways to manage dependencies:

• Using a combination of CMAKE_PREFIX_PATH and find_package()
• Using CMake’s FetchContent module
• Using Conan as a package manager

The first option works well if the library you are trying to use has provided all the required *.cmake files. There are some slight complications arising from using find_package(), in that it can be operated in either CONFIG or MODULE mode, which will determine how CMake is looking for the required files. We have only been using CONFIG mode, and that is the preferred choice nowadays. But for completeness, here is a good answer for the difference.

The FetchContent module is a nice idea and useful as an onboard tool if you want to keep dependency management within CMake. However, it doesn’t always work the way you want, and you can spend ages trying to get the configuration settings right. You can always opt for the ExternalProject module instead (which we did not elaborate on, but it can be regarded as a more flexible alternative to FetchContent), but this is just hardcoding build requirements that may need to change over time and thus can lead to code-breaking all of a sudden.

Using a package manager, in my view, is a much better option, and we reviewed, at least conceptually, available options for C++ in an earlier article. Chances are, if you are using either FetchContent or ExternalProject, you are just one step away from using a package manager, and you may as well make use of all the additional features a package manager can offer you, compared to CMake, for which both FetchContent and ExternalProject are just two additional modules (and much less powerful in comparison to, say, Conan).

So, I hope to have made my point in previous articles why using a package manager is, in general, a good idea. We have used Conan extensively in this series to bring in dependencies (primarily CGNS, Gtest, and Doxygen). But what if we wanted to use Conan to package our own library so that others can use our library through Conan?

Well, this is what we will look at in this article. If you already understand the install step in CMake, and the conanfile.txt that we have looked at previously, then making the next step to writing our own recipe for Conan to handle our library is not that much more difficult (and, I suppose, the difficulty level will be determined by your confidence in using Python!).

This is what we will do in today’s article:

• First, we will make necessary changes to transform our mesh reading library to use Conan not just to bring in dependencies but also to package our library so that it can be used and consumed by other projects
• Then, we will take a look at what changes needs to be done for the linear algebra solver library as well. These are, however, minimal, as you will see.
• Finally, we will reuse our test project from the previous article to test that both the mesh reading and linear algebra solver library were both successfully found and exposed by Conan so that CMake can make use of them.

Let’s get started and make dependency management an issue of the past!

Creating a Conan recipe for our mesh reader library

This section will walk you through how to change the mesh reading library we have prepared in the previous article so that we can use Conan not just to handle our required libraries for this project but also to expose it to other projects that may want to use it. There is a rather nice summary on how to do that over in the Conan documentation (which I am heavily leaning on in my write-up). Feel free to consult it for some more in-depth discussions; it is really well-written and worth a read!

I am assuming you have Conan already installed, if not, you can read my write-up on how to get started with Conan, including the default profile you have to create before doing anything useful with Conan.

Creating a new Conan package from scratch

The first thing we need to do is to create a starter project from scratch. Conan does provide us with a handy command that we can use:

conan new cmake_lib -d name=meshReaderLib -d version=1.0

Here, conan new states that we want to create a new project, and the first argument to this command is the template we want to use. In our case, we want to create a CMake-based library, so our template is cmake_lib. Have a look at the conan new command for all available templates. We provide two additional inputs: the name and version of the project. Once we have executed that command, we will create the following folder structure:

root
├── include/
│   └── meshReaderLib.h
├── src/
│   └── meshReaderLib.cpp
├── test_package/
│   ├── src/
│   │   └── example.cpp
│   ├── CMakeLists.txt
│   └── conanfile.py
├── CMakeLists.txt
└── conanfile.py

We could go ahead and compile this project now using CMake, i.e. there is sensible content in each of these generated files. It is intended to be used to develop a library from scratch. However, we have already developed the library so we only need some parts of this generated project.

In particular, we are interested in the conanfile.py and the test_package/ folder. You have probably noticed that the conanfile.py shared close resemblance to the previously used conanfile.txt. Indeed, both of these are very similar. While the conanfile.txt is very rigid (we can only declare dependencies globally), the conanfile.py is much more flexible and allows, for example, to handle conditional dependencies based on the settings and platform. It also allows additional steps to package our own library and make it available to others!

Suffice it to say that if we want to export our own library to other users, we need to use the more flexible conanfile.py, and the file we have generated using conan new already has most of the boilerplate code inserted; all we have to do is provide the required dependencies and package information. The Conan documentation explains the details of the conanfile.txt and conanfile.py rather well, have a look if you are interested to learn more.

Project structure

In this section, I want to look at our modified project structure. We are not going through all the different folders again; we did have a deep-dive into them in the previous article. I want to look at the changes only that we made, and this is shown in the modified project structure below (where the content of the folders we have looked at previously is left out and replaced by ...).

root
├── build/
├── cmake/
│   └── ...
├── docs/
│   └── ...
├── mesh/
│   └── ...
├── meshReaderLib/
│   └── ...
├── test_package/
│   ├── src/
│   │   └── example.cpp
│   ├── CMakeLists.txt
│   ├── conanfile.py
│   └── structured2D.cgns
├── tests/
│   └── ...
├── ...
└── conanfile.py

I have simply copied the test_package/ folder into our mesh reading library folder, as well as the conanfile.py. You will later see that if you have both a conanfile.py and conanfile.txt in your project, you’ll confuse Conan, and it will abort. You can safely remove the conanfile.txt at this point, as we will no longer need it.

One additional file I have copied into the test_package/ folder is the structured2D.cgns file which is coming from the mesh/ directory. This will be used to test that we can open a mesh file, as you will see when we test the generated Conan package. For now, though, let us have a look at the conanfile.py, which is the main focus of this article.

Creating the required instructions for Conan: conanfile.py

If you have never used Python before, then you may want to read up on, at least, the basics of Python. Here is a 4-minute introduction, which you can supplement with this more advanced crash course, which also covers object-orientated concepts, i.e. classes. Well, I probably need my own series on Python for high-performance computing, don’t I? It is on my to-do list …

If you feel ready to be exposed to some Python code, let’s have a look at the conanfile.py in the top-level directory, and then go through it afterwards, section by section:

from conan import ConanFile
from conan.tools.cmake import CMakeToolchain, CMake, cmake_layout, CMakeDeps


class meshReaderLibRecipe(ConanFile):
    name = "meshreaderlib"
    version = "1.0"
    package_type = "library"

    # Optional metadata
    license = "MIT"
    author = "Tom-Robin Teschner"
    url = "https://cfd.university"
    description = "A mesh reading library for multi-block structured and unstructured grids using the CGNS file format."
    topics = ("cfd", "mesh", "grids", "cgns", "structured", "unstructured")

    # Binary configuration
    settings = "os", "compiler", "build_type", "arch"
    options = {
        "shared": [True, False],
        "fPIC": [True, False],
        "with_gtest": [True, False],
        "with_doxygen": [True, False]
    }
    default_options = {
        "shared": False,
        "fPIC": True,
        "with_gtest": True,
        "with_doxygen": False
    }

    # Sources are located in the same place as this recipe, copy them to the recipe
    exports_sources = "CMakeLists.txt", "cmake/*", "docs/*", "tests/*", "mesh/*", "meshReaderLib/*"

    def config_options(self):
        if self.settings.os == "Windows":
            self.options.rm_safe("fPIC")

    def configure(self):
        if self.options.shared:
            self.options.rm_safe("fPIC")

    def layout(self):
        cmake_layout(self)

    def requirements(self):
        self.requires("cgns/4.3.0", transitive_headers=True)
        if self.options.with_gtest:
            self.requires("gtest/1.14.0", transitive_headers=True)
    
    def build_requirements(self):        
        if self.options.with_doxygen:
            self.requires("doxygen/1.9.4")

    def generate(self):
        deps = CMakeDeps(self)
        deps.generate()
        tc = CMakeToolchain(self)
        tc.variables["ENABLE_SHARED"] = self.options.shared
        tc.variables["ENABLE_TESTS"] = self.options.with_gtest
        tc.variables["BUILD_DOCS"] = self.options.with_doxygen
        tc.generate()

    def build(self):
        cmake = CMake(self)
        cmake.configure()
        cmake.build()

    def package(self):
        cmake = CMake(self)
        cmake.install()

    def package_info(self):
        # set global project settings
        self.cpp_info.set_property("cmake_file_name", "meshReaderLib")
        self.cpp_info.set_property("cmake_target_name", "meshReaderLib::meshReaderLib")

        # handle Window's amazing naming convention for debug libraries
        if self.settings.os == "Windows" and self.settings.build_type == "Debug":
            self.cpp_info.libs = ["meshReaderLibd"]
        else:
            self.cpp_info.libs = ["meshReaderLib"]

        # add pre-processor definitions to resolve __declspec()s correctly on Windows for shared libraries
        if self.settings.os == "Windows":
            if self.options.shared:
                self.cpp_info.defines = ["COMPILEDLL"]
            else:
                self.cpp_info.defines = ["COMPILELIB"]

        # link against dependencies
        self.cpp_info.requires = ["cgns::cgns"]

        if self.options.with_gtest:
            self.cpp_info.requires.append("gtest::gtest")

Import statements, class declaration, and class variables

First, we provide a few import statements, which are equivalent to #include "file-name.hpp" and linking to a library in C++ at the same time, and then declare our meshReaderLibRecipe class, which inherits from the ConanFile class:

from conan import ConanFile
from conan.tools.cmake import CMakeToolchain, CMake, cmake_layout, CMakeDeps


class meshReaderLibRecipe(ConanFile):

This is followed by some public attributes (variables) in our class, that define things like the name and version of the library, the license, author, URL for additional information, as well as a description and a list of topics that this library falls in to.

    name = "meshreaderlib"
    version = "1.0"
    package_type = "library"

    # Optional metadata
    license = "MIT"
    author = "Tom-Robin Teschner"
    url = "https://cfd.university"
    description = "A mesh reading library for multi-block structured and unstructured grids using the CGNS file format."
    topics = ("cfd", "mesh", "grids", "cgns", "structured", "unstructured")

Next, we specify some settings and options. We have seen them, to some extend, already before but here is a more detailed differentiation: Settings are platform specific. They can change for different users (i.e. they may have different operating systems) but once set for a project, they will not change (i.e. you would not change the operating system or architecture during your project).

Options, on the other hand, are project-specific and can change during the compilation. These are things like building a shared (dynamic) or static library, enabling tests and documentation, and so on. These are the settings and options specified in our project:

    # Binary configuration
    settings = "os", "compiler", "build_type", "arch"
    options = {
        "shared": [True, False],
        "fPIC": [True, False],
        "with_gtest": [True, False],
        "with_doxygen": [True, False]
    }
    default_options = {
        "shared": False,
        "fPIC": True,
        "with_gtest": False,
        "with_doxygen": False
    }

This is generated by the conan new command, the only additions are the with_gtest and with_doxygen variables that I have added. We see that we specify the available options in the options dictionary (which is equivalent to a C++ std::map), and then we set the default options in the default_options dictionary, i.e. these will be used if we do not specify them on the command line.

I have set with_doxygen to False by default, as it will break the build process; let me elaborate: The CGNS library is linked against the mesh reader library, and we need it, so it is not an option. GTest, however, is only required if we want to build the tests (and it is typical not to install them at this stage where we provide the library to our users through Conan; we’d hope that it has been thoroughly tested and it shouldn’t be the responsibility of our users to test everything). So we set this to False by default (but setting it to True will work just fine).

Doxygen, however, is not a library but rather an executable. When we had our conanfile.txt before, we would list Doxygen as a tool instead of a library, and we had to execute the conanbuild.ps1 or conanbuild.sh to discover the executable. This was working fine; however, when we create our mesh reading library with the conanfile.py that we have just created, then Conan will set its own environment in which Doxygen is not loaded. Subsequently, the Doxygen executable will never be found, and the entire package will not install.

This seems to be an issue with Conan and we can’t simply fix it within our recipe. It would be ridiculous not to be able to install the library if we can’t find Doxygen, which is not an essential part of the build process anyway. We could change our CMake file to only build the documentation if Doxygen was found, but that means we set with_doxygen to True in our options, we think the documentation will be generated and then we can’t find it.

So, the solution to this problem is to install Doxygen systemwide (use your favourite package manager, i.e. winget, brew, apt, etc.). If Doxygen can be discovered somewhere on your system, you’ll be able to build the documentation again and can set with_doxygen to True, otherwise, leave it to False and develop a minute grudge over Conan.

Declaring required source files

Moving on, we have to declare the sources that we need to build this project. It is similar to the install step in CMake, where we have to specify which header files, executables or libraries, and additional files (e.g. documentation) we need to build this project from source successfully. Think about it this way: If you were to copy this project somewhere else to build it (this is what Conan will do), which files (and folders) do you need to copy to ensure you can compile everything from scratch? All these files go into the exports_sources variable.

    # Sources are located in the same place as this recipe, copy them to the recipe
    exports_sources = "CMakeLists.txt", "cmake/*", "docs/*", "tests/*", "mesh/*", "meshReaderLib/*"

Conan is actually more powerful here, and under normal circumstances, you would likely not define the exports_sources variable at all, but instead provide a build() function that downloads the source code from a remote repository, which is then used instead. For example, if we wanted to provide our own Conan recipe for the CGNS library, we could have the following build() method:

from conan.tools.scm import Git

class cgnsRecipe(ConanFile):
    name = "CGNS"
    version = "4.4.0"

    ...

    def source(self):
        git = Git(self)
        git.clone(url="https://github.com/CGNS/CGNS.git", target=".")
        git.checkout("v4.4.0")

    ...

Here, we are telling Conan to use Git and which repository we want to clone (download), where we want to download it to (indicated by the target, i.e. here into the current folder indicated by the dot (.)), as well as which tag or commit in particular we want to check out and use. Then, we can proceed as normal.

One last thing about getting sources: Conan also provides support to extend the recipe easily for multiple versions of the library, in this way, you can provide a range of versions that your (Conan) users can make use of in their project. For that, we need to provide a conandata.yml file, where the extension *.yml stands for YAML (yet another markup language), and it is very similar to a JSON file that we have seen previously. And if you don’t know what JSON files are, think: Configuration file.

For the CGNS library, we may have the following conandata.yml file (which will be located in the same folder as the conanfile.py):

sources:
  "4.3.0":
    url: "https://github.com/CGNS/CGNS/archive/refs/tags/v4.3.0.zip"
    sha256: "7c8fbd44f3efa99d83b87f009df50813a04491e6e89d3bfef110c5dc27cf1623"
    strip_root: true
  "4.4.0":
    url: "https://github.com/CGNS/CGNS/archive/refs/tags/v4.4.0.zip"
    sha256: "72ad499936089102c1d77e3cdd78a623d61d1c6ad94c4df3b23f36db9ddfad6a"
    strip_root: true

Now, we can make use of this in our build() function as follows:

...

class cgnsRecipe(ConanFile):
    name = "CGNS"
    version = "4.4.0"
    
    ...

def source(self):
    data = self.conan_data["sources"][self.version]
    get(self, data["url"], sha256=data["sha256"], strip_root=data["strip_root"])
    
    ...

Conan will now look specifically for an entry titled 4.4.0 within the sources entry in the conandata.yml file and get the required download information this way. It is handy if you want to keep your recipe unmodified (apart from changing the version) and then have a separate file that allows you to add additional versions as you need. You can even further shortcut this with the export_conandata_patches() function, which will essentially replace lines 10-11 above and look for the right version automatically. This is what the CGNS recipe is doing.

This was a crash course in handling sources with Conan and mainly intended to show you what we can do here, as our exports_sources variable makes sense in our case, but most likely, you will see a link to a repository within a build() method instead. If you want to take a closer look, feel free to look up the source handling section in the Conan documentation.

Information for the configuration step

There are a few functions we use to help with the project configuration. First, we have the config_options() and configure() functions, which may seem very similar. Looking at the relevant section in the Conan documentation, we see that config_options() is used to look at the available project options and remove any that may not make sense for the current settings while configure() is used to, well, configure the remaining ones. In our case, we leave the defaults for both options provided to us by Conan:

    def config_options(self):
        if self.settings.os == "Windows":
            self.options.rm_safe("fPIC")

    def configure(self):
        if self.options.shared:
            self.options.rm_safe("fPIC")

We specify the layout of our build folder using the layout() function. This will determine how the generated files will be written to disk (i.e. what folder structure to apply). Since we are using CMake, Conan will automatically dump all files within a build/ folder, as this is what CMake is expecting (if we specified Meson, for example, this would be builddir/ as this is Meson’s default). The internal structure within the build/ folder will also be adjusted, instead of dumping everything into the build/ folder itself.

    def layout(self):
        cmake_layout(self)

As a side note, within our conanfile.txt file, we can also specify a section called [layout] and then pass cmake_layout as the layout option, which will achieve the same. I have not used that in any of our previous Conan examples, but you will see this popping up in Conan central if you look for example usages. It is not required, but, well, most people will expect this layout if they are already using Conan and other CMake-based projects, so let’s be kind to them and give them what they expect.

Finally, we need to tell our project what dependencies we have, and we can split these into the requirements() and build_requirements() functions. The difference here is that the requirements() function lists all dependencies that we would list under the [requires] section in our conanfile.txt (i.e. libraries), while the build_requirements() function lists all dependencies we need to build our project, which would go under [tool_requires] (i.e. executables).

    def requirements(self):
        self.requires("cgns/4.3.0", transitive_headers=True)
        if self.options.with_gtest:
            self.requires("gtest/1.14.0", transitive_headers=True)
    
    def build_requirements(self):        
        if self.options.with_doxygen:
            self.requires("doxygen/1.9.4")

Since we can decide if we want to use GTest and Doxygen in our build, we first check if this option was set to true and only then do we require it in our build. Hmm, what on earth are transitive_headers I hear you ask? Don’t get me started. That was probably a good two days of debugging Conan and CMake.

In a nutshell, when you build a project that depends on a library (in our case, the mesh reader library we are developing depends on the CGNS library), then Conan will forward all libraries (e.g. cgns.lib or libcgns.a) to our mesh reader library, but not their header files. So when we have an #include "cgnslib.h" statement somewhere in our code, CMake does not know where to find this file as Conan refuses to provide that information (but it has no problem telling CMake where to find the library and how to link against it!)

To overcome this issue, we can set the transitive_header variable to True, which, to someone at Conan, made sense. I learned about this from a bug report on Conan’s Git repository. Yes, someone pointed out that they have found a bug in Conan, and the developer got back and essentially replied “it’s not a bug, it’s a feature!” (and changed the bug report to a question …). Great, Conan is expecting requirements to break as soon as you use them unless you change the default behaviour! Software engineering at its best!

We don’t need to forward any headers for Doxygen, as we only work with its executable, which already contains all the information to run automatically without dependencies.

Finally, the configuration step is brought to a close with the generate() function, which, in our case, will create the CMake toolchain file that will contain all the locations of our libraries in our local Conan cache, as well as any project-specific variables we need to set in our CMake environment. This function is shown below:

    def generate(self):
        deps = CMakeDeps(self)
        deps.generate()
        tc = CMakeToolchain(self)
        tc.variables["ENABLE_SHARED"] = self.options.shared
        tc.variables["ENABLE_TESTS"] = self.options.with_gtest
        tc.variables["BUILD_DOCS"] = self.options.with_doxygen
        tc.generate()

The CMakeDeps generator will create all required files for CMake to find the required dependencies (these are the <package-name>Config.cmake files we wrote manually during our install article). This allows CMake to know where to find specific libraries, and we can simply write find_package(<library-name> REQUIRED) within our CMake script, and it will know where to find them.

The CMake toolchain part from lines 4-8 is concerned with setting project options based on our settings in the conanfile.py. Since we can specify if we want to have Gtest or Doxygen as a requirement to build our library, we can pass this information to CMake, as it would make no sense to build tests if we did not require GTest to be available. This also means that we only need to pass the toolchain file to CMake, and we no longer have to manually set the required options. Quite handy!

So, for example, in the previous article, we saw that to configure our project, we had the following command:

cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_TOOLCHAIN_FILE="conan_toolchain.cmake" -DBUILD_DOCS=ON -DENABLE_SHARED=ON -DENABLE_TESTS=ON -G Ninja ..

Since we specify the options now in the toolchain, we can now simply write

cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_TOOLCHAIN_FILE="conan_toolchain.cmake" -G Ninja ..

Ahhh, Conan, I forgive you for transitive_header mess! This is looking much nicer, neater, and cleaner! (though, since we used a CMake layout now, the toolchain file will be located elsewhere, which we will see in a second).

Building our library with Conan

The next step, after having configured our project successfully, is the build step. This is fairly straightforward, as it was with CMake itself when we manually built projects ourselves. This is the build() function in our Conan recipe:

    def build(self):
        cmake = CMake(self)
        cmake.configure()
        cmake.build()

We see that we configure and build our project in the same function, though all of the heavy lifting was done in the previous functions, which means we can simply call configure() here without any arguments and it will know how to configure our CMake project. The same goes for the build() methods.

Package and install our library into our local Conan cache

After we have built our library, we can install it simply with

    def package(self):
        cmake = CMake(self)
        cmake.install()

This is similar to manually installing targets with CMake, i.e. a simple call to CMake with the --install flag is sufficient to install the project. We have not specified the installation prefix, i.e. the location where Conan should install this library. This is handled automatically by Conan internally through some clever hash calculation that will influence the installation location.

Just to give you a bit of a background, all of our project settings will be used to compute a hash. This means that if we try to compile the library again with the same settings, the computed hash will be the same. Conan makes use of that by installing all of its builds to a specific folder, which includes the hash. If you need the same library in the future and use the same settings, Conan does not need to recompile the library; instead, it reuses the already compiled version. This also allows you to have the same library installed multiple times for different settings.

The last bits of information we have to provide are located in the package_info() function. This function collects the name of the library that should be used by CMake to find this dependency later, the namespace, library name, and so on. Even though all of that information is already provided in our CMake scripts, we have to duplicate it here (which allows us to provide missing information in case we write a recipe for a third-party library where we can’t modify the CMake scripts). Have a look through this function, and we will look at it afterwards:

    def package_info(self):
        # set global project settings
        self.cpp_info.set_property("cmake_file_name", "meshReaderLib")
        self.cpp_info.set_property("cmake_target_name", "meshReaderLib::meshReaderLib")

        # handle Window's amazing naming convention for debug libraries
        if self.settings.os == "Windows" and self.settings.build_type == "Debug":
            self.cpp_info.libs = ["meshReaderLibd"]
        else:
            self.cpp_info.libs = ["meshReaderLib"]

        # add pre-processor definitions to resolve __declspec()s correctly on Windows for shared libraries
        if self.settings.os == "Windows":
            if self.options.shared:
                self.cpp_info.defines = ["COMPILEDLL"]
            else:
                self.cpp_info.defines = ["COMPILELIB"]

        # link against dependencies
        self.cpp_info.requires = ["cgns::cgns"]

        if self.options.with_gtest:
            self.cpp_info.requires.append("gtest::gtest")

Lines 3-4 specify the name of our library (used in the find_package() function), as well as the namespace to use (used in the target_link_libraries() function. Lines 6-10 take care of Window’s naming convention for debug libraries, where the letter d is suffixed to the library name, and then more Windows stuff happens on lines 12-17, where we add the required pre-processor statements to the compiler so that we are correctly exporting our library (with the required __declspec() directives for shared (dynamic) libraries).

Line 20 tells Conan that the CGNS library is a dependency. So, if we link against our mesh reader library, Conan also needs to link against the CGNS library. It will do so for us in the background, without us knowing or seeing it (and, hopefully, it will not forget to also pass on information on where to find the required header files!). We do so for GTest as well on lines 22-23, but only if we requested to build the tests.

Creating a test for Conan

Right, that was quite a journey. We looked through the entire conanfile.py and saw how to prepare the mesh reading library so that Conan can compile and install it automatically, as well as forward any required information. Before we release this, though, we probably want to test it first and make sure that Conan can correctly do all of these things we want it to do. This is where the test_package folder comes in.

In Conan, it is customary to provide a test_package folder (using the exact same name as Conan will look for this explicitly). Here, we want to write a small test program that will exercise our library to make sure it can be used. It is similar to a unit test, but we are not testing the library here; rather, we are testing that Conan was able to build and install the library correctly.

As a reminder, this is the structure of our test_package folder:

root
├── ...
├── test_package/
│   ├── src/
│   │   └── example.cpp
│   ├── CMakeLists.txt
│   ├── conanfile.py
│   └── structured2D.cgns
└── ...

Let’s look through these files in the next section.

Creating a C++ test: example.cpp

The first file is located at test_package/src/example.cpp and provides a source file that we can fill however we see fit to exercise the library sufficiently so that we are confident everything is working correctly. In our case, the library is so small that simply reading a mesh file for either a structured or unstructured grid is sufficient. This is shown below:

#include <vector>
#include <string>

#include "meshReaderLib/meshReader.hpp"

int main() {
  ReadStructuredMesh structuredMesh("structured2D.cgns");
  structuredMesh.readMesh();
  return 0;
}

The importance here is that we can include our mesh reading library correctly and read a mesh without problems. If we can, then the rest of the library will likely also work correctly.

Build the C++ test: CMakeLists.txt

The CMake file is very simplistic. We set our boilerplate cmake_minimum_required() and project() information, and then require that the meshReaderLib library is available through the find_package() command. We then create an executable and link it against the meshReaderLib library. Since we are also using a mesh file here, we copy it next to the executable so we can read it once it is executed. This is shown in the CMakeLists.txt file below, located within the test_package/ folder:

cmake_minimum_required(VERSION 3.15)
project(PackageTest CXX)

find_package(meshReaderLib CONFIG REQUIRED)

add_executable(example src/example.cpp)
target_link_libraries(example meshReaderLib::meshReaderLib)

add_custom_command(TARGET example POST_BUILD
  COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_SOURCE_DIR}/structured2D.cgns $<TARGET_FILE_DIR:example>/structured2D.cgns
)

Handling dependencies: conanfile.py

Our test project also has its own conanfile.py file, but this is typically just a boilerplate file, which we don’t have to edit. You will see that within the requirements() function shown on line 12 below, we say that we require the self.tested_reference_str, which will forward any packages that are required (including the mesh reading library itself) from the main conanfile.py.

import os

from conan import ConanFile
from conan.tools.cmake import CMake, cmake_layout
from conan.tools.build import can_run


class meshReaderLibTestConan(ConanFile):
    settings = "os", "compiler", "build_type", "arch"
    generators = "CMakeDeps", "CMakeToolchain"

    def requirements(self):
        self.requires(self.tested_reference_str)

    def build(self):
        cmake = CMake(self)
        cmake.configure()
        cmake.build()

    def layout(self):
        cmake_layout(self)

    def test(self):
        if can_run(self):
            cmd = os.path.join(self.cpp.build.bindir, "example")
            self.run(cmd, env="conanrun")

The only addition here is the test() function, which essentially just executes some command line instructions to run our executable. If everything works correctly, our example.cpp file will return 0; at the end of the file, which Conan captures to verify that the code is executed correctly. If it didn’t, some other return code would be given, and Conan would raise an error.

Command line instructions to create the package

Ok, so let’s turn our attention now to how we can create the Conan package for our mesh reading library and store it in our local Conan cache. But first, let’s consider our approach. If we want to export our library, we typically want to share a Release version of our library, not a Debug version. We can steer this with our Conan profile. We will have, at a minimum, a default profile, which is typically set up automatically by Conan in such a way that we will default to Release builds. If we don’t specify any profile at all, then we will default to that.

So, let’s try to build then a Release version of our library. This is achieved by running the following command:

conan create . --build=missing

Alternatively, we can specify the profile we want to use as we did in previous articles where we used Conan using the -pr flag, such as:

conan create . -pr dev --build=missing

Both of these commands will execute the instructions encoded in our conanfile.py and compile our library against all of our dependencies, and make it available for easy consumption. There are a few pitfalls here, which may result in your package not to be created. Most of the time it has to do with user privileges, i.e. you are not allowed to execute scripts on your PC (see my write-up on how to circumvent this issue on Windows in particular).

Sometimes, it may just be a particular setting or system configuration that lets you down (e.g., the wrong C++ standard set in the Conan profile or not an up-to-date version of your compiler, CMake, etc.). These are difficult to predict, but I have seen Conan fail to work with a Debug profile while a release version is working just fine. It is not a perfect tool, but it is better than not using any tool at all.

Once we have created the package and hopefully did not get any error message, we are done. Part of the conan create command is to look into the test_package folder and try to compile and run the test_package/srx/example.cpp file, which should exercise our library. If that worked without any issue, then we can assume that our library has been compiled and installed correctly.

If you look at the output Conan generates, you can see the various different sections it goes through:

======== Input profiles ========
...
======== Computing dependency graph ========
...
======== Computing necessary packages ========
...
======== Installing packages ========
...
======== Launching test_package ========
...
======== Computing dependency graph ========
...
======== Computing necessary packages ========
...
======== Installing packages ========
...
======== Testing the package ========
...
======== Testing the package: Executing test ========
meshreaderlib/1.0 (test package): Running test()
meshreaderlib/1.0 (test package): RUN: .\example

The first four sections deal with the mesh reading library itself, i.e. gathering input files (determining the profile and deriving inputs from it), looking at dependencies and which need to be built from the source, if any, and then installing those packages, including the generated mesh reading library. We then go into the Launching test_package section where we do the same steps, only this time for our mini-project within the test_package/ folder.

As long as this step is executed without any issues, we should have confidence that our library was correctly compiled and stored within our local Conan cache.

I really want others to use my amazing library, what should I do?

Well, it sounds like you have an amazing piece of work lined up. Congratulations! If you can’t wait to share your library with the rest of the world, you can upload it to Conan central. This is rather straightforward; you can do all of that with the conan upload command. Since the mesh reader library is not intended for widespread use (at least not yet), I have no intention of putting it on Conan central, but if you want to publish your own library, I’d suggest having a look at the Conan documentation for uploading packages.

Notes on the linear algebra library changes

Similar to our last article, where we added CMake to both the mesh reader and linear algebra solver library, there were a lot of commonalities, and that is the case for the conanfile.py and its associated test project as well. In fact, the conanfile.py of the linear algebra solver library is pretty much a search and replace exercise for replacing the library name in various places (and removing the CGNS dependence). The biggest difference is in the test_package/srx/example.cpp file, which now tests the linear algebra solver library instead.

#include <vector>
#include <string>

#include "linearAlgebraLib/linearAlgebraLib.hpp"

int main() {
  linearAlgebraLib::SparseMatrixCSR matrix(3, 3);
  return 0;
}

Apart from that, you won’t find any other differences and you can treat both libraries the same way when it comes to installing them into your local Conan cache.

Consuming the newly Conan-based libraries in our test project

In our previous article, we looked at how we can use our now CMake-based mesh reader and linear algebra solver library within another project. The whole motivation for the current article was to replace the competing dependency management, where we had to tell CMake where to find our own libraries, while Conan was handling the external dependencies on the CGNS and GTest libraries (as well as Doxygen …).

Since everything lives now in the Conan ecosystem, we do no longer need to tell CMake where to find dependencies manually, and Conan will resolve all of that for us through the toolchain file it generates. Let’s see how we can achieve that next.

Changes to the test project

As a brief reminder, the project structure we developed for the test project is given below:

root
├── build/
├── mesh/
│   └── structured2D.cgns
├── CMakeLists.txt
├── conanfile.txt
└── main.cpp

While there are no changes to the structure itself, we have to change the conanfile.txt file. In the previous article, I showed you that this file had the following content:

[requires]
cgns/4.3.0

[generators]
CMakeDeps
CMakeToolchain

We explicitly required the CGNS library to be available, even though our project only depended on the mesh reader and linear algebra solver library. It is somewhat frustrating (but unavoidable!) to have to handle the dependencies of our dependencies, so we used Conan to make the CGNS library available so that our mesh reader library could function properly.

With our mesh reader library now available within the Conan cache, it can find dependencies automatically, which means that our conanfile.txt now becomes:

[requires]
meshreaderlib/1.0
linearalgebralib/1.0

[generators]
CMakeDeps
CMakeToolchain

How beautiful is that? We only list requirements that we actually have for our project, and all the dependencies that are required for these libraries are resolved in the background. No need for us to get involved. Yes, all the changes we have made thus far in this article are to change the conanfile.txt as shown above. But of course, we now are all Conan wizards and can use this skillset to develop and share our own libraries with the world!

Command line instructions

Let’s have a look at the beautified command line instructions. They are given below for configuring and building the project:

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE="conan_toolchain.cmake" -G Ninja ..
cmake --build . --config Release

No more -DCMAKE_PREFIX_PATH, just a build type, generator, and a toolchain, which will resolve all dependencies for us. If you have worked with CMake for a while, this feels a bit like magic!

We can test that the libraries were correctly found and linked against our executables using

.\testLibs.exe

on Windows or

./testLibs

on UNIX (Linux, macOS). But of course, we are already expecting our test program to compile and execute, as we have already tested that with the test_packages provided in both libraries. Congratulations, you have reached the end, not just of this article, but also of this series!

Summary

This concludes our series on build automation with CMake. In this article, we finalised our skillset by looking at how we can make our own libraries available through Conan. While we did not go through uploading our package to Conan central, we saw that this is just a few steps away and we could achieve this with conan upload relatively easily.

Dependency management is a difficult topic, and it is entirely possible that despite testing the code developed in this article on two separate machines and 3 different operating systems, you still will run into issues. I could probably talk about this issue alone for an entire article (or series on containerisation, e.g. Docker or Kubernetes). You will find that most of the time in this series, we have been looking at how to resolve external dependencies, and it is such an important topic that receives little attention (in the CFD community) that I think it is worth writing about.

Of course, automatisation is only one aspect, but we are CFD engineers, and we are interested in applying this to real CFD problems. We have re-used our mesh reader and linear algebra solver library to showcase how we can use Conan with our CFD libraries. I hope you are starting to see a picture here: Developing the library is one thing, but if we have ambitions for anyone using our work (or, indeed, getting paid to work on a CFD solver), then we need to know how to isolate our work, put it into a library and share it with colleagues and users.

CMake and Conan provide us with tools that put our CFD development on steroids, and we should use these tools to make our lives as developers easier. The educated user will thank you for it and this means that compiling, installing, and using your work will be easy, and perhaps even an enjoyable exercise, especially if you are used to installing software like OpenFOAM, which requires a good amount of nerves!

---

Tom-Robin Teschner is a senior lecturer in computational fluid dynamics and course director for the MSc in computational fluid dynamics and the MSc in aerospace computational engineering at Cranfield University.