TIP

This blog post is also available on the TiKV blog and the CNCF blog.

Update

Based on this blog post, pprof now ships with a custom profiler for Criterion since version 0.4.2! Make sure to check out this example and enable "criterion" and "flamegraph" crate features.

Introduction

Criterion is a well-known and often-used benchmarking framework in the Rust ecosystem. If you’ve not yet heard of it, definitely check it out!

Since v0.3, Criterion supports in-process profiling hooks. They allow us to use a custom profiler while running benchmarks. After registering a custom profiler, we can enable them by running the benchmark suite with the --profile-time flag.

Having hooks for a custom profiler for your benchmarks allows you to do awesome stuff, for example generating flamegraphs for every benchmark with pprof.

Background

One question I often ask myself when looking at benchmarks is:

“These numbers are great. But how can we do this faster? Where do we spend the most time?”

This question can often be answered by looking at flamegraphs.

Previously, I got my hands dirty using cargo-flamegraph, which is a really good tool on its own. But the problem is that when using cargo-flamegraph with benchmarks, you will usually get some unnecessary clutter, e.g. the setup routine for your benchmark and the enclosing framework code from Criterion.

And here comes another cool Rust library into play: pprof with the flamegraph feature!

pprof is a CPU profiler that can be used to profile specific parts of your program on Linux. The best of all: It even supports generating flamegraphs with the feature flag flamegraph.

Let’s directly dive in and create a pprof integration for Criterion!

Setup

We will now create a custom profiler for criterion with pprof that will print flamegraphs for each benchmark.

First, add the following dependencies to your Cargo.toml:

[dev-dependencies]
pprof = { version = "0.3", features = ["flamegraph"] }
criterion = "0.3"
# criterion-macro = "0.3"  # optional, if you use custom test frameworks

Now create a new file in your benches/ folder. Let’s call it perf.rs. Here we will implement our new FlamegraphProfiler.

The interface for implementing a custom profiler is actually quite straight-forward. We only need to implement the criterion::profiler::Profiler trait from Criterion and we are good to go.

use std::{fs::File, os::raw::c_int, path::Path};

use criterion::profiler::Profiler;
use pprof::ProfilerGuard;


/// Small custom profiler that can be used with Criterion to create a flamegraph for benchmarks.
/// Also see [the Criterion documentation on this][custom-profiler].
///
/// ## Example on how to enable the custom profiler:
///
/// ```
/// mod perf;
/// use perf::FlamegraphProfiler;
///
/// fn fibonacci_profiled(criterion: &mut Criterion) {
///     // Use the criterion struct as normal here.
/// }
///
/// fn custom() -> Criterion {
///     Criterion::default().with_profiler(FlamegraphProfiler::new())
/// }
///
/// criterion_group! {
///     name = benches;
///     config = custom();
///     targets = fibonacci_profiled
/// }
/// ```
///
/// The neat thing about this is that it will sample _only_ the benchmark, and not other stuff like
/// the setup process.
///
/// Further, it will only kick in if `--profile-time <time>` is passed to the benchmark binary.
/// A flamegraph will be created for each individual benchmark in its report directory under
/// `profile/flamegraph.svg`.
///
/// [custom-profiler]: https://bheisler.github.io/criterion.rs/book/user_guide/profiling.html#implementing-in-process-profiling-hooks
pub struct FlamegraphProfiler<'a> {
    frequency: c_int,
    active_profiler: Option<ProfilerGuard<'a>>,
}

impl<'a> FlamegraphProfiler<'a> {
    #[allow(dead_code)]
    pub fn new(frequency: c_int) -> Self {
        FlamegraphProfiler {
            frequency,
            active_profiler: None,
        }
    }
}

impl<'a> Profiler for FlamegraphProfiler<'a> {
    fn start_profiling(&mut self, _benchmark_id: &str, _benchmark_dir: &Path) {
        self.active_profiler = Some(ProfilerGuard::new(self.frequency).unwrap());
    }

    fn stop_profiling(&mut self, _benchmark_id: &str, benchmark_dir: &Path) {
        std::fs::create_dir_all(benchmark_dir).unwrap();
        let flamegraph_path = benchmark_dir.join("flamegraph.svg");
        let flamegraph_file = File::create(&flamegraph_path)
            .expect("File system error while creating flamegraph.svg");
        if let Some(profiler) = self.active_profiler.take() {
            profiler
                .report()
                .build()
                .unwrap()
                .flamegraph(flamegraph_file)
                .expect("Error writing flamegraph");
        }
    }
}

To use the new profiler, we need to register it with Criterion. See also Criterion’s advanced configuration.

If you are using the custom test framework feature with criterion-macro, you can configure it with:

#![feature(custom_test_frameworks)]
#![test_runner(criterion::runner)]

use criterion::{Criterion, black_box};
use criterion_macro::criterion;

mod perf;


fn fibonacci(n: u64) -> u64 {
    match n {
        0 | 1 => 1,
        n => fibonacci(n - 1) + fibonacci(n - 2),
    }
}


fn custom_criterion() -> Criterion {
    Criterion::default().with_profiler(perf::FlamegraphProfiler::new(100))
}

#[criterion(custom_criterion())]
fn bench_custom(c: &mut Criterion) {
    c.bench_function("Fibonacci-Custom", |b| b.iter(|| fibonacci(black_box(20))));
}

Else, you can do it like:

mod perf;

criterion_group!{
    name = benches;
    // This can be any expression that returns a `Criterion` object.
    config = Criterion::default().with_profiler(perf::FlamegraphProfiler::new(100);
    targets = bench
}

Now that we have set up our custom profiler we can actually use it for our benchmarks as described in the next section.

Running the Benchmarks and Getting Results

Enable Performance Profiling for Unprivileged Users

To enable performance profiling without running the benchmarks as root, you may need to adjust the value of perf_event_paranoid in the Linux kernel to an appropriate value for your environment. The most permissive value is -1.

echo -1 | sudo tee /proc/sys/kernel/perf_event_paranoid

Running the Benchmarks

Now that everything is set up, we can run our benchmark with

cargo bench --bench my_bench -- --profile-time=5

This command will run the benchmarks in benches/my_bench.rs for 5 seconds with the use of our custom FlamegraphProfiler.

Note: You need to specify the name of your benchmark because otherwise you might get the error Unrecognized option: 'profile-time'. See also cargo bench gives “Unrecognized Option” errors for valid command-line options.

Viewing the Flamegraph

We will now find a file called flamegraph.svg in target/criterion/<name-of-benchmark>/profile/flamegraph.svg.

For the above example, we get the following result:

Cool feature. I really like it 😊

Recently, I was working on a benchmarking framework for block partitioning in databases (currently private, but will be released anytime soon) and I came across the problem of (de-)serializing user-defined derived classes of a base class in C++. While serializing is not too much of a problem (e.g. define an abstract function virtual void serialize(std::ostream& out) = 0 in the base class and implement the function in the derived classes), deserialization is much harder. Ultimately, I wanted every derived class to implement its own (de-)serialization functions for potentially complex types. But choosing the correct deserialization routine when reading a serialized file is tricky.

To borrow the example from the excellent ISO CPP article, here is some code that visualizes our problem:

We have an abstract base class Shape and a derived class Rectangle. Our goal is to correctly deserialize all derived classes of Shape by picking the correct concrete implementation. If we pick the wrong deserialization function (e.g. Circle instead of Rectangle), we will end up with garbage.

class Shape {
 public:
  static void serialize(std::ostream& out, Shape& shape) {
    shape.serialize(out);  // Forward to the derived implementation.
  }

  static std::unique_ptr<Shape> deserialize(std::istream& in) {
    // How should we know how to correctly deserialize the object?
    // In other words: which object do we even want to create from the stream??
  }

 protected:
  virtual void serialize(std::ostream& out) = 0;
};

class Rectangle : public Shape {
 public:
  void serialize(std::ostream& out) override {
    // Write properties of the current Rectangle to the output stream.
  }

  static std::unique_ptr<Shape> deserialize(std::istream& in) {
    // Create a new Rectangle from the input stream.
  }
 private:
  int64_t width_, height_;
};

Some notes on the example:

Generally speaking, this strongly resembles the “Factory Method Pattern”, but combines the abstract Factory/Creator and the Interface/Product into one class (Shape) and also combines the concrete implementations of RectangleFactory and the concrete Shape Rectangle into one class. It is left to the reader to figure out which function belongs to which actor in the factory pattern.

Further, I personally prefer two static methods serialize() and deserialize() in the Shape class so that both functions have a similar interface (although serialize() could also be non-static).

In the following sections we will gradually build a nice solution for this and discuss why our approach works. And for the busy reader, there will be a complete code example in the end 😘

Thoughts on a Solution

The general idea of the solution behind this problem is easily explained: When serializing, write some sort of “class identifier” to the file before the actual data of the object. This might be the name of the derived class or some sort of (user supplied) class identifier. When deserializing again, we first read the class identifier and pick the correct derived class based on that identifier. Then we call the deserialization function for this particular class, which will now handle the correct deserialization of the object.

A very good starting point is the extensive description from ISO CPP about serialization. It handles many caveats over serialization and also covers our current problem! However, our goal is to do this as nicely as possible.

Tip

If you are interested in how the same problem can be nicely solved in Rust, have a look at typetag from the developer of serde. This is a great solution and is very similar to our approach.

In the previous paragraph I mentioned that we “pick the correct serialization function”.

But how can a correct mapping between “class identifier” and the corresponding class be achieved?

Without loss of generality, we will pick the name of the derived class as a class identifier. Usually, there will be a static std::map<K, V> from class name to class prototypes of the derived class or function pointers that will take the input stream and produce an object of the correct derived class. I like the idea of functions more because then we don’t have “partly constructed objects only used to create a full object” laying around. The signature of the deserialization function’s pointer would look like std::unique_ptr<Shape> (*)(std::istream&).

For this, we can put a static std::map<std::string, Shape* (*)(std::istream&)> into the Shape class. To handle things nicely, we can alter the functions in the Shape class:

class Shape {
  // <snip>

  static void serialize(std::ostream& out, Shape& shape) {
    // Before we serialize the actual object,
    // write the class identifier to the output stream.
    const std::string& class_id = shape.class_identifier();
    out.write(class_id.c_str(), class_id.size() + 1 /*terminating null*/);

    shape.serialize(out);  // Forward to the derived implementation.
  }

  static std::unique_ptr<Shape> deserialize(std::istream& in) {
    // Read the class identifier.
    std::string class_id;
    std::getline(in, class_id, '\0');

    // Pick the correct deserialization function from the type registry
    // by looking at the class identifier.
    auto deserialize_func = type_registry[class_id];
    std::unique_ptr<Shape> derived_object = deserialize_func(in);
    return derived_object;
  }

  // Type registry which maps class identifiers to deserialization functions,
  // which produce an object of the corresponding derived class.
  static std::map<std::string, std::unique_ptr<Shape> (*)(std::istream&)> type_registry_;
};

Nice, this looks already quite good! Neither the serialization function nor the deserialization function of the derived class needs to know that they are part of this weird problem! 😄

While this is already quite nice, one big open question is: how do we reliably populate the type_registry_? Of course, we can do this per hand and write down every derived class… okay, just kidding 😆
As soon as someone implements a new derived class and forgets to alter the contents at a different location, we can’t read the file anymore.

We are dealing with a more general problem here. Remember when I said that the structure closely resembles the factory pattern? What we want is self-registering types in a factory!

Self-Registering Classes in a Factory

Luckily, there were people out there who already looked at this problem in the past. While this solution is largely based on the ideas of these two sources, I go one step further: Solve some problems from the previous approaches and also do it header-only! And you even get a complete and running example in the end for free! Excited? Let’s get started!

Okay, so the idea is to create a static bool register_type(...) function in the Shape class that can be called from the derived classes and adds one entry to the type_registry.

class Shape {
  // <snip>

  using deserialize_func = std::unique_ptr<Shape> (*)(std::istream&);

  static bool register_type(const std::string& class_id,
                            const deserialize_func factory_method) {
    if (type_registry_.find(class_id) != type_registry_.end()) {
      std::cerr << "Class ID " << class_id << " is already used." << std::endl;
      return false;
    }
    type_registry_.insert(std::make_pair(class_id, factory_method));
    return true;
  }

  static std::map<std::string, deserialize_func> type_registry_;
};

Nice, looks useful! But think about it: How does this even help? How should we call this function? In the constructor of the derived class? This won’t really help, because then types are only registered when at least one object has been created before.

The idea is to call the register_type() function from a static context inside the derived class! Why? Because then it will be called when static objects are initialized at program startup and this ensures that the function is called before any useful work is done! Nice! How can we do this? By creating a static field inside each derived class that gets assigned the returned value from register_type().

More precisely, this would look like:

class Rectangle : public Shape {
  static std::unique_ptr<Shape> deserialize_rectangle(std::istream& in) {
    // ...
  }

  static const bool registered_;
};

// Usually we would do this in the .cpp file, but we wanted it to do header-only, right?
const bool Rectangle::registered_ = Shape::register_type("Rectangle", &deserialize_rectangle); 

When the program is started, one of the first steps is to initialize static objects. Even if registered_ appears to be unused, we can be sure that it won’t be optimized away, because this is forbidden by the C++ standard.

Thus, registered_ will be initialized by calling the register_type() function and our map in the Shape class is being automatically populated at program startup, right?

Sadly, not necessarily. The chances that you end up with an uninitialized map are quite high. And even worse, this will not just result in types not being registered, but rather in a program crash. The problem is that we run into the “static initialization order fiasco” because the order in which static values are initialized is undefined when they are in different translation units.

One of the original posts mentioned that this wouldn’t happen because of “Zero initialization”, but this sadly won’t help us if the two classes are in different translations units, which they probably are.

Circumventing the Static Initialization Order Fiasco

But luckily, there is a solution to this problem: If the classes are in different translation units, we can force the initialization of the type_registry_ by putting it inside a function call and call this function instead. How? By making use of static variables inside functions in our Shape class. This is also called the Construct On First Use Idiom.

class Shape {
  // <snip>

 private:
  static std::map<std::string, std::unique_ptr<Shape> (*)(std::istream&)>& get_type_registry() {
    // Static: One and the same instance for all function calls.
    static std::map<std::string, std::unique_ptr<Shape> (*)(std::istream&)> type_registry;
    return type_registry;
  }
};

Nice! Now we even know how to deal with random static initialization order! Putting it all together, we end up with the following code:

Complete Solution

#include <iostream>
#include <map>
#include <memory>
#include <string>

class Shape {
  // Function pointer that takes a stream and produces an object of a
  // derived class.
  using deserialize_func = std::unique_ptr<Shape> (*)(std::istream&);

 public:
  virtual ~Shape() = default;

  // Provide a high-level serialization function that stores the
  // class_identifier.
  static void serialize(std::ostream& out, Shape& shape) {
    const std::string& class_id = shape.class_identifier();
    out.write(class_id.c_str(), class_id.size() + 1 /*terminating null*/);
    shape.serialize(out);
  }

  // Provide a high-level deserialization function
  // that dispatches to the correct deserialization function based
  // on class_identifier.
  static std::unique_ptr<Shape> deserialize(std::istream& in) {
    std::string class_id;
    std::getline(in, class_id, '\0');
    // Look up the class_id in the type_registry
    // and call the proper deserialization function.
    return get_type_registry()[class_id](in);
  }

 protected:
  // Return the identifier of the derived class.
  virtual std::string class_identifier() const = 0;
  // Serialize the derived class to a stream.
  virtual void serialize(std::ostream& out) const = 0;

  // This function needs to be called by a derived class inside a 
  // *static* context.
  static bool register_type(const std::string& class_id,
                            const deserialize_func factory_method) {
    if (get_type_registry().find(class_id) != get_type_registry().end()) {
      std::cerr << "Class ID " << class_id << " is already used." << std::endl;
      return false;
    }
    get_type_registry().insert(std::make_pair(class_id, factory_method));
    return true;
  }

 private:
  // Prevent "static initialization order fiasco".
  static std::map<std::string, deserialize_func>& get_type_registry() {
    static std::map<std::string, deserialize_func> type_registry;
    return type_registry;
  }
};

class Rectangle : public Shape {
 public:
  std::string class_identifier() const override { return "Rectangle"; }

  void serialize(std::ostream& out) const override {
    // ...
  }

  static std::unique_ptr<Shape> deserialize_rectangle(std::istream& in) {
    // ...
  }

 private:
  static const bool registered_;

  int64_t width_, height_;
};

// Statically register the "Rectangle" class to the factory.
const bool Rectangle::registered_ = Shape::register_type("Rectangle", &deserialize_rectangle);

Further Work

One obvious downside of this approach is that derived classes need to be registered manually. It would be much nicer if they could be forced to register. Following up this blog post, Alexander van Renen proposed a solution to this problem by introducing a new ShapeInterface. Check out his solution on GitHub!

Last time I used it, I wanted to include it with CMake. However, I struggled a little with it. So this post will firstly serve as a reminder for me and secondly make you hopefully struggle less than me.

Project Layout

First of all, we assume a typical CMake project layout:

My_Project
├── CMakeLists.txt
├── include
│   └── ...
├── src
│   ├── my_project.cc
│   ├── ...
│   └── local.cmake
├── vendor
│   └── rapidjson.cmake
└── ...

Download RapidJSON via CMake

CMake can manage external sources as external projects We now want to use CMake’s ExternalProject_Add to automatically download and add RapidJSON. Now the cool thing is that RapidJSON is a “header-only” library, meaning that we can simply copy and include the headers from rapidjson/include and we can then use everything.

Thus, our rapidjson.cmake will only download the library from GitHub and then set RAPIDJSON_INCLUDE_DIR that contains the path to the header files.

With that, our rapidjson.cmake looks like this:

# Download RapidJSON
ExternalProject_Add(
    rapidjson
    PREFIX "vendor/rapidjson"
    GIT_REPOSITORY "https://github.com/Tencent/rapidjson.git"
    GIT_TAG f54b0e47a08782a6131cc3d60f94d038fa6e0a51
    TIMEOUT 10
    CMAKE_ARGS
        -DRAPIDJSON_BUILD_TESTS=OFF
        -DRAPIDJSON_BUILD_DOC=OFF
        -DRAPIDJSON_BUILD_EXAMPLES=OFF
    CONFIGURE_COMMAND ""
    BUILD_COMMAND ""
    INSTALL_COMMAND ""
    UPDATE_COMMAND ""
)

# Prepare RapidJSON (RapidJSON is a header-only library)
ExternalProject_Get_Property(rapidjson source_dir)
set(RAPIDJSON_INCLUDE_DIR ${source_dir}/include)

Include RapidJSON in our Project

In CMakeLists.txt we now need to declare two things:

1. The rapidjson.cmake file exists and should be executed.
2. Add RAPIDJSON_INCLUDE_DIR to the list of included directories.

The first step is done by including the following line somewhere in the CMakeLists.txt file (you probably already have other include() there as well):

include("${CMAKE_SOURCE_DIR}/vendor/rapidjson.cmake")

The next step is to add the header files from RapidJSON to the included directories. This can be done with CMake’s include_directories():

include_directories(${RAPIDJSON_INCLUDE_DIR})

You might already have other included directories there. In the end, the CMakeLists.txt might look similar to this:

# CMakeLists.txt

project(my_project)
cmake_minimum_required(VERSION 3.7)

include("${CMAKE_SOURCE_DIR}/vendor/rapidjson.cmake")

include_directories(
    ${CMAKE_SOURCE_DIR}/include
    ${GFLAGS_INCLUDE_DIR}
    ${BENCHMARK_INCLUDE_DIR}
    ${RAPIDJSON_INCLUDE_DIR}
)

Add RapidJSON as a Dependency

Now one thing is left before we can actually use RapidJSON. Right now, we only tell CMake what to do and not when to do it.

Thus, we want to add RapidJSON as a dependency to our source code so that it will be downloaded before our code is compiled.

Currently, our src/local.cmake might look like this:

# Add source files
file(GLOB_RECURSE SRC_CC "src/*.cc")

# Compile my_project as a library
add_library(my_project SHARED ${SRC_CC})
# Link gflags to my_project
target_link_libraries(my_project gflags Threads::Threads)

Now add the following line under add_library() (or add_executable() in your case):

...
add_library(my_project SHARED ${SRC_CC})
add_dependencies(my_project rapidjson)
...

Now CMake will first download RapidJSON and include its headers and then compile our project.

In my_project.cc we can now import RapidJSON!

#include "rapidjson/document.h"

Now you should be able to run make and everything should compile.

.NET programs are compiled to CIL (common intermediate language) code. When executing such a program the CIL is piped through a JIT (just-in-time) compiler which creates the binary code on the fly. This is similar to Java Bytecode and the JVM. Still, some kind of runtime environment is required on the target machine to interpret the CIL, regardless if being implemented by Microsoft or the Mono project, to run the compiled C# code.

But Mono gives us the power to compile C# code to native binary files and to pack everything into one executable so that we do not need a runtime environment on the target machine! Therefore, our user neither needs Mono nor .NET installed!

In this tutorial, we will create a small hello world program, compile it to native code (on Linux) and pack it into one executable file that can be run on any other Linux machine without having Mono installed. This approach, of course, is not platform-independent any more. Although Mono lets you compile your code for other platforms, the final binary will only work on Linux or Windows machines, but not both. But what you gain is being independent of a third-party runtime installed on the user’s computer.

I assume that Mono is already installed on your machine. If not, please follow the official installation instructions: http://www.mono-project.com/download/. You should have at least version 5.0 installed.

Creating a new program

Let’s start by creating a new file hello.cs with a small C# program that prints out “Hello World”:

using System;
 
public class HelloWorld
{
 static public void Main()
 {
 Console.WriteLine("Hello World");
 }
}

Compile this program by typing in the command line:

mcs hello.cs

This will create a binary file hello.exe in the same folder. Although the newly created file has a .exe file extension this is just platform-independent CIL code (so no worries Linux users).

We can now execute our program within the Mono runtime by typing

mono hello.exe

And it should print “Hello World” to the command line. But this file still requires a runtime for execution. We will now get rid of it ;)

Packing the files into one executable

Mono comes with a tool called mkbundle which does exactly what we want to accomplish.

mkbundle --deps hello.exe -o hello

The --deps options bundles all referenced assemblies into one self-contained image. We could also cross-compile to other platforms with the --cross option.

If the error occurs that some default libraries can not be found you could try including the option --sdk /usr/. This is needed if the default sdk location is not set (e.g. this is the case on Debian with Mono 5.0 from the official repositories)

The command produces a native binary file called hello with all dependencies packed into it. We can ensure this by running ldd hello which gives us the following output:

$ ldd hello
linux-vdso.so.1 (0x00007fff05971000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7b14bc3000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f7b149bb000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7b147b7000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7b1459a000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f7b14383000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7b13fe4000)
/lib64/ld-linux-x86-64.so.2 (0x00007f7b154fa000)

ldd shows us all shared libraries for the executable file, that have to be provided by the operating system on the target machine. As you can see no Mono dependencies need to be linked dynamically. This means there is no need to have Mono installed on the target machine.

Finally, you can distribute the resulting hello file by just copying it to another Linux machine. If you want to release your software to other operating systems you can use the cross-compile functionality of mkbundle.

We now learned how to bundle CIL files into one executable native binary. Although this feature is seldom used, it provides a lot of power.