Due to the unique coroutine-based architecture of FoundationDB, profiling FoundationDB may be a little bit tricky sometimes. I’d like to share some methods and tips we use at Snowflake to efficiently profile FoundationDB.
Isolate the Roles
The very first thing you need to do is to isolate the roles, so you can identify which role is being the bottleneck for your workload. If you are running FoundationDB on a single machine (a development machine, for example), binding each process to a specific CPU in conjunction with using
htop command or
top -H -c command will greatly help you to identify the hot process.
command = /usr/bin/numactl --physcpubind=0 fdbserver
class = master
command = /usr/bin/numactl --physcpubind=1 fdbserver
class = proxy
command = /usr/bin/numactl --physcpubind=2 fdbserver
class = resolution
… and so on …
perf is one of the most commonly used profiling tools on Linux, and it works perfectly for FoundationDB. Once you identify the hot
fdbserver process, you can use
perf top to take a quick look at the top functions.
perf top -p <fdbserver PID>
If you need to drill down even further, use
perf record and
perf record --call-graph dwarf -o <profile.data> -p <fdbserver PID>
This will collect the CPU-cycle profile and save it as <profile.data> file. If you want to look at another metric than CPU-cycle, specify
-e <event> option (For example,
-e cache-misses). For the complete list of events supported on your system, run
--call-graph dwarf is an option to use DWARF´s CFI (Call Frame Information) to collect call-graph information instead of using the frame pointer (default in
If your FoundationDB does not contain the debug information (e.g. Release binary), then you need to install
debuginfo package globally, or extract the debuginfo data under an arbitrary location and specify
--symfs argument pointing to the directory which contains your
Once you collect the profiles, then you can use
perf report to navigate through the data.
perf report -g -i <profile.data> --no-children
This will open an interactive viewer in the bottom-up view, which we found more useful to easily identify the bottlenecks. If you do not specify
perf will open a top-down view.
If you pipe the
perf reportoutput to a file or another command, such as
less, it will generate an expanded view. It is useful if you want to share the output with someone.
Another profiler we regularly use is
valgrind is a set of various profilers, such as
cachegrind (cache usage) or
callgrind (CPU usage).
You cannot attach
Valgrind to an already running process, so you need to start
fdbserver process through
valgrind manually. In order to do so, you need to comment out your process from
foundationdb.conf file first. As soon as you make the change,
fdbmonitor process will automatically stop the
ps -ef | grep fdbserver” before stopping it, and save the exact command line, so you can copy and paste it as
valgrind --tool=callgrind /path/to/fdbserver --cluster_file ...
fdbserver joins the cluster, then run your workload.
When the workload is finished, or you collected enough data, stop
valgrind by hitting ctrl-c. It will generate a file like
Once you have the
valgrind output file, you can copy it to your local machine and open it with
qcachegrind and visually analyze it.
Here’s a screenshot of
qcachegrind on my mac.
We’d like to hear your tips too!
I didn’t describe here, but I also use Intel Pin Tool sometimes, especially when collecting instruction counts. Good old
pstack is sometimes good enough (and super quick!). I’m pretty sure other people in this forum have different methods and tips. Please share little bit of your secrets too.