A debuginfod service for Debian
Table of Contents
Hi there. Long time no write!
This last Tuesday, February 23, 2021, I made an announcement at debian-devel-announce about a new service that I configured for Debian: a debuginfod server.
This post serves two purposed: pay the promise I made to Jonathan Carter that I would write a blog post about the service, and go into a bit more detail about it.
What’s debuginfod?
From the announcement above:
debuginfod is a new-ish project whose purpose is to serve
ELF/DWARF/source-code information over HTTP. It is developed under the
elfutils umbrella. You can find more information about it here:
https://sourceware.org/elfutils/Debuginfod.html
In a nutshell, by using a debuginfod service you will not need to
install debuginfo (a.k.a. dbgsym) files anymore; the symbols will be
served to GDB (or any other debuginfo consumer that supports debuginfod)
over the network. Ultimately, this makes the debugging experience much
smoother (I myself never remember the full URL of our debuginfo
repository when I need it).
Perhaps not everybody knows this, but until last year I was a Debugger
Engineer (a.k.a. GDB hacker) at Red Hat. I was not involved with
the creation of debuginfod directly, but I witnessed discussions
about “having way to serve debug symbols over the internet” multiple
times during my tenure at the company. So this is not a new idea, and
it’s not even the first implementation, but it’s the first time that
some engineers actually got their hands dirty enough to have something
concrete in hands.
The idea to set up a debuginfod server for Debian started to brew
after 2019’s GNU Tools
Cauldron, but as usual several
things happened in $LIFE (including a global pandemic and leaving Red
Hat and starting a completely different job at Canonical) which had
the effect of shuffling my TODO list “a little”.
Benefits for Debian
Debian unfortunately is lagging behind when it comes to offer its
users a good debugging experience. Before the advent of our
debuginfod server, if you wanted to debug a package in Debian you
would need to:
-
Add the
debian-debugapt repository to your/etc/apt/sources.list. -
Install the
dbgsympackage that contains the debug symbols for the package you are debugging. Note that the version of thedbgsympackage needs to be exactly the same as the version of the package you want to debug. -
Figure out which shared libraries your package uses and install the
dbgsympackages for all of them. Arguably, this step is optional but recommended if you would like to perform a more in-depth debugging. -
Download the package source, possibly using
apt sourceor some equivalent command. -
Open GDB, and make sure you adjust the source paths properly (more below). This can be non-trivial.
-
Finally, debug the program.
Now, with the new service, you will be able to start from step 4,
without having to mess with sources.list, dbgsym packages and
version mismatches.
The package source
It is important to mention an existing (but perhaps not well-known)
limitation of our debugging experience in Debian: the need to manually
download the source packages and adjust GDB to properly find them
(see step 4 above). debuginfod is able to serve source code as
well, but our Debian instance is not doing that at the moment.
Debian does not provide a patched source tree that is ready to be
consumed by GDB nor debuginfod (for a good example of a distribution
that does that, see Fedora’s debugsource packages). Let me show you
an example of debugging GDB itself (using debuginfod) on Debian:
$ HOME=/tmp DEBUGINFOD_URLS=https://debuginfod.debian.net gdb -q gdb
Reading symbols from gdb...
Downloading separate debug info for /tmp/gdb...
Reading symbols from /tmp/.cache/debuginfod_client/02046bac4352940d19d9164bab73b2f5cefc8c73/debuginfo...
(gdb) start
Temporary breakpoint 1 at 0xd18e0: file /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c, line 28.
Starting program: /usr/bin/gdb
Downloading separate debug info for /lib/x86_64-linux-gnu/libreadline.so.8...
Downloading separate debug info for /lib/x86_64-linux-gnu/libz.so.1...
Downloading separate debug info for /lib/x86_64-linux-gnu/libncursesw.so.6...
Downloading separate debug info for /lib/x86_64-linux-gnu/libtinfo.so.6...
Downloading separate debug info for /tmp/.cache/debuginfod_client/d6920dbdd057f44edaf4c1fbce191b5854dfd9e6/debuginfo...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Downloading separate debug info for /lib/x86_64-linux-gnu/libexpat.so.1...
Downloading separate debug info for /lib/x86_64-linux-gnu/liblzma.so.5...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libbabeltrace.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libbabeltrace-ctf.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libipt.so.2...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libmpfr.so.6...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libsource-highlight.so.4...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libxxhash.so.0...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libdebuginfod.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libstdc++.so.6...
Downloading separate debug info for /lib/x86_64-linux-gnu/libgcc_s.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0...
Downloading separate debug info for /tmp/.cache/debuginfod_client/dbfea245d26065975b4084f4e9cd2d83c65973ee/debuginfo...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libdw.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libelf.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libuuid.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libgmp.so.10...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libboost_regex.so.1.74.0...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4...
Downloading separate debug info for /lib/x86_64-linux-gnu/libbz2.so.1.0...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libicui18n.so.67...
Downloading separate debug info for /tmp/.cache/debuginfod_client/acaa831dbbc8aa70bb2131134e0c83206a0701f9/debuginfo...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libicuuc.so.67...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libnghttp2.so.14...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libidn2.so.0...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/librtmp.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libssh2.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libpsl.so.5...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libnettle.so.8...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libgnutls.so.30...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libbrotlidec.so.1...
Downloading separate debug info for /tmp/.cache/debuginfod_client/39739740c2f8a033de95c1c0b1eb8be445610b31/debuginfo...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libunistring.so.2...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libhogweed.so.6...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libgcrypt.so.20...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libp11-kit.so.0...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libtasn1.so.6...
Downloading separate debug info for /lib/x86_64-linux-gnu/libcom_err.so.2...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libsasl2.so.2...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libbrotlicommon.so.1...
Downloading separate debug info for /lib/x86_64-linux-gnu/libgpg-error.so.0...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libffi.so.7...
Downloading separate debug info for /lib/x86_64-linux-gnu/libkeyutils.so.1...
Temporary breakpoint 1, main (argc=1, argv=0x7fffffffebf8) at /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c:28
28 /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c: Directory not empty.
(gdb) list
23 in /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c
(gdb)
(See all those Downloading separate debug info for... lines? Nice!)
As you can see, when we try to list the contents of the file we’re
in, nothing shows up. This happens because GDB doesn’t know where the
file is. So you have to tell it. In this case, it’s relatively easy:
you see that the GDB package’s build directory is
/build/gdb-Nav6Es/gdb-10.1/. When you apt source gdb, you will
have a directory called $PWD/gdb-10.1/ containing the full source of the
package. Notice that the last directory’s name in both paths is the
same, so in this case we can use GDB’s set substitute-path command
do the job for us (in this example $PWD is /tmp/):
$ HOME=/tmp DEBUGINFOD_URLS=https://debuginfod.debian.net gdb -q gdb
Reading symbols from gdb...
Reading symbols from /tmp/.cache/debuginfod_client/02046bac4352940d19d9164bab73b2f5cefc8c73/debuginfo...
(gdb) set substitute-path /build/gdb-Nav6Es/ /tmp/
(gdb) start
Temporary breakpoint 1 at 0xd18e0: file /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c, line 28.
Starting program: /usr/bin/gdb
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Temporary breakpoint 1, main (argc=1, argv=0x7fffffffebf8) at /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c:28
warning: Source file is more recent than executable.
28 memset (&args, 0, sizeof args);
(gdb) list
23 int
24 main (int argc, char **argv)
25 {
26 struct captured_main_args args;
27
28 memset (&args, 0, sizeof args);
29 args.argc = argc;
30 args.argv = argv;
31 args.interpreter_p = INTERP_CONSOLE;
32 return gdb_main (&args);
(gdb)
Much better, huh? The problem is that this process is manual, and will change depending on how the package you’re debugging was built.
What can we do to improve this? What I personally would like to see is something similar to what the Fedora project already does: create a new debug package which will contain the full, patched source package. This would mean changing our building infrastructure and possibly other somewhat complex things.
Using the service (by default)
At the time of this writing, I am working on an elfutils Merge
Request
whose purpose is to implement a debconf question to ask the user
whether she wants to use our service by default.
If you would like to start using the service right now, all you have to do is set the following environment variable in your shell:
DEBUGINFOD_URLS="https://debuginfod.debian.net"
More information
You can find more information about our debuginfod service
here. Try to keep an eye on the
page as it’s being constantly updated.
If you’d like to get in touch with me, my email is my domain at debian dot org.
I sincerely believe that this service is a step in the right direction, and hope that it can be useful to you :-).