GDB and SystemTap Probes -- part 3
Hi everybody :-).
I finally got some time to finish this series of posts, and I hope you like the overall result. For those of you who are reading this blog for the first time, you can access the first post here, and the second here.
My goal with this third post is to talk a little bit about how you can
SDT probes with
GDB. Maybe this
particular feature will not be so helpful to you, but I recommend
reading the post either way. I will also give a brief explanation about
SDT probes are laid out inside the binary. So, let’s start!
In my last post, I forgot to mention that the
SDT probe support
present on older versions of Fedora
GDB is not exactly as the way I
described here. This is because Fedora
GDB adopted this feature much
earlier than upstream
GDB itself, so while this has a great positive
aspect in terms of how the distro’s philosophy works (i.e., Fedora
contains leading-edge features, so if you want to know how to FLOSS
community will be in a few months, use it!), it also has the downside of
delivering older/different versions of features in older Fedoras. But of
SDT feature will be fully available on Fedora 18, to be
My suggestion is that if you use a not-so-recent Fedora (like Fedora 16,
15, etc), please upgrade it to the last version, or compile your own
GDB yourself (it’s not that hard, I will make a post about
it in the next days/weeks!).
With that said, let’s move on to our main topic here.
SDT Probes and Tracepoint
Before anything else, let me explain what a
tracepoint is. Think of it
as a breakpoint which doesn’t stop the program’s execution
when it hits. In fact, it’s a bit more than that: you can define
actions associated with a
tracepoint, and those actions will be
performed when the
tracepoint is hit. Neat, huh? :-)
There is a nice description of what a
tracepoint in the GDB
I recommend you give it a reading to understand the concept.
Ok, so now we have to learn how to put
tracepoints in our code, and
how to define actions for them. But before that, let’s remember our
Very simple, isn’t it? Ok, to the
tracepoints now, my friends.
In order to properly use
GDB, you will need to
gdbserver, a tiny version of
GDB suitable for debugging programs
remotely, over the net or serial line. In short, this is because
cannot put tracepoints on a program running directly under it, so we
have to run it inside
gdbserver and then connect
GDB to it.
Running our program inside
In our case, we will just start
gdbserver in our machine, order it to
listen to some high port, and connect to it through
there will be no need to have access to another computer or device.
First of all, make sure you have
gdbserver installed. If you use
Fedora, the package name you will have to install is
you have it installed, you can do:
The second argument passed to
gdbserver instructs it to listen on the
port 3001 of your loopback interface, a.k.a.
You will notice that
gdbserver will stay there indefinitely, waiting
for new connections to arrive. Don’t worry, we will connect to it soon!
Connecting an instance of
Now, go to another terminal and start
GDB with our program:
The command you have to use inside
target remote. It takes as
an argument the host and the port to which you want to connect. In our
case, we just want it to connect to
localhost, port 3001. If you saw
an output like the above, great, things are working for you (don’t pay
attention to the messages about
glibc debug information). If you didn’t see it, please check to see if
you’re connecting to the right port, and if no other service is using
Ok, so now it is time to start our trace experiment!
Every command should be issued on GDB, not on gdbserver!
GDB prompt, put a
tracepoint in the probe named
As you can see, the
trace command takes exactly the same arguments as
break command. Thus, you need to use the
-probe-stap modified in
order to instruct
GDB to put the
tracepoint in the probe.
And now, let’s define the actions associated with this
To do that, we use the
actions command, which is an interactive
GDB. It takes some specific keywords, and if you want
to learn more about it, please take a look at this
For this example, we will use only the
collect keyword, which tells
GDB to… hm… collect something :-). In our case, it will collect
the probe’s first argument, or
$_probe_arg0, as you may remember.
Simple as that. Finally, we have to define a
breakpoint in the last
instruction of our program, because it is necessary to keep it running
gdbserver in order to examine the
tracepoints later. If we didn’t
breakpoint, our program would finish and
not be able to provide information about what happened with our
tracepoints. In our case, we will simply put a
breakpoint on line
10, i.e., on the
Running the trace experiment
Ok, time to run our trace experiment. First, we must issue a
GDB to start monitoring the
tracepoints. And then, we can
continue our program normally.
GDB is not going to stop your program, because
tracepoints are designed to not interfere with the execution of it.
Also notice that we have also stopped the trace experiment after the
breakpoint hit, by using the
Now, we will be able to examine what the
tracepoint has collected.
First, we will the
tfind command to make sure the
hit, and then we can inspect what we ordered it to collect:
And it works! Notice that we are printing the probe argument using the
same notation as with
breakpoints, even though we are not exactly
STAP_PROBE1 instruction. What does it mean? Well, with
tfind start command we tell
GDB to actually use the trace frame
collected during the program’s execution, which, in this case, is the
probe argument. If you know
GDB, think of it as if we were using the
frame command to jump back to a specific frame, where we would have
access to its state.
This is a very simple example of how to use the
SDT probe support in
tracepoints. There is much more you can do, but I hope I
could explain the basics so that you can start playing with this
SDT probe is laid out in the binary
You might be interested in learning how the probes are created inside
the binary. Other than reading the source code of
/usr/include/sys/sdt.h, which is the heart of the whole feature, I
also recommend this
which explains in detail what’s going on under the hood. I also
recommend that you study a little about how the ELF format works,
specifically about notes in the ELF file.
After this series of blog posts, I expect that you will now be able to
use the not-so-new feature of
SDT probe support on
GDB. Of course,
if you find some bug while using this, please feel free to report it
using our bugzilla. And if you have
some question, use the comment system below and I will answer ASAP :-).
See ya, and thanks for reading!