You can read a detailed description of the problem in the message Gary sent to the gdb-patches mailing list, but to summarize: GDB needs to interface with the linker in order to identify which shared libraries were loaded during the inferior’s (i.e., program being debugged) life.
Nowadays, what GDB does is to put a breakpoint in _dl_debug_state
,
which is an empty function called by the linker every time a shared
library is loaded (the linker calls it twice, once before modifying the
list of loaded shlibs, and once after). But GDB has no way to know what
has changed in the list of loaded shlibs, and therefore it needs to load
the entire list every time something happens. You can imagine how bad
this is for performance…
What Gary did was to put SDT probes strategically on the linker, so that
GDB could make use of them when examining for changes in the list of
loaded shlibs. It improves performance a lot, because now GDB doesn’t
need to stop twice every time a shlib is loaded (it just needs to do
that when stop-on-solib-events
is set); it just needs to stop at the
right probe, which will inform the address of the link-map entry of the
first newly added library. It means GDB also won’t need to walk through
the list of shlibs and identify what has changed: you get that for free
by examining the probe’s argument.
Gary also mentions a discrepancy that happened on Solaris libc, which has also been solved by his patch.
And now, the most impressing thing: the numbers! Take a look at this table, which displays the huge improvement in the performance when using lots of shlibs (the time is in seconds):
Number of shlibs | 128 | 256 | 512 | 1024 | 2048 | 4096 |
---|---|---|---|---|---|---|
Old interface | > 0 | > 1 | > 4 | > 12 | > 47 | > 185 |
New interface | > 0 | > 0 | > 2 | > 4 | > 10 | > 36 |
Impressive, isn’t it?
This is one the things I like most in Free Software projects: the possibility of extending and improving things by using what others did before. When I hacked GDB to implement the integration between itself and SystemTap, I had absolutely no idea that this could be used for improving the interface between the linker and the debugger (though I am almost sure that Tom was already thinking ahead!). And I can say it is a pleasure and I feel proud when I see such things happening. It just makes me feel more and more certain that Free Software is the way to go :-).
]]>I finally got some time to finish this series of posts, and I hope you like the overall result. For those of you who are reading this blog for the first time, you can access the first post here, and the second here.
My goal with this third post is to talk a little bit about how you can
use the SDT
probes with tracepoints
inside GDB
. Maybe this
particular feature will not be so helpful to you, but I recommend
reading the post either way. I will also give a brief explanation about
how the SDT
probes are laid out inside the binary. So, let’s start!
In my last post, I forgot to mention that the SDT
probe support
present on older versions of Fedora GDB
is not exactly as the way I
described here. This is because Fedora GDB
adopted this feature much
earlier than upstream GDB
itself, so while this has a great positive
aspect in terms of how the distro’s philosophy works (i.e., Fedora
contains leading-edge features, so if you want to know how to FLOSS
community will be in a few months, use it!), it also has the downside of
delivering older/different versions of features in older Fedoras. But of
course, this SDT
feature will be fully available on Fedora 18, to be
announced soon.
My suggestion is that if you use a not-so-recent Fedora (like Fedora 16,
15, etc), please upgrade it to the last version, or compile your own
version of GDB
yourself (it’s not that hard, I will make a post about
it in the next days/weeks!).
With that said, let’s move on to our main topic here.
Before anything else, let me explain what a tracepoint
is. Think of it
as a breakpoint which doesn’t stop the program’s execution
when it hits. In fact, it’s a bit more than that: you can define
actions associated with a tracepoint
, and those actions will be
performed when the tracepoint
is hit. Neat, huh? :-)
There is a nice description of what a tracepoint
in the GDB
documentation,
I recommend you give it a reading to understand the concept.
Ok, so now we have to learn how to put tracepoints
in our code, and
how to define actions for them. But before that, let’s remember our
example program:
#include <sys/sdt.h>
int
main (int argc, char *argv[])
{
int a = 10;
STAP_PROBE1 (test_program, my_probe, a);
return 0;
}
Very simple, isn’t it? Ok, to the tracepoints
now, my friends.
tracepoints
inside GDB
In order to properly use tracepoints
inside GDB
, you will need to
use gdbserver
, a tiny version of GDB
suitable for debugging programs
remotely, over the net or serial line. In short, this is because GDB
cannot put tracepoints on a program running directly under it, so we
have to run it inside gdbserver
and then connect GDB
to it.
gdbserver
In our case, we will just start gdbserver
in our machine, order it to
listen to some high port, and connect to it through localhost
, so
there will be no need to have access to another computer or device.
First of all, make sure you have gdbserver
installed. If you use
Fedora, the package name you will have to install is gdb-gdbserver
. If
you have it installed, you can do:
$ gdbserver :3001 ./test_program
Process ./test_program created; pid = 17793
Listening on port 3001
The second argument passed to gdbserver
instructs it to listen on the
port 3001 of your loopback interface, a.k.a. localhost
.
You will notice that gdbserver
will stay there indefinitely, waiting
for new connections to arrive. Don’t worry, we will connect to it soon!
GDB
to gdbserver
Now, go to another terminal and start GDB
with our program:
$ gdb ./test_program
...
(gdb) target remote :3001
Remote debugging using :3001
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x0000003d60401530 in _start () from /lib64/ld-linux-x86-64.so.2
The command you have to use inside GDB
is target remote
. It takes as
an argument the host and the port to which you want to connect. In our
case, we just want it to connect to localhost
, port 3001. If you saw
an output like the above, great, things are working for you (don’t pay
attention to the messages about
glibc debug information). If you didn’t see it, please check to see if
you’re connecting to the right port, and if no other service is using
it.
Ok, so now it is time to start our trace experiment!
tracepoints
Every command should be issued on GDB, not on gdbserver!
In your GDB
prompt, put a tracepoint
in the probe named my_probe
:
(gdb) trace -probe-stap my_probe
Tracepoint 1 at 0x4005a9
As you can see, the trace
command takes exactly the same arguments as
the break
command. Thus, you need to use the -probe-stap
modified in
order to instruct GDB
to put the tracepoint
in the probe.
And now, let’s define the actions associated with this tracepoint
.
To do that, we use the actions
command, which is an interactive
command inside GDB
. It takes some specific keywords, and if you want
to learn more about it, please take a look at this
link.
For this example, we will use only the collect
keyword, which tells
GDB
to… hm… collect something :-). In our case, it will collect
the probe’s first argument, or $_probe_arg0
, as you may remember.
(gdb) actions
Enter actions for tracepoint 1, one per line.
End with a line saying just "end".
>collect $_probe_arg0
>end
(gdb)
Simple as that. Finally, we have to define a breakpoint
in the last
instruction of our program, because it is necessary to keep it running
on gdbserver
in order to examine the tracepoints
later. If we didn’t
put this breakpoint
, our program would finish and gdbserver
would
not be able to provide information about what happened with our
tracepoints
. In our case, we will simply put a breakpoint
on line
10, i.e., on the return 0;
:
Ok, time to run our trace experiment. First, we must issue a tstart
to
tell GDB
to start monitoring the tracepoints
. And then, we can
continue our program normally.
(gdb) tstart
(gdb) continue
Continuing.
Breakpoint 1, main (argc=1, argv=0x7fffffffde88) at /tmp/test_program.c:10
10 return 0;
(gdb) tstop
(gdb)
Remember, GDB
is not going to stop your program, because
tracepoints
are designed to not interfere with the execution of it.
Also notice that we have also stopped the trace experiment after the
breakpoint
hit, by using the tstop
command.
Now, we will be able to examine what the tracepoint
has collected.
First, we will the tfind
command to make sure the tracepoint
has
hit, and then we can inspect what we ordered it to collect:
(gdb) tfind start
Found trace frame 0, tracepoint 1
8 STAP_PROBE1 (test_program, my_probe, a);
(gdb) p $_probe_arg0
$1 = 10
And it works! Notice that we are printing the probe argument using the
same notation as with breakpoints
, even though we are not exactly
executing the STAP_PROBE1
instruction. What does it mean? Well, with
the tfind start
command we tell GDB
to actually use the trace frame
collected during the program’s execution, which, in this case, is the
probe argument. If you know GDB
, think of it as if we were using the
frame
command to jump back to a specific frame, where we would have
access to its state.
This is a very simple example of how to use the SDT
probe support in
GDB
with tracepoints
. There is much more you can do, but I hope I
could explain the basics so that you can start playing with this
feature.
SDT
probe is laid out in the binaryYou might be interested in learning how the probes are created inside
the binary. Other than reading the source code of
/usr/include/sys/sdt.h
, which is the heart of the whole feature, I
also recommend this
page,
which explains in detail what’s going on under the hood. I also
recommend that you study a little about how the ELF format works,
specifically about notes in the ELF file.
After this series of blog posts, I expect that you will now be able to
use the not-so-new feature of SDT
probe support on GDB
. Of course,
if you find some bug while using this, please feel free to report it
using our bugzilla. And if you have
some question, use the comment system below and I will answer ASAP :-).
See ya, and thanks for reading!
]]>It’s been a long time since I wrote the first post about this subject, and since then the patches have been accepted upstream, and GDB 7.5 now has official support for userspace SystemTap probes :-). Yay!
Well, but enough of cheap talk, let’s get to the business!
Frank Ch. Eigler, one of SystemTap’s maintainers, kindly mentioned something that I should say about SystemTap userspace probes.
Basically, it should be clear that SDT
probes are not the only kind of
userspace probing one can do with SystemTap. There is yet another kind
of probe (maybe even more powerful, depending on the goals):
DWARF-based function/statement probes. SystemTap supports this kind
of probing mechanism for quite a while now.
It is not the goal of this post to explain it in detail, but you might
want to give it a try by compiling your binary with debuginfo support
(use the -g
flag on GCC
), and do something like:
$ stap -e 'probe process("/bin/foo").function("name") { log($$parms) }' -c /bin/foo
$ stap -e 'probe process("/bin/foo").statement("*@file.c:443") { log($$vars) }' -c /bin/foo
And that’s it. You can read SystemTap’s documentation, or this guide to learn how to add userspace probes.
Well, now let’s get to the interesting part. It is time to make GDB
work with the SDT
probe that we have put in our example code. Let’s
remember it:
#include <sys/sdt.h>
int
main (int argc, char *argv[])
{
int a = 10;
STAP_PROBE1 (test_program, my_probe, a);
return 0;
}
It is a very simple example, and we will have to extend it later in order to show more features. But for now, it will do.
The first thing to do is to open GDB
(with SystemTap support, of
course!), and check to see if it can actually see probe inserted in our
example.
$ gdb ./test_program
GNU gdb (GDB) 7.5.50.20121014-cvs
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
...
(gdb) info probes
Provider Name Where Semaphore Object
test_program my_probe 0x00000000004004ae /home/sergio/work/src/git/build/gdb/test_program
Wow, it actually works! :-)
If you have seen something like the above, it means your GDB
is
correctly recognizing SDT
probes. If you see an error, or if your
GDB
doesn’t have the info probes
command, then you’d better make
sure you have a recent version of GDB
otherwise you won’t be able to
use the SDT
support.
Anyway, now it is time to start using this support. The first thing I want to show you is how to put a breakpoint in a probe.
(gdb) break -probe-stap my_probe
Breakpoint 1 at 0x4004ae
That’s all! We have chosen to extend the break
command in order to
support the new -probe-stap
parameter. If you’re wondering … why
the -probe prefix?, it is because I was asked to implement a complete
abstraction layer inside GDB
in order to allow more types of probes to
be added in the future. So, for example, if someone implements support
for an hypothetical type of probe called xyz
, you would have
break -probe-xyz
. It took me a little more time to implement this
layer, but it is worth the effort.
Anyway, as you have see above, GDB
recognize the probe’s name and
correctly put a breakpoint in it. You can also confirm that it has done
the right thing by matching the address reported by info probes
with
the one reported by break
: they should be the same.
Ok, so now, with our breakpoint
in place, let’s run the program and
see what happens.
(gdb) run
Starting program: /home/sergio/work/src/git/build/gdb/test_program
Breakpoint 1, main (argc=1, argv=0x7fffffffdf68) at /tmp/example-stap.c:8
8 STAP_PROBE1 (test_program, my_probe, a);
As you can see, GDB
stopped at the exact location of the probe.
Therefore, you are now able to put marks (i.e., probes) in your source
code which are location-independent. It means that it doesn’t really
matter where in the source code your probe is, and it also doesn’t
matter if you change the code around it, changing the line numbers, or
even moving it to another file. GDB
will always find your probe, and
always stop at the right location. Neat!
But wait, there’s more! Remember when I told you that you could also inspect the probe’s arguments? Yes, let’s do it now!
Just remember that, in SDT
’s parlance, the current probe’s argument is
a
. So let’s print its value.
(gdb) p $_probe_arg0
$1 = 10
(gdb) p a
$2 = 10
“Hey, captain, it seems the boat really floats!”
Check the source code above, and convince yourself that a
’s value is
10
:-). As you might have seen, I have used a fairly strange way of
printing it. It is because the probe’s arguments are available inside
GDB
by means of convenience variables. You can see a list of them
here.
Since SDT
probes can have up to 12 arguments (i.e., you can use
STAP_PROBE1
… STAP_PROBE12
), we have created inside GDB
12
convenience variables, named $_probe_arg0
until $_probe_arg11
. I
know, it is not an easy name to remember, and even the relation between
SDT
naming and GDB
naming is not direct (i.e., you have to subtract
1 from the SDT
probe number). If you are not satisfied with this,
please open a bug in our bugzilla and
I promise we will discuss other options.
I would like to emphasize something here: just as you don’t need
debuginfo support for dealing with probes inside GDB
, you also don’t
need debuginfo support for dealing with their arguments as well. It
means that you can actually compile your code without debuginfo support,
but still have access to some important variables/expressions when
debugging it. Depending on how GCC
optimizes your code, you may
experience some difficulties with argument printing, but so far I
haven’t heard of anything like that.
Ok, now we have covered more things about the SDT
probe support inside
GDB
, and I hope you understood all the concepts. It is not hard to get
things going with this, specially because you don’t need extra libraries
to make it work.
In the next post, I intend to finish this series by explaining how to
use tracepoints
with SDT
probes. Also, as I said in the previous
post of this series, maybe I will talk a little bit about how the SDT
probes are organized within the binary.
See you soon!
]]>With this post I will start to talk about the integration between GDB and SystemTap. This is something that Tom Tromey and I did during the last year. The patch is being reviewed as I write this post, and I expect to see it checked-in in the next few days/weeks. But let’s get our hands dirty…
You probably use (or have at least heard of) SystemTap, and maybe you think the tool is only useful for kernel inspections. If that’s your case, I have a good news: you’re wrong! You can actually use SystemTap to inspect userspace applications too, by using what we call SDT probes, or Static Defined Tracing probes. This is a very cheap and easy way to include probes in your application, and you can even specify arguments to those probes.
In order to use the probes (see an example below), you must include the
<sys/sdt.h>
header file in your source code. If you are using Fedora
systems, you can obtain this header file by installing the package
systemtap-sdt-devel
, version equal or greater than 1.4
.
Here’s a simple example of an application with a one-argument probe:
#include <sys/sdt.h>
int
main (int argc, char *argv[])
{
int a = 10;
STAP_PROBE1 (test_program, my_probe, a);
return 0;
}
As you can see, this is a very simple program with one probe, which contains one argument. You can now compile the program:
$ gcc test_program.c -o test_program
Now you must be thinking: “Wait, wait… Didn’t you just forget to link
this program against some SystemTap-specific library or something?” And
my answer is no. One of the spetacular things about this
<sys/sdt.h>
header is that it does not have any dependencies at all!
As Tom said in his blog post, this is
“a virtuoso display of ELF and GCC asm wizardy”.
If you want to make sure your probe was inserted in the binary, you can
use readelf
command:
$ readelf -x .note.stapsdt ./test_program
Hex dump of section '.note.stapsdt':
0x00000000 08000000 3a000000 03000000 73746170 ....:.......stap
0x00000010 73647400 86044000 00000000 88054000 sdt...@.......@.
0x00000020 00000000 00000000 00000000 74657374 ............test
0x00000030 5f70726f 6772616d 006d795f 70726f62 _program.my_prob
0x00000040 65002d34 402d3428 25726270 29000000 e.-4@-4(%rbp)...
(I will think about writing an explanation on how the probes are laid
out on the binary, but for now you just have to care if you actually
see an output from this readelf
command.)
You can also use SystemTap to perform this verification:
$ stap -L 'process("./test_program").mark("*")'
process("./test_program").mark("my_probe") $arg1:long
So far, so good. If you see an output like the one above, it means your probe is correctly inserted. You could obviously use SystemTap to inspect this probe, but I won’t do this right now because this is not the purpose of this post.
For now, we have learned how to:
SDT probe
in our source code, and compile it;In the next post, I will talk about the GDB support that allows you to
inspect, print arguments, and gather other information about
SDT probes
. I hope you like it!