I finally got some time to finish this series of posts, and I hope you
like the overall result. For those of you who are reading this blog
for the first time, you can access the first
My goal with this third post is to talk a little bit about how you can
use the SDT probes with tracepoints inside GDB. Maybe this
particular feature will not be so helpful to you, but I recommend
reading the post either way. I will also give a brief explanation about
how the SDT probes are laid out inside the binary. So, let’s start!
In my last post, I forgot to mention that the SDT probe support
present on older versions of Fedora GDB is not exactly as the way I
described here. This is because Fedora GDB adopted this feature much
earlier than upstream GDB itself, so while this has a great positive
aspect in terms of how the distro’s philosophy works (i.e., Fedora
contains leading-edge features, so if you want to know how to FLOSS
community will be in a few months, use it!), it also has the downside of
delivering older/different versions of features in older Fedoras. But of
course, this SDT feature will be fully available on Fedora 18, to be
My suggestion is that if you use a not-so-recent Fedora (like Fedora 16,
15, etc), please upgrade it to the last version, or compile your own
version of GDB yourself (it’s not that hard, I will make a post about
it in the next days/weeks!).
With that said, let’s move on to our main topic here.
SDT Probes and Tracepoint
Before anything else, let me explain what a tracepoint is. Think of it
as a breakpoint which doesn’t stop the program’s execution
when it hits. In fact, it’s a bit more than that: you can define
actions associated with a tracepoint, and those actions will be
performed when the tracepoint is hit. Neat, huh? :-)
There is a nice description of what a tracepoint in the GDB
I recommend you give it a reading to understand the concept.
Ok, so now we have to learn how to put tracepoints in our code, and
how to define actions for them. But before that, let’s remember our
Very simple, isn’t it? Ok, to the tracepoints now, my friends.
Using tracepoints inside GDB
In order to properly use tracepoints inside GDB, you will need to
use gdbserver, a tiny version of GDB suitable for debugging programs
remotely, over the net or serial line. In short, this is because GDB
cannot put tracepoints on a program running directly under it, so we
have to run it inside gdbserver and then connect GDB to it.
Running our program inside gdbserver
In our case, we will just start gdbserver in our machine, order it to
listen to some high port, and connect to it through localhost, so
there will be no need to have access to another computer or device.
First of all, make sure you have gdbserver installed. If you use
Fedora, the package name you will have to install is gdb-gdbserver. If
you have it installed, you can do:
The second argument passed to gdbserver instructs it to listen on the
port 3001 of your loopback interface, a.k.a. localhost.
You will notice that gdbserver will stay there indefinitely, waiting
for new connections to arrive. Don’t worry, we will connect to it soon!
Connecting an instance of GDB to gdbserver
Now, go to another terminal and start GDB with our program:
The command you have to use inside GDB is target remote. It takes as
an argument the host and the port to which you want to connect. In our
case, we just want it to connect to localhost, port 3001. If you saw
an output like the above, great, things are working for you (don’t pay
attention to the messages about
glibc debug information). If you didn’t see it, please check to see if
you’re connecting to the right port, and if no other service is using
Ok, so now it is time to start our trace experiment!
Creating the tracepoints
Every command should be issued on GDB, not on gdbserver!
In your GDB prompt, put a tracepoint in the probe named my_probe:
As you can see, the trace command takes exactly the same arguments as
the break command. Thus, you need to use the -probe-stap modified in
order to instruct GDB to put the tracepoint in the probe.
And now, let’s define the actions associated with this tracepoint.
To do that, we use the actions command, which is an interactive
command inside GDB. It takes some specific keywords, and if you want
to learn more about it, please take a look at this
For this example, we will use only the collect keyword, which tells
GDB to… hm… collect something :-). In our case, it will collect
the probe’s first argument, or $_probe_arg0, as you may remember.
Simple as that. Finally, we have to define a breakpoint in the last
instruction of our program, because it is necessary to keep it running
on gdbserver in order to examine the tracepoints later. If we didn’t
put this breakpoint, our program would finish and gdbserver would
not be able to provide information about what happened with our
tracepoints. In our case, we will simply put a breakpoint on line
10, i.e., on the return 0;:
Running the trace experiment
Ok, time to run our trace experiment. First, we must issue a tstart to
tell GDB to start monitoring the tracepoints. And then, we can
continue our program normally.
Remember, GDB is not going to stop your program, because
tracepoints are designed to not interfere with the execution of it.
Also notice that we have also stopped the trace experiment after the
breakpoint hit, by using the tstop command.
Now, we will be able to examine what the tracepoint has collected.
First, we will the tfind command to make sure the tracepoint has
hit, and then we can inspect what we ordered it to collect:
And it works! Notice that we are printing the probe argument using the
same notation as with breakpoints, even though we are not exactly
executing the STAP_PROBE1 instruction. What does it mean? Well, with
the tfind start command we tell GDB to actually use the trace frame
collected during the program’s execution, which, in this case, is the
probe argument. If you know GDB, think of it as if we were using the
frame command to jump back to a specific frame, where we would have
access to its state.
This is a very simple example of how to use the SDT probe support in
GDB with tracepoints. There is much more you can do, but I hope I
could explain the basics so that you can start playing with this
How the SDT probe is laid out in the binary
You might be interested in learning how the probes are created inside
the binary. Other than reading the source code of
/usr/include/sys/sdt.h, which is the heart of the whole feature, I
also recommend this
which explains in detail what’s going on under the hood. I also
recommend that you study a little about how the ELF format works,
specifically about notes in the ELF file.
After this series of blog posts, I expect that you will now be able to
use the not-so-new feature of SDT probe support on GDB. Of course,
if you find some bug while using this, please feel free to report it
using our bugzilla. And if you have
some question, use the comment system below and I will answer ASAP :-).
In the last days, there was some fuzz about Mark Shuttleworth’s
post about some news in
Ubuntu. Well, to be more precise, the post was about some secrets
regarding the development of some applications. In my opinion, it
summarizes some of what this distribution has become, and I would like
to talk a little bit about it here.
My goal is not say bad things about Ubuntu (though I won’t promise that,
because the post is intended as a critic), nor about Mr. Shuttleworth,
whom I don’t know personally and have nothing against. My purpose is to
discuss what we can learn from this surprising movement (at least for
me) from a distribution (and a company, of
course), and what mistakes they are doing by keeping some things for
their own until they decide to unveil, as Mark said in his post.
Canonical and Ubuntu are both drifting away from the free
software (or even from the
open source) movement, and they are doing this
by adopting some ugly tactics like the one mentioned above. Free
software is a win-win game only if you respect its first rule:
freedom. This word has infinite meanings, but for
there are 4 basic rules (or freedoms) that should be obeyed:
The freedom to run the program, for any purpose (freedom 0).
The freedom to study how the program works, and change it so it
does your computing as you wish (freedom 1). Access to the source
code is a precondition for this.
The freedom to redistribute copies so you can help your neighbor
The freedom to distribute copies of your modified versions to
others (freedom 3). By doing this you can give the whole community
a chance to benefit from your changes. Access to the source code
is a precondition for this.
I won’t be polemical and say that Ubuntu is not obeying some of these
rules (though I could…). But it should be obvious that, by developing
something in secrecy, you are not respecting one’s freedom to decide
which way they think the software should behave (or not behave). In a
true free software community everyone can have a voice (though sometimes
people refuse to hear it, but that’s another problem which I’ll probably
talk about in another post); everyone can help making a decision, or can
try to influence some bad design choice (in his/her opinion). To
summarize, they can participate.
As much as I don’t like the way Ubuntu hardly gives something back to
the communities (if you compare to what they take from them), at
least their development model (or at least what I know from it) so far
was open. Every company has its sensible topics that need to be
discussed internally. Even Red Hat, the
company I work for (and now maybe you think this post is totally
biased…), does. But you should keep it to the really minimum, and
really discuss things in the open whenever you can. And you should
never do software development like the not-so-old-but-still-obscure
days: locked in your room, without external contact and feedback, trying
to guess what your users want, and still holding the freedom flag
sometimes. Sorry, but this is non-sense.
I tell you this: it is depressing when you realize that you spent more
time struggling with blog engines than writing posts on your blog!
It’s been a long time since I wrote
first post about
this subject, and since then the patches have been accepted upstream,
and GDB 7.5
now has official support for userspace SystemTap probes :-). Yay!
Well, but enough of cheap talk, let’s get to the business!
Errata for my last post
Frank Ch. Eigler, one of
SystemTap’s maintainers, kindly mentioned something that I should say
about SystemTap userspace probes.
Basically, it should be clear that SDT probes are not the only kind of
userspace probing one can do with SystemTap. There is yet another kind
of probe (maybe even more powerful, depending on the goals):
DWARF-based function/statement probes. SystemTap supports this kind
of probing mechanism for quite a while now.
It is not the goal of this post to explain it in detail, but you might
want to give it a try by compiling your binary with debuginfo support
(use the -g flag on GCC), and do something like:
And that’s it. You can read SystemTap’s documentation, or
guide to learn how to add userspace probes.
Using GDB with SystemTap SDT Probes
Well, now let’s get to the interesting part. It is time to make GDB
work with the SDT probe that we have put in our example code. Let’s
It is a very simple example, and we will have to extend it later in
order to show more features. But for now, it will do.
The first thing to do is to open GDB (with SystemTap support, of
course!), and check to see if it can actually see probe inserted in our
Wow, it actually works! :-)
If you have seen something like the above, it means your GDB is
correctly recognizing SDT probes. If you see an error, or if your
GDB doesn’t have the info probes command, then you’d better make
sure you have a recent version of GDB otherwise you won’t be able to
use the SDT support.
Putting breakpoints in the code
Anyway, now it is time to start using this support. The first thing I
want to show you is how to put a breakpoint in a probe.
That’s all! We have chosen to extend the break command in order to
support the new -probe-stap parameter. If you’re wondering … why
the -probe prefix?, it is because I was asked to implement a complete
abstraction layer inside GDB in order to allow more types of probes to
be added in the future. So, for example, if someone implements support
for an hypothetical type of probe called xyz, you would have
break -probe-xyz. It took me a little more time to implement this
layer, but it is worth the effort.
Anyway, as you have see above, GDB recognize the probe’s name and
correctly put a breakpoint in it. You can also confirm that it has done
the right thing by matching the address reported by info probes with
the one reported by break: they should be the same.
Ok, so now, with our breakpoint in place, let’s run the program and
see what happens.
As you can see, GDB stopped at the exact location of the probe.
Therefore, you are now able to put marks (i.e., probes) in your source
code which are location-independent. It means that it doesn’t really
matter where in the source code your probe is, and it also doesn’t
matter if you change the code around it, changing the line numbers, or
even moving it to another file. GDB will always find your probe, and
always stop at the right location. Neat!
Examining probes’ arguments
But wait, there’s more! Remember when I told you that you could also
inspect the probe’s arguments? Yes, let’s do it now!
Just remember that, in SDT’s parlance, the current probe’s argument is
a. So let’s print its value.
“Hey, captain, it seems the boat really floats!”
Check the source code above, and convince yourself that a’s value is
10 :-). As you might have seen, I have used a fairly strange way of
printing it. It is because the probe’s arguments are available inside
GDB by means of convenience variables. You can see a list of them
Since SDT probes can have up to 12 arguments (i.e., you can use
STAP_PROBE1 … STAP_PROBE12), we have created inside GDB 12
convenience variables, named $_probe_arg0 until $_probe_arg11. I
know, it is not an easy name to remember, and even the relation between
SDT naming and GDB naming is not direct (i.e., you have to subtract
1 from the SDT probe number). If you are not satisfied with this,
please open a bug in our bugzilla and
I promise we will discuss other options.
I would like to emphasize something here: just as you don’t need
debuginfo support for dealing with probes inside GDB, you also don’t
need debuginfo support for dealing with their arguments as well. It
means that you can actually compile your code without debuginfo support,
but still have access to some important variables/expressions when
debugging it. Depending on how GCC optimizes your code, you may
experience some difficulties with argument printing, but so far I
haven’t heard of anything like that.
More to come
Ok, now we have covered more things about the SDT probe support inside
GDB, and I hope you understood all the concepts. It is not hard to get
things going with this, specially because you don’t need extra libraries
to make it work.
In the next post, I intend to finish this series by explaining how to
use tracepoints with SDT probes. Also, as I said in the previous
post of this series, maybe I will talk a little bit about how the SDT
probes are organized within the binary.
TL;DR:Gostei de ter tido a oportunidade de dar as apresentações,
e principalmente de ter feito minha primeira palestra
do Projeto Fedora no Brasil. Sobre a
palestra a respeito do GDB, também gostei
do jeito que ela foi conduzida. Notei algumas falhas que precisam ser
corrigidas, mas no geral a experiência foi muito boa.
Apresentação “O Projeto Fedora”
Foi a primeira apresentação da noite, de acordo com a grade de
programação. Começou meia hora atrasada, pois a organização pediu para
esperarmos mais pessoas chegarem (estava chovendo bastante no momento, o
que dificultou a locomoção).
Comecei a palestra falando um pouco sobre o Projeto Fedora. Acabei
passando rapidamente pelas origens do projeto, uma falha que pretendo
corrigir em próximas ocasiões. Dei muita ênfase na definição de
comunidade e no que isso significa quando lidamos com software
livre. Confesso que fiz algumas comparações com o Ubuntu, o que talvez
não tenha sido uma boa idéia (de acordo com os guidelines do Projeto
Fedora para Embaixadores). De qualquer modo, a mensagem foi passada e
notei que algumas pessoas se interessaram em conhecer mais a respeito do
projeto e da filosofia.
Pontos positivos: Creio ter conseguido informar as pessoas a
respeito do projeto, com a ajuda dos ótimos slides do Paul W.
Frields. É sempre
gratificante dar palestras, mesmo que apenas uma ou duas pessoas no
final acabem se interessando de verdade. Além disso, me senti bem por
estar divulgando um projeto que respeita as liberdades dos usuários (ou
pelo menos tenta fazer isso ao máximo), e que eu realmente uso e gosto.
Pontos a serem melhorados: Fazer uma palestra um pouco menos
“pessoal”. É muito difícil conseguir isso, mas tenho a forte impressão
de que minha orientação totalmente pró-software-livre acaba (às vezes)
afastando algumas pessoas, que vêem no entusiasta por software livre uma
pessoa “radical” e “xiita”. Preciso pensar um pouco a respeito do
A conclusão é que fiquei bastante satisfeito com o resultado da
palestra. Percebi que, depois dela, algumas pessoas vieram comentar que
estavam utilizando Fedora, ou que já andavam pensando em trocar de
distribuição, que agora o Fedora era uma opção. O objetivo foi cumprido
Apresentação “GDB Crash Course”
Creio que essa já é a quarta vez que apresento essa palestra, e a
terceira vez junto com meu amigo
Edjunior. Sempre que ela termina,
fico(amos) com a impressão de que ainda não acertamos no ponto, e dessa
vez não foi diferente.
A palestra começou em ponto, às 21h, e decidimos tentar uma abordagem um
pouco diferente. A última vez que apresentamos a palestra foi no evento
da Semana Integrada da PUC
Campinas. Naquela ocasião, tínhamos
optado por começar falando mais sobre os comandos do GDB, e depois
mostrarmos como a coisa funciona, estilo hands-on. Dessa vez,
resolvemos ir mostrando a prática junto com a teoria. Ficou melhor, e
acho que a apresentação ficou mais fluida, mas ainda assim esbarramos no
velho problema da interdependência dos comandos: quando íamos falar
sobre breakpoints, precisávamos ter mostrado algum outro comando que
só iria ser explicado mais à frente, que por sua vez iria precisar de
outro comando, que iria precisar de breakpoints, etc. Enfim, no final
acabamos sendo obrigados a pular alguns comandos, e a adiantar a
explicação de outros, quebrando um pouco o fluxo dos slides.
Notei que algumas pessoas estavam bastante interessadas no GDB, talvez
por já programarem há algum tempo. As outras, aparentemente, ainda não
conseguiam ver muita utilidade para um depurador, mas mesmo assim
tentavam aprender algo que talvez fosse lhes servir no futuro.
Já era de se esperar, mas mesmo assim não deixo de me surpreender quando
vejo que uma palestra técnica consegue atrair muito mais atenção do que
uma palestra “filosófica”, como foi a do Projeto Fedora. Talvez seja
reflexo da sociedade em que vivemos, ou talvez seja apenas uma impressão
errônea da minha parte.
A conclusão, finalmente, é que a palestra parece ter sido útil para
algumas pessoas (mesmo que poucas), e isso nos dá ainda mais fôlego pra
continuarmos tentando divulgar esse projeto pouco conhecido (mas muito
útil) que é o GDB.
Não poderia deixar de agradecer primeiramente à organização da SECCOMP
da UNESP de Rio Claro pelo ótimo evento. Fiquei surpreso com a
infra-estrutura e, principalmente, com a receptividade das pessoas.
Gostei muito do ambiente descontraído, e espero não ter decepcionado
muita gente por lá com meus comentários informais e caipiras durante as
Também agradeço ao meu amigo Edjunior por ter me acompanhado até sua
alma matter para me ajudar na realização da palestra sobre o GDB.
A primeira palestra será sobre o Projeto
Fedora. Vai ser a primeira vez que
falarei sobre o projeto depois de ter me tornado
Embaixador do Fedora no
Brasil. Confesso que estou um pouco apreensivo, mas escolhi slides muito
bons feitos pelo Paul W.
Frields, ex-líder do
Projeto e bastante competente em suas apresentações. Pretendo fazer um
relato sobre a palestra na quarta-feira.