I tell you this: it is depressing when you realize that you spent more time struggling with blog engines than writing posts on your blog!

It's been a long time since I wrote the first post about this subject, and since then the patches have been accepted upstream, and GDB 7.5 now has official support for userspace SystemTap probes :-). Yay!

Well, but enough of cheap talk, let's get to the business!

Errata for my last post

Frank Ch. Eigler, one of SystemTap's maintainers, kindly mentioned something that I should say about SystemTap userspace probes.

Basically, it should be clear that SDT probes are not the only kind of userspace probing one can do with SystemTap. There is yet another kind of probe (maybe even more powerful, depending on the goals): DWARF-based function/statement probes. SystemTap supports this kind of probing mechanism for quite a while now.

It is not the goal of this post to explain it in detail, but you might want to give it a try by compiling your binary with debuginfo support (use the -g flag on GCC), and do something like:

1
2
$ stap -e 'probe process("/bin/foo").function("name") { log($$parms) }' -c /bin/foo
$ stap -e 'probe process("/bin/foo").statement("*@file.c:443") { log($$vars) }' -c /bin/foo

And that's it. You can read SystemTap's documentation, or this guide to learn how to add userspace probes.

Using GDB with SystemTap SDT Probes

Well, now let's get to the interesting part. It is time to make GDB work with the SDT probe that we have put in our example code. Let's remember it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
#include <sys/sdt.h>

int
main (int argc, char *argv[])
{
  int a = 10;

  STAP_PROBE1 (test_program, my_probe, a);

  return 0;
}

It is a very simple example, and we will have to extend it later in order to show more features. But for now, it will do.

The first thing to do is to open GDB (with SystemTap support, of course!), and check to see if it can actually see probe inserted in our example.

1
2
3
4
5
6
7
8
$ gdb ./test_program
GNU gdb (GDB) 7.5.50.20121014-cvs
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
...
(gdb) info probes
Provider     Name     Where              Semaphore Object
test_program my_probe 0x00000000004004ae           /home/sergio/work/src/git/build/gdb/test_program

Wow, it actually works! :-)

If you have seen something like the above, it means your GDB is correctly recognizing SDT probes. If you see an error, or if your GDB doesn't have the info probes command, then you'd better make sure you have a recent version of GDB otherwise you won't be able to use the SDT support.

Putting breakpoints in the code

Anyway, now it is time to start using this support. The first thing I want to show you is how to put a breakpoint in a probe.

1
2
(gdb) break -probe-stap my_probe
Breakpoint 1 at 0x4004ae

That's all! We have chosen to extend the break command in order to support the new -probe-stap parameter. If you're wondering ... why the -probe prefix?, it is because I was asked to implement a complete abstraction layer inside GDB in order to allow more types of probes to be added in the future. So, for example, if someone implements support for an hypothetical type of probe called xyz, you would have break -probe-xyz. It took me a little more time to implement this layer, but it is worth the effort.

Anyway, as you have see above, GDB recognize the probe's name and correctly put a breakpoint in it. You can also confirm that it has done the right thing by matching the address reported by info probes with the one reported by break: they should be the same.

Ok, so now, with our breakpoint in place, let's run the program and see what happens.

1
2
3
4
5
(gdb) run
Starting program: /home/sergio/work/src/git/build/gdb/test_program

Breakpoint 1, main (argc=1, argv=0x7fffffffdf68) at /tmp/example-stap.c:8
8  STAP_PROBE1 (test_program, my_probe, a);

As you can see, GDB stopped at the exact location of the probe. Therefore, you are now able to put marks (i.e., probes) in your source code which are location-independent. It means that it doesn't really matter where in the source code your probe is, and it also doesn't matter if you change the code around it, changing the line numbers, or even moving it to another file. GDB will always find your probe, and always stop at the right location. Neat!

Examining probes' arguments

But wait, there's more! Remember when I told you that you could also inspect the probe's arguments? Yes, let's do it now!

Just remember that, in SDT's parlance, the current probe's argument is a. So let's print its value.

1
2
3
4
(gdb) p $_probe_arg0
$1 = 10
(gdb) p a
$2 = 10

"Hey, captain, it seems the boat really floats!"

Check the source code above, and convince yourself that a's value is 10 :-). As you might have seen, I have used a fairly strange way of printing it. It is because the probe's arguments are available inside GDB by means of convenience variables. You can see a list of them here.

Since SDT probes can have up to 12 arguments (i.e., you can use STAP_PROBE1 ... STAP_PROBE12), we have created inside GDB 12 convenience variables, named $_probe_arg0 until $_probe_arg11. I know, it is not an easy name to remember, and even the relation between SDT naming and GDB naming is not direct (i.e., you have to subtract 1 from the SDT probe number). If you are not satisfied with this, please open a bug in our bugzilla and I promise we will discuss other options.

I would like to emphasize something here: just as you don't need debuginfo support for dealing with probes inside GDB, you also don't need debuginfo support for dealing with their arguments as well. It means that you can actually compile your code without debuginfo support, but still have access to some important variables/expressions when debugging it. Depending on how GCC optimizes your code, you may experience some difficulties with argument printing, but so far I haven't heard of anything like that.

More to come

Ok, now we have covered more things about the SDT probe support inside GDB, and I hope you understood all the concepts. It is not hard to get things going with this, specially because you don't need extra libraries to make it work.

In the next post, I intend to finish this series by explaining how to use tracepoints with SDT probes. Also, as I said in the previous post of this series, maybe I will talk a little bit about how the SDT probes are organized within the binary.

See you soon!


Conforme eu havia comentado no post anterior, segue o relato sobre as apresentações que fiz na Semana da Computação da UNESP de Rio Claro.

TL;DR: Gostei de ter tido a oportunidade de dar as apresentações, e principalmente de ter feito minha primeira palestra como Embaixador do Projeto Fedora no Brasil. Sobre a palestra a respeito do GDB, também gostei do jeito que ela foi conduzida. Notei algumas falhas que precisam ser corrigidas, mas no geral a experiência foi muito boa.

Apresentação "O Projeto Fedora"

Foi a primeira apresentação da noite, de acordo com a grade de programação. Começou meia hora atrasada, pois a organização pediu para esperarmos mais pessoas chegarem (estava chovendo bastante no momento, o que dificultou a locomoção).

Comecei a palestra falando um pouco sobre o Projeto Fedora. Acabei passando rapidamente pelas origens do projeto, uma falha que pretendo corrigir em próximas ocasiões. Dei muita ênfase na definição de comunidade e no que isso significa quando lidamos com software livre. Confesso que fiz algumas comparações com o Ubuntu, o que talvez não tenha sido uma boa idéia (de acordo com os guidelines do Projeto Fedora para Embaixadores). De qualquer modo, a mensagem foi passada e notei que algumas pessoas se interessaram em conhecer mais a respeito do projeto e da filosofia.

Pontos positivos: Creio ter conseguido informar as pessoas a respeito do projeto, com a ajuda dos ótimos slides do Paul W. Frields. É sempre gratificante dar palestras, mesmo que apenas uma ou duas pessoas no final acabem se interessando de verdade. Além disso, me senti bem por estar divulgando um projeto que respeita as liberdades dos usuários (ou pelo menos tenta fazer isso ao máximo), e que eu realmente uso e gosto.

Pontos a serem melhorados: Fazer uma palestra um pouco menos "pessoal". É muito difícil conseguir isso, mas tenho a forte impressão de que minha orientação totalmente pró-software-livre acaba (às vezes) afastando algumas pessoas, que vêem no entusiasta por software livre uma pessoa "radical" e "xiita". Preciso pensar um pouco a respeito do assunto...

A conclusão é que fiquei bastante satisfeito com o resultado da palestra. Percebi que, depois dela, algumas pessoas vieram comentar que estavam utilizando Fedora, ou que já andavam pensando em trocar de distribuição, que agora o Fedora era uma opção. O objetivo foi cumprido :-).

Apresentação "GDB Crash Course"

Creio que essa já é a quarta vez que apresento essa palestra, e a terceira vez junto com meu amigo Edjunior. Sempre que ela termina, fico(amos) com a impressão de que ainda não acertamos no ponto, e dessa vez não foi diferente.

A palestra começou em ponto, às 21h, e decidimos tentar uma abordagem um pouco diferente. A última vez que apresentamos a palestra foi no evento da Semana Integrada da PUC Campinas. Naquela ocasião, tínhamos optado por começar falando mais sobre os comandos do GDB, e depois mostrarmos como a coisa funciona, estilo hands-on. Dessa vez, resolvemos ir mostrando a prática junto com a teoria. Ficou melhor, e acho que a apresentação ficou mais fluida, mas ainda assim esbarramos no velho problema da interdependência dos comandos: quando íamos falar sobre breakpoints, precisávamos ter mostrado algum outro comando que só iria ser explicado mais à frente, que por sua vez iria precisar de outro comando, que iria precisar de breakpoints, etc. Enfim, no final acabamos sendo obrigados a pular alguns comandos, e a adiantar a explicação de outros, quebrando um pouco o fluxo dos slides.

Notei que algumas pessoas estavam bastante interessadas no GDB, talvez por já programarem há algum tempo. As outras, aparentemente, ainda não conseguiam ver muita utilidade para um depurador, mas mesmo assim tentavam aprender algo que talvez fosse lhes servir no futuro.

Já era de se esperar, mas mesmo assim não deixo de me surpreender quando vejo que uma palestra técnica consegue atrair muito mais atenção do que uma palestra "filosófica", como foi a do Projeto Fedora. Talvez seja reflexo da sociedade em que vivemos, ou talvez seja apenas uma impressão errônea da minha parte.

A conclusão, finalmente, é que a palestra parece ter sido útil para algumas pessoas (mesmo que poucas), e isso nos dá ainda mais fôlego pra continuarmos tentando divulgar esse projeto pouco conhecido (mas muito útil) que é o GDB.

Agradecimentos

Não poderia deixar de agradecer primeiramente à organização da SECCOMP da UNESP de Rio Claro pelo ótimo evento. Fiquei surpreso com a infra-estrutura e, principalmente, com a receptividade das pessoas. Gostei muito do ambiente descontraído, e espero não ter decepcionado muita gente por lá com meus comentários informais e caipiras durante as palestras :-).

Também agradeço ao meu amigo Edjunior por ter me acompanhado até sua alma matter para me ajudar na realização da palestra sobre o GDB.

Até a próxima!


Hoje, dia 23/10/2012, estarei na UNESP de Rio Claro para dar duas apresentações na Semana da Computação.

A primeira palestra será sobre o Projeto Fedora. Vai ser a primeira vez que falarei sobre o projeto depois de ter me tornado Embaixador do Fedora no Brasil. Confesso que estou um pouco apreensivo, mas escolhi slides muito bons feitos pelo Paul W. Frields, ex-líder do Projeto e bastante competente em suas apresentações. Pretendo fazer um relato sobre a palestra na quarta-feira.

A segunda apresentação será sobre o GDB. Essa apresentação vai ser mais um crash course sobre como utilizar a ferramenta, e os slides estão disponíveis em https://github.com/sergiodj/gdb-unicamp2011.

Espero que ambas as palestras sejam bem recebidas pelo público! Volto depois pra contar como foi :-).

Abraços.


First Blog post, again...


Tags:

Well, well. I really wasn't entirely satisfied with Octopress, so I decided to change the blog engine yet another time!! Here it is, using blohg, a blog engine based on Mercurial (yeah, I know, nothing is perfect), and made by a Brazilian :-).

I intend to resume my activities in the blog pretty soon, especially because I have many things to share here. First of all, I will convert the old posts from Markup to rst.

Hope you like it!


After a long time, here we are again :-).

With this post I will start to talk about the integration between GDB and SystemTap. This is something that Tom Tromey and I did during the last year. The patch is being reviewed as I write this post, and I expect to see it checked-in in the next few days/weeks. But let's get our hands dirty...

SystemTap Userspace Probes

You probably use (or have at least heard of) SystemTap, and maybe you think the tool is only useful for kernel inspections. If that's your case, I have a good news: you're wrong! You can actually use SystemTap to inspect userspace applications too, by using what we call SDT probes, or Static Defined Tracing probes. This is a very cheap and easy way to include probes in your application, and you can even specify arguments to those probes.

In order to use the probes (see an example below), you must include the <sys/sdt.h> header file in your source code. If you are using Fedora systems, you can obtain this header file by installing the package systemtap-sdt-devel, version equal or greater than 1.4.

Here's a simple example of an application with a one-argument probe:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
#include <sys/sdt.h>

int
main (int argc, char *argv[])
{
  int a = 10;

  STAP_PROBE1 (test_program, my_probe, a);

  return 0;
}

As you can see, this is a very simple program with one probe, which contains one argument. You can now compile the program:

1
$ gcc test_program.c -o test_program

Now you must be thinking: "Wait, wait... Didn't you just forget to link this program against some SystemTap-specific library or something?" And my answer is no. One of the spetacular things about this <sys/sdt.h> header is that it does not have any dependencies at all! As Tom said in his blog post, this is "a virtuoso display of ELF and GCC asm wizardy".

If you want to make sure your probe was inserted in the binary, you can use readelf command:

1
2
3
4
5
6
7
8
$ readelf -x .note.stapsdt ./test_program

Hex dump of section '.note.stapsdt':
  0x00000000 08000000 3a000000 03000000 73746170 ....:.......stap
  0x00000010 73647400 86044000 00000000 88054000 sdt...@.......@.
  0x00000020 00000000 00000000 00000000 74657374 ............test
  0x00000030 5f70726f 6772616d 006d795f 70726f62 _program.my_prob
  0x00000040 65002d34 402d3428 25726270 29000000 e.-4@-4(%rbp)...

(I will think about writing an explanation on how the probes are laid out on the binary, but for now you just have to care if you actually see an output from this readelf command.)

You can also use SystemTap to perform this verification:

1
2
$ stap -L 'process("./test_program").mark("*")'
process("./test_program").mark("my_probe") $arg1:long

So far, so good. If you see an output like the one above, it means your probe is correctly inserted. You could obviously use SystemTap to inspect this probe, but I won't do this right now because this is not the purpose of this post.

For now, we have learned how to:

  1. Include an SDT probe in our source code, and compile it;
  2. Verify if the probe was correctly inserted.

In the next post, I will talk about the GDB support that allows you to inspect, print arguments, and gather other information about SDT probes. I hope you like it!


This post is actually a "reply" to Gary Benson's Working on gdb post.

I have been working with GDB for quite some time now, and even though the project officially uses CVS (yes, you read it correctly, it is CVS indeed!) as its version control system, fortunately we also have a git mirror. In the end, what happens is that almost every developer uses the git mirror and just goes to CVS to commit something. But this is another discussion. Aside of this git mirror, we also have the Archer repository (which uses git by default).

My plan here is to show you how I do my daily work with GDB. The workflow is pretty simple, but maybe you will see something here that might help you.

Checking out the code

The first thing to do is to check out the code. I only have one GDB repository here, and I make branches out of it whenever I want to hack. So, to check out (or clone, in git's parlance) the code, I do (or did):

With this, we have just cloned the GDB repository, and also added another remote (i.e., repository). This is useful because we might want to hack on a branch which is on Archer, but use GDB's master branch as a base.

Create a new branch for your work

So, now it's time to create a new branch for you. Here I use one of my little "tricks" (taught to me by my friend Dodji), which is the command git-new-workdir. This is a nice command because it creates a new working directory for your project!

Maybe you're wondering why this is so cool. Well, if you ever worked with git, and more specifically, if you ever used more than one branch at a time, then maybe you will understand my excitement. In this scenario, having to constantly switch between the branches is not something rare. When you have uncommited work in your tree you can always use git stash, but that is not the ideal solution (for me). Sometimes I would forget what was on the stash, and later when I checked it, it was full of crap. Also, I like to have a separate directory for every project I am working on.

It is also important to mention that git-new-workdir is under the directory /usr/share/doc/git-VERSION/contrib/workdir/, so I created an alias that will automagically call the script for me:

So, after setting up the script, here is what I do:

Build GDB

In order to build the project, I create a build-64 directory inside my project directory (which, in the example above, is work/lazy-debuginfo-reading).

GDB fortunately supports VPATH building (i.e., build the project outside of the source tree). I strongly recommend you to use it.

As you may have noticed, I use -g3 (include debuginfo) and -O0 (do not optimize the code) in CFLAGS. Also, since some of the features I work on may affect code in other architectures, I use --enable-targets=all. It will tell configure to compile everything related to all architectures (not only x86_64, for example). At last, I specify a separate debug directory which GDB should use to search for debuginfo files.

Finalizing (for now)

After that, you will have a fresh GDB binary compiled in the build-64 directory. But that is not enough yet, since you will also want to test GDB and make sure you didn't insert a bug while hacking on it. In my next post, I will explain what is my "testflow". I hope it will be useful for someone :-).

Stay tuned!