Relato da Apresentação sobre o GDB no SoLiSC 2012

Nesta última sexta-feira, dia 30/11/2012, estive presente na sétima edição do SoLiSC 2012, em Florianópolis, para apresentar uma palestra introdutória sobre o GDB. Este é um relato sobre minha particição no evento :-).

Impressões sobre o evento

Foi a primeira vez que fui ao SoLiSC. Já tive vontade de ir em anos anteriores, mas infelizmente sempre havia algo para atrapalhar. No entanto, nesse ano felizmente tudo correu bem, e inclusive tive uma palestra aceita! Ou seja, um ótimo motivo para visitar Floripa e rever o mar :-D.

Peguei um vôo saindo às 6h de Campinas, e cheguei lá às 7h10min. Estava bastante cansado, pois não havia dormido de quinta pra sexta, só que a ansiedade estava conseguindo me deixar ligado :-).

O evento aconteceu Universidade Estácio de Sá, que fica em São José. Cheguei por lá às 8h, e fui bem recebido pelo pessoal do evento. Já tentei me enturmar, e conheci algumas pessoas que também iam palestrar no evento. Como minha palestra estava marcada para começar às 14h, resolvi ficar batendo papo e de olho na grade de palestras.

Por coincidência (ou não!), acabei ficando na sala onde aconteceria o primeiro LibreOffice Hack Day no Brasil. Acabei ficando na sala o dia todo, ajudando o pessoal a resolver alguns problemas chatos com o firewall da Universidade, e depois com git. Foi uma experiência muito legal, nunca tinha participado de um Hack Day antes, e foi uma honra poder presenciar e ajudar no primeiro evento do tipo que o pessoal do LibreOffice fez no Brasil :-). Aliás, foi muito interessante conhecer um pouco mais sobre um projeto tão grande e complexo quanto o LibreOffice, e inclusive fiz um “jabá” sobre o GDB para eles :-).

No final, também conheci algumas pessoas muito interessadas em contribuir com projetos de software livre, o que é sempre bom! Isso me ajuda a ter mais motivação para continuar a fazer esse trabalho de divulgação. Você pode ler uma descrição mais detalhada sobre o LibreOffice Hack Day (inclusive com fotos) aqui.

Apresentação “GDB Crash Course”

Eu já estava esperando pouca gente na palestra, até porque falar sobre o GDB está ficando cada vez mais complicado… As pessoas em geral não sabem (e nem se interessam) pelo software, então é normal ficar meio “de escanteio” nesses eventos :-). Quem sabe um dia eu não escreva um post sobre isso?

Bem, mas mesmo com pouco público, creio que palestra correu bem. Dessa vez, meu amigo Edjunior não foi, então levei a palestra sozinho :-). Existem vantagens e desvantagens nisso, mas de modo geral acho que a palestra ficou um pouco mais rápida.

Adicionei alguns slides extras para falar sobre a Red Hat, e sobre o que estamos fazendo pelas comunidades de software livre por aí – não só na do GDB, mas também em muitas outras. Essa parte da apresentação realmente foi bacana, porque o orgulho de se trabalhar nessa empresa é grande!

Depois que terminei minha palestra e voltei à sala do LibreOffice Hack Day, alguns desenvolvedores que estavam por lá me perguntaram como foi, e disseram que tinham se arrependido de não ter ido… Sabe como é, preferiram ficar fazendo patches, então eu entendo :-P. Bem, pra não deixar ninguém insatisfeito, acabei fazendo uma segunda rodada da palestra dentro do Hack Day, e também foi muito bacana :-).

Várias pessoas me pediram os slides, então aqui estão eles:

Conclusão

Gostaria de agradecer especialmente à Eliane Domingos, ao David Jourdain e ao Olivier Hallot, todos membros da TDF e contribuidores do LibreOffice, pelos momentos prazerosos e pelas conversas divertidas que tivemos durante todo o evento!

Também gostaria de agradecer à organização do SoLiSC pela oportunidade de participar de um evento tão bacana! O Klaibson Ribeiro foi a pessoa com quem troquei alguns e-mails antes do evento, então um “muito obrigado” a ele também :-).

Nos vemos no próximo SoLiSC!


HEADS UP: Comment system is offline temporarily

Hi there. This little post is just a heads about an issue that I am facing with the comment system that I run. Unfortunately, you will not be able to post comments on the blog until, at least, next Wednesday (November 21).

For those of you wondering which comment system I use, it is called Juvia. Due to privacy concerns, I chose not to use anything like Disqus because it tracks you and your comments (read their privacy policy if you want more details). On the other hand, Juvia runs on my private personal server, and does not collect any kind of personal information when you make a comment. The cons of this approach is that when my personal server is down (like now), the blog doesn’t have comments. But that’s a minor price to pay for the respect of privacy, I think.

Anyway, I hope to have the comments back online next week. Until there, I plan to continue making posts here, so save your comments for some time!

Thanks!


GDB and SystemTap Probes -- part 3

Hi everybody :-).

I finally got some time to finish this series of posts, and I hope you like the overall result. For those of you who are reading this blog for the first time, you can access the first post here, and the second here.

My goal with this third post is to talk a little bit about how you can use the SDT probes with tracepoints inside GDB. Maybe this particular feature will not be so helpful to you, but I recommend reading the post either way. I will also give a brief explanation about how the SDT probes are laid out inside the binary. So, let’s start!

Complementary information

In my last post, I forgot to mention that the SDT probe support present on older versions of Fedora GDB is not exactly as the way I described here. This is because Fedora GDB adopted this feature much earlier than upstream GDB itself, so while this has a great positive aspect in terms of how the distro’s philosophy works (i.e., Fedora contains leading-edge features, so if you want to know how to FLOSS community will be in a few months, use it!), it also has the downside of delivering older/different versions of features in older Fedoras. But of course, this SDT feature will be fully available on Fedora 18, to be announced soon.

My suggestion is that if you use a not-so-recent Fedora (like Fedora 16, 15, etc), please upgrade it to the last version, or compile your own version of GDB yourself (it’s not that hard, I will make a post about it in the next days/weeks!).

With that said, let’s move on to our main topic here.

SDT Probes and Tracepoint

Before anything else, let me explain what a tracepoint is. Think of it as a breakpoint which doesn’t stop the program’s execution when it hits. In fact, it’s a bit more than that: you can define actions associated with a tracepoint, and those actions will be performed when the tracepoint is hit. Neat, huh? :-)

There is a nice description of what a tracepoint in the GDB documentation, I recommend you give it a reading to understand the concept.

Ok, so now we have to learn how to put tracepoints in our code, and how to define actions for them. But before that, let’s remember our example program:

#include <sys/sdt.h>

int
main (int argc, char *argv[])
{
  int a = 10;

  STAP_PROBE1 (test_program, my_probe, a);

  return 0;
}

Very simple, isn’t it? Ok, to the tracepoints now, my friends.

Using tracepoints inside GDB

In order to properly use tracepoints inside GDB, you will need to use gdbserver, a tiny version of GDB suitable for debugging programs remotely, over the net or serial line. In short, this is because GDB cannot put tracepoints on a program running directly under it, so we have to run it inside gdbserver and then connect GDB to it.

Running our program inside gdbserver

In our case, we will just start gdbserver in our machine, order it to listen to some high port, and connect to it through localhost, so there will be no need to have access to another computer or device.

First of all, make sure you have gdbserver installed. If you use Fedora, the package name you will have to install is gdb-gdbserver. If you have it installed, you can do:

$ gdbserver :3001 ./test_program
Process ./test_program created; pid = 17793
Listening on port 3001

The second argument passed to gdbserver instructs it to listen on the port 3001 of your loopback interface, a.k.a. localhost.

You will notice that gdbserver will stay there indefinitely, waiting for new connections to arrive. Don’t worry, we will connect to it soon!

Connecting an instance of GDB to gdbserver

Now, go to another terminal and start GDB with our program:

$ gdb ./test_program
...
(gdb) target remote :3001
Remote debugging using :3001
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x0000003d60401530 in _start () from /lib64/ld-linux-x86-64.so.2

The command you have to use inside GDB is target remote. It takes as an argument the host and the port to which you want to connect. In our case, we just want it to connect to localhost, port 3001. If you saw an output like the above, great, things are working for you (don’t pay attention to the messages about glibc debug information). If you didn’t see it, please check to see if you’re connecting to the right port, and if no other service is using it.

Ok, so now it is time to start our trace experiment!

Creating the tracepoints

Every command should be issued on GDB, not on gdbserver!

In your GDB prompt, put a tracepoint in the probe named my_probe:

(gdb) trace -probe-stap my_probe
Tracepoint 1 at 0x4005a9

As you can see, the trace command takes exactly the same arguments as the break command. Thus, you need to use the -probe-stap modified in order to instruct GDB to put the tracepoint in the probe.

And now, let’s define the actions associated with this tracepoint. To do that, we use the actions command, which is an interactive command inside GDB. It takes some specific keywords, and if you want to learn more about it, please take a look at this link. For this example, we will use only the collect keyword, which tells GDB to… hm… collect something :-). In our case, it will collect the probe’s first argument, or $_probe_arg0, as you may remember.

(gdb) actions 
Enter actions for tracepoint 1, one per line.
End with a line saying just "end".
>collect $_probe_arg0
>end
(gdb)

Simple as that. Finally, we have to define a breakpoint in the last instruction of our program, because it is necessary to keep it running on gdbserver in order to examine the tracepoints later. If we didn’t put this breakpoint, our program would finish and gdbserver would not be able to provide information about what happened with our tracepoints. In our case, we will simply put a breakpoint on line 10, i.e., on the return 0;:

Running the trace experiment

Ok, time to run our trace experiment. First, we must issue a tstart to tell GDB to start monitoring the tracepoints. And then, we can continue our program normally.

(gdb) tstart 
(gdb) continue
Continuing.

Breakpoint 1, main (argc=1, argv=0x7fffffffde88) at /tmp/test_program.c:10
10        return 0;
(gdb) tstop
(gdb)

Remember, GDB is not going to stop your program, because tracepoints are designed to not interfere with the execution of it. Also notice that we have also stopped the trace experiment after the breakpoint hit, by using the tstop command.

Now, we will be able to examine what the tracepoint has collected. First, we will the tfind command to make sure the tracepoint has hit, and then we can inspect what we ordered it to collect:

(gdb) tfind start
Found trace frame 0, tracepoint 1
8         STAP_PROBE1 (test_program, my_probe, a);
(gdb) p $_probe_arg0
$1 = 10

And it works! Notice that we are printing the probe argument using the same notation as with breakpoints, even though we are not exactly executing the STAP_PROBE1 instruction. What does it mean? Well, with the tfind start command we tell GDB to actually use the trace frame collected during the program’s execution, which, in this case, is the probe argument. If you know GDB, think of it as if we were using the frame command to jump back to a specific frame, where we would have access to its state.

This is a very simple example of how to use the SDT probe support in GDB with tracepoints. There is much more you can do, but I hope I could explain the basics so that you can start playing with this feature.

How the SDT probe is laid out in the binary

You might be interested in learning how the probes are created inside the binary. Other than reading the source code of /usr/include/sys/sdt.h, which is the heart of the whole feature, I also recommend this page, which explains in detail what’s going on under the hood. I also recommend that you study a little about how the ELF format works, specifically about notes in the ELF file.

Conclusion

After this series of blog posts, I expect that you will now be able to use the not-so-new feature of SDT probe support on GDB. Of course, if you find some bug while using this, please feel free to report it using our bugzilla. And if you have some question, use the comment system below and I will answer ASAP :-).

See ya, and thanks for reading!


Mark Shuttleworth and the secrecy of Ubuntu

In the last days, there was some fuzz about Mark Shuttleworth’s post about some news in Ubuntu. Well, to be more precise, the post was about some secrets regarding the development of some applications. In my opinion, it summarizes some of what this distribution has become, and I would like to talk a little bit about it here.

My goal is not say bad things about Ubuntu (though I won’t promise that, because the post is intended as a critic), nor about Mr. Shuttleworth, whom I don’t know personally and have nothing against. My purpose is to discuss what we can learn from this surprising movement (at least for me) from a distribution (and a company, of course), and what mistakes they are doing by keeping some things for their own until they decide to unveil, as Mark said in his post.

Canonical and Ubuntu are both drifting away from the free software (or even from the open source) movement, and they are doing this by adopting some ugly tactics like the one mentioned above. Free software is a win-win game only if you respect its first rule: freedom. This word has infinite meanings, but for FLOSS there are 4 basic rules (or freedoms) that should be obeyed:

  1. The freedom to run the program, for any purpose (freedom 0).
  2. The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this.
  3. The freedom to redistribute copies so you can help your neighbor (freedom 2).
  4. The freedom to distribute copies of your modified versions to others (freedom 3). By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.

I won’t be polemical and say that Ubuntu is not obeying some of these rules (though I could…). But it should be obvious that, by developing something in secrecy, you are not respecting one’s freedom to decide which way they think the software should behave (or not behave). In a true free software community everyone can have a voice (though sometimes people refuse to hear it, but that’s another problem which I’ll probably talk about in another post); everyone can help making a decision, or can try to influence some bad design choice (in his/her opinion). To summarize, they can participate.

As much as I don’t like the way Ubuntu hardly gives something back to the communities (if you compare to what they take from them), at least their development model (or at least what I know from it) so far was open. Every company has its sensible topics that need to be discussed internally. Even Red Hat, the company I work for (and now maybe you think this post is totally biased…), does. But you should keep it to the really minimum, and really discuss things in the open whenever you can. And you should never do software development like the not-so-old-but-still-obscure days: locked in your room, without external contact and feedback, trying to guess what your users want, and still holding the freedom flag sometimes. Sorry, but this is non-sense.


GDB and SystemTap Probes -- part 2

I tell you this: it is depressing when you realize that you spent more time struggling with blog engines than writing posts on your blog!

It’s been a long time since I wrote the first post about this subject, and since then the patches have been accepted upstream, and GDB 7.5 now has official support for userspace SystemTap probes :-). Yay!

Well, but enough of cheap talk, let’s get to the business!

Errata for my last post

Frank Ch. Eigler, one of SystemTap’s maintainers, kindly mentioned something that I should say about SystemTap userspace probes.

Basically, it should be clear that SDT probes are not the only kind of userspace probing one can do with SystemTap. There is yet another kind of probe (maybe even more powerful, depending on the goals): DWARF-based function/statement probes. SystemTap supports this kind of probing mechanism for quite a while now.

It is not the goal of this post to explain it in detail, but you might want to give it a try by compiling your binary with debuginfo support (use the -g flag on GCC), and do something like:

$ stap -e 'probe process("/bin/foo").function("name") { log($$parms) }' -c /bin/foo
$ stap -e 'probe process("/bin/foo").statement("*@file.c:443") { log($$vars) }' -c /bin/foo

And that’s it. You can read SystemTap’s documentation, or this guide to learn how to add userspace probes.

Using GDB with SystemTap SDT Probes

Well, now let’s get to the interesting part. It is time to make GDB work with the SDT probe that we have put in our example code. Let’s remember it:

#include <sys/sdt.h>

int
main (int argc, char *argv[])
{
  int a = 10;

  STAP_PROBE1 (test_program, my_probe, a);

  return 0;
}

It is a very simple example, and we will have to extend it later in order to show more features. But for now, it will do.

The first thing to do is to open GDB (with SystemTap support, of course!), and check to see if it can actually see probe inserted in our example.

$ gdb ./test_program
GNU gdb (GDB) 7.5.50.20121014-cvs
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
...
(gdb) info probes
Provider     Name     Where              Semaphore Object
test_program my_probe 0x00000000004004ae           /home/sergio/work/src/git/build/gdb/test_program

Wow, it actually works! :-)

If you have seen something like the above, it means your GDB is correctly recognizing SDT probes. If you see an error, or if your GDB doesn’t have the info probes command, then you’d better make sure you have a recent version of GDB otherwise you won’t be able to use the SDT support.

Putting breakpoints in the code

Anyway, now it is time to start using this support. The first thing I want to show you is how to put a breakpoint in a probe.

(gdb) break -probe-stap my_probe
Breakpoint 1 at 0x4004ae

That’s all! We have chosen to extend the break command in order to support the new -probe-stap parameter. If you’re wondering … why the -probe prefix?, it is because I was asked to implement a complete abstraction layer inside GDB in order to allow more types of probes to be added in the future. So, for example, if someone implements support for an hypothetical type of probe called xyz, you would have break -probe-xyz. It took me a little more time to implement this layer, but it is worth the effort.

Anyway, as you have see above, GDB recognize the probe’s name and correctly put a breakpoint in it. You can also confirm that it has done the right thing by matching the address reported by info probes with the one reported by break: they should be the same.

Ok, so now, with our breakpoint in place, let’s run the program and see what happens.

(gdb) run
Starting program: /home/sergio/work/src/git/build/gdb/test_program

Breakpoint 1, main (argc=1, argv=0x7fffffffdf68) at /tmp/example-stap.c:8
8  STAP_PROBE1 (test_program, my_probe, a);

As you can see, GDB stopped at the exact location of the probe. Therefore, you are now able to put marks (i.e., probes) in your source code which are location-independent. It means that it doesn’t really matter where in the source code your probe is, and it also doesn’t matter if you change the code around it, changing the line numbers, or even moving it to another file. GDB will always find your probe, and always stop at the right location. Neat!

Examining probes’ arguments

But wait, there’s more! Remember when I told you that you could also inspect the probe’s arguments? Yes, let’s do it now!

Just remember that, in SDT’s parlance, the current probe’s argument is a. So let’s print its value.

(gdb) p $_probe_arg0
$1 = 10
(gdb) p a
$2 = 10

“Hey, captain, it seems the boat really floats!”

Check the source code above, and convince yourself that a’s value is 10 :-). As you might have seen, I have used a fairly strange way of printing it. It is because the probe’s arguments are available inside GDB by means of convenience variables. You can see a list of them here.

Since SDT probes can have up to 12 arguments (i.e., you can use STAP_PROBE1STAP_PROBE12), we have created inside GDB 12 convenience variables, named $_probe_arg0 until $_probe_arg11. I know, it is not an easy name to remember, and even the relation between SDT naming and GDB naming is not direct (i.e., you have to subtract 1 from the SDT probe number). If you are not satisfied with this, please open a bug in our bugzilla and I promise we will discuss other options.

I would like to emphasize something here: just as you don’t need debuginfo support for dealing with probes inside GDB, you also don’t need debuginfo support for dealing with their arguments as well. It means that you can actually compile your code without debuginfo support, but still have access to some important variables/expressions when debugging it. Depending on how GCC optimizes your code, you may experience some difficulties with argument printing, but so far I haven’t heard of anything like that.

More to come

Ok, now we have covered more things about the SDT probe support inside GDB, and I hope you understood all the concepts. It is not hard to get things going with this, specially because you don’t need extra libraries to make it work.

In the next post, I intend to finish this series by explaining how to use tracepoints with SDT probes. Also, as I said in the previous post of this series, maybe I will talk a little bit about how the SDT probes are organized within the binary.

See you soon!