Tag: free-software (subscribe)

It is really nice to see something you did in a project influence in future features and developments. I always feel happy and proud when I notice such scenarios happening, and this time was no different. Gary Benson, a colleague at Red Hat who works in the GDB team as well, has implemented a way of improving the interface between the linker and the debugger, and one of the things he used to achieve this is the GDB <-> SystemTap integration that I implemented with Tom Tromey 2 years ago. Neat!

The problem

You can read a detailed description of the problem in the message Gary sent to the gdb-patches mailing list, but to summarize: GDB needs to interface with the linker in order to identify which shared libraries were loaded during the inferior's (i.e., program being debugged) life.

Nowadays, what GDB does is to put a breakpoint in _dl_debug_state, which is an empty function called by the linker every time a shared library is loaded (the linker calls it twice, once before modifying the list of loaded shlibs, and once after). But GDB has no way to know what has changed in the list of loaded shlibs, and therefore it needs to load the entire list every time something happens. You can imagine how bad this is for performance...

The solution

What Gary did was to put SDT probes strategically on the linker, so that GDB could make use of them when examining for changes in the list of loaded shlibs. It improves performance a lot, because now GDB doesn't need to stop twice every time a shlib is loaded (it just needs to do that when stop-on-solib-events is set); it just needs to stop at the right probe, which will inform the address of the link-map entry of the first newly added library. It means GDB also won't need to walk through the list of shlibs and identify what has changed: you get that for free by examining the probe's argument.

Gary also mentions a discrepancy that happened on Solaris libc, which has also been solved by his patch.

And now, the most impressing thing: the numbers! Take a look at this table, which displays the huge improvement in the performance when using lots of shlibs (the time is in seconds):

Number of shlibs 128 256 512 1024 2048 4096
Old interface > 0 > 1 > 4 > 12 > 47 > 185
New interface > 0 > 0 > 2 > 4 > 10 > 36

Impressive, isn't it?

Conclusion

This is one the things I like most in Free Software projects: the possibility of extending and improving things by using what others did before. When I hacked GDB to implement the integration between itself and SystemTap, I had absolutely no idea that this could be used for improving the interface between the linker and the debugger (though I am almost sure that Tom was already thinking ahead!). And I can say it is a pleasure and I feel proud when I see such things happening. It just makes me feel more and more certain that Free Software is the way to go :-).


Hi, there! This is the report of the Document Freedom Day event that took place in Campinas, São Paulo state, Brazil. I will talk a little bit about how we (keep reading to know who "we" are!) organized it, and the conclusions that can be drawn to help for the next edition.

Organization

The DFD (or Document Freedom Day) 2013 in Campinas was organized by the LibrePlanet São Paulo (link in pt_BR) group. If you follow this blog, and if you speak portuguese, then you have probably read the announcement of the group that I made last year. If you haven't: LibrePlanet São Paulo is part of the LibrePlanet project (sponsored by the Free Software Foundation), and "... is a global network of free software activists and teams working together to help further the ideals of software freedom by advocating and contributing to free software.".

The DFD 2013 was an important event to us because it was the first serious event that we organized as a group. Despite some mistakes and errors, I believe we did fine and were able to learn some great lessons for the next events that we plan to do. By the way, if you want to see the official page which we used to promote the event (and organize it too), take a look here. The page is in pt_br.

Basically, we should have: (a) focused more on defining the venue as soon as possible, because that would have made it possible to (b) start sending announcements about the event earlier. We also should have contacted the Document Freedom organization and asked swags and banners earlier, because when we did it was too late for the shipment to arrive in time. And last but not least, we should really have taken pictures!! Unfortunately, I have absolutely no pictures to post here, so you will have to believe just in the words I write...

But well, nothing is perfect, and hey, the event happened!. So let's talk it :-).

The Event

DFD 2013 occurred on Wednesday, March 27th. After some discussion, we decided to schedule the event from 13h (1 p.m.) to 17h (5 p.m.), with 4 presentations of 50 minutes each, approximately. The venue chosen was CCUEC, the Center of Computing at the University of Campinas, UNICAMP. This center has some great people working on it who are involved with Free Software since the beginning of the movement, particularly Rubens Queiroz de Almeida, a very nice guy (very famous in the Brazilian Free Software scene) who helped us a lot with the organization of this event.

We understand that doing the event on a Wednesday afternoon was something that made it very hard for most people to attend, and that is probably the main reason for the low attendance: only 8 people in the audience. I have to say I was a little frustrated in the beginning, but hey, what really matters is that we spread the word about Free Software to 8 brave souls there, who will hopefully spread the word again to more people, and so on :-). So, it was time for the show to begin!

Our schedule was (presentation titles translated):

  1. "What is Free Software?", by me
  2. "Free Documents or the End of the World", by Rubens Queiroz de Almeida
  3. "HTML5: all the faces of the new standard", by Ricardo Panaggio
  4. "EPUB3: The book in the XXI century", by Raniere Silva

So my presentation was scheduled to be the first one, and I really liked it (surprise!). It was virtually the first time I gave a "philosophical" talk, and a very important one: a general presentation about Free Software, its history, the present, and a little bit of the future. In my opinion, what I liked about my talk is that I focused less on the "freedom" part, and more on the "respect" part of the philosophy. This is something I did because I wanted to use a different argument that was on my head for a long time: that the main thing behing the Free Software is the respect towards others, and only with that one can achieve freedom.

I watched Rubens too, who gave an excelent presentation about why we need free documents and standards. Rubens is very talkative and warm, which makes the audience feel relaxed. People liked his presentation a lot, from what I noticed.

Unfortunately, Ricardo Panaggio had a problem with his computer before his presentation, so we decided to switch: Raniere Silva would take his place as the third presenter, while Ricardo tried to fix the problem. I helped him with his problems, and because of this I was unable to watch Raniere's talk. In the end, we could not solve Ricardo's problem and he decided to give his presentation without any slides. In my opinion, he managed to catch everyone's attention (also because HTML5 is such a hot topic today), so I guess the missing slides were not so important after all!

At 17h o'clock, we declared DFD 2013 finished. I still had time to distribute some Free Software stickers (from FSF), and talk a little with two or three people there, who were satisfied with the presentations! It made my day, of course :-). And just because of that I now feel motivated to organized another DFD next year!

Acknowledgements

I would like to thank Rubens Queiroz for helping with the promotion, the location, and the presentation during the event. DFD 2013 would have been impossible without his help. Thanks, Rubens!

The LibrePlanet São Paulo team, specially Ricardo Panaggio, were also deeply involved with me in the organization. And I hope we manage to make a bigger event next year!

Finally, I would like to thank everyone who attended the event, even for watch only one talk. Your presence there was really, really important to all of us. See you all next year!


This will probably be one of those controversial posts, but I really cannot just be silent about a behaviour that I am constantly seeing around me.

Since my childhood, I am fascinated by the power of the words. I always liked reading a lot, and despite not knowing the grammar rules (either in pt_BR or en_US, the former being my native language, the latter being the only idiom I can consider myself fluent in), I am deeply interested in what words (and their infinite meanings) can do to us. (If you can read in portuguese, and if you also like to study or admire in this subject, I strongly recommend a romance by José Saramago called "O Homem Duplicado"). So now, what I am seeing everywhere is that people are being as careless as ever with words, their meanings, and specially their implications.

The problem I am seeing, and it is a serious problem in my opinion, is the constant use of the term "free software" when "open source" should be used. This is obviously not a recent problem, and I really cannot recall when was the first time I noticed this happening. But maybe because I am much more involved with (real) free software movements now, I have the strong impression that this "confusion" is starting to grow out of control. So here I am, trying to convince some people to be a little more coherent.

When you create a group to talk about free software, or when you join a group whose goal is to promote free software ideas, you should really do that. First of all, you should understand what free software is about. It is not about open source, for starters. It is also a political movement, not only a technical one.

I was part of a group in my former university which had "Free Software" in its name. For a long time, I believed the group really was about free software, even after receiving e-mails with heavy negative critics about my opinions when I defended something related to the free software ideology (e.g., when I suggested that we should not have a Facebook page, which had been created for the group by one of its members). Well, when I really could not hide the truth from myself anymore, I packed my things and left the group (this was actually the start of a new free software group that I founded with other friends in Brazil).

I also like a lot to go to events. And not only because of the presentations, but mostly because I really like to talk to people. Brazilians are fortunately very warm and talkative, so events here are really a fertile soil for my social skills :-). However, even when the event has "free software" in its name and description, it is very hard to find someone who really understands the philosophy behind the term. And I'm not just talking about the attendees: the event staff is also usually ignorant (and prefer to remain like this)! I feel really depressed when I start to defend the (real) free software, and people start looking at me and saying "You're radical.". It's like going in a "Debugger Conference" and feel ridicularized when you start talking about GDB! I cannot understand this...

But the worst part of all this is that newcomers are learning that "free software" is "Linux", or something which is not free software. This is definitely not a good thing, because people should be aware that the world is not just about software development: there are serious issues, including privacy and freedom menaces by Facebook/Google/Apple/etc, which we should fight against. Free software is about that as well. Awareness should be raised, actions should be taken, and people should refuse those impositions.

So, to finish what I want to say, if you do not consider yourself a free software activist, please consider becoming one. And if, after giving it a thought, you decided that you really do not want to be a free software activist, then do not use the name "free software" in your event/group/whatever, unless you really intend to talk about it and not open source.. In other words, if you don't want to help, please don't spread confusion.


Olá a todos!

Finalmente consegui um pouco de tempo na minha agenda, e resolvi escrever no blog para anunciar a criação do grupo LibrePlanet São Paulo!

O que é o LibrePlanet

O projeto LibrePlanet teve início em 2006, durante a reunião de membros da FSF (a Free Software Foundation). Ele foi criado para ajudar a organizar maneiras de levar o movimento de Software Livre ao conhecimento da população em geral.

Os grupos são organizados geograficamente, e cada um é responsável por definir metas e estratégias visando fomentar o Software Livre na região. É importante deixar claro: o objetivo é trabalhar em prol do Software Livre, e não do open source. Para saber mais a respeito da definição de Software Livre, recomendo que leia este artigo.

O surgimento do LibrePlanet São Paulo

Essa história é um pouco longa, mas vou tentar resumir :-).

Tudo começou quando eu, Ricardo Panaggio, Ivan S. Freitas e Raniere Gaia Silva começamos a trocar alguns e-mails sobre assuntos como privacidade, software livre, soluções e serviços livres, etc. Eu e o Panaggio já estávamos nos sentindo muito insatisfeitos com os rumos que um grupo local, teoricamente "pró software livre", estava tomando (como quase tudo hoje em dia, o nome "software livre" está lá simplesmente porque ninguém se tocou de que devia ser "open source" ainda...). E essa insatisfação já vinha nos fazendo querer criar um novo grupo, fiel à ideologia do Software Livre, no qual pudéssemos dar nossas opiniões sem medo de sermos esmagados por uma maioria que não se importa com "essas coisas".

Bem, começamos a conversar, e logo o Ivan e o Raniere deram sinais de que eles topariam participar do grupo, sem problemas. Portanto, o solo já estava fértil para novas idéias :-).

Um dia, eu acordei e vi na minha INBOX uma mensagem do Raniere dizendo que havia encontrado algo sobre um projeto interessante, o LibrePlanet, na Internet. Foi a faísca que faltava pra começar a movimentação! Recordei-me de que eu já havia conversado com o Matt Lee, também da FSF, sobre o LibrePlanet, e depois de uma rápida busca na wiki do projeto, vi que ainda não havia nenhum grupo brasileiro. Então, depois de alguma conversa interna, decidimos criar um grupo para o Estado de São Paulo.

Hoje, pouco mais de 2 semanas depois da criação, contamos com 10 membros cadastrados na Wiki, e aproximadamente 7 membros ativos no nosso canal de IRC. Também temos uma lista de discussão, e já estamos começando a conversar sobre possíveis projetos para 2013.

Como você pode fazer parte do grupo?

É simples! Siga os seguintes passos:

  1. Entre na nossa Wiki, e leia todas as informações presentes lá antes de qualquer coisa!
  2. Depois disso, efetue a criação de seu usuário na FSF, indo até este link de cadastro e preenchendo as informações. Repare que você não precisa tornar-se membro da FSF (os membros são pessoas que contribuem financeiramente com a Fundação), mas se você puder, iria ser bem legal :-).
  3. Ok, agora que você já possui um usuário, efetue o login na Wiki do LibrePlanet, e crie sua página pessoal lá. Para isso, vá até este link, clique no link Edit, e insira algumas informações sobre lá. Se quiser, utilize minha página pessoal como exemplo. É importante que você insira, no final de todo o conteúdo, a seguinte linha: {% raw %}{{user SP}}{% endraw %}. Ele faz com que você passe a pertencer ao grupo LibrePlanet de São Paulo.
  4. Agora, é importante que você também efetue sua inscrição na nossa lista de discussão. Vá até esta página de inscrição e preencha as informações necessárias! Também recomendamos fortemente que você envie uma mensagem de apresentação para a lista. Nada formal, só para termos uma idéia do tamanho do grupo!
  5. Ufa, último passo! Se você utiliza IRC e frequenta a rede Freenode, entre no nosso canal: #lp-br-sp! É lá que a maior parte das discussões acontece, então seria muito legal se você também pudesse participar delas!

Acho que é isso :-). Se você ainda tiver alguma dúvida sobre qualquer assunto tratado neste post (objetivos do grupo, inscrição, etc), ou se quiser fazer algum comentário, sinta-se à vontade!

Saudações livres!


Hi everybody :-).

I finally got some time to finish this series of posts, and I hope you like the overall result. For those of you who are reading this blog for the first time, you can access the first post here, and the second here.

My goal with this third post is to talk a little bit about how you can use the SDT probes with tracepoints inside GDB. Maybe this particular feature will not be so helpful to you, but I recommend reading the post either way. I will also give a brief explanation about how the SDT probes are laid out inside the binary. So, let's start!

Complementary information

In my last post, I forgot to mention that the SDT probe support present on older versions of Fedora GDB is not exactly as the way I described here. This is because Fedora GDB adopted this feature much earlier than upstream GDB itself, so while this has a great positive aspect in terms of how the distro's philosophy works (i.e., Fedora contains leading-edge features, so if you want to know how to FLOSS community will be in a few months, use it!), it also has the downside of delivering older/different versions of features in older Fedoras. But of course, this SDT feature will be fully available on Fedora 18, to be announced soon.

My suggestion is that if you use a not-so-recent Fedora (like Fedora 16, 15, etc), please upgrade it to the last version, or compile your own version of GDB yourself (it's not that hard, I will make a post about it in the next days/weeks!).

With that said, let's move on to our main topic here.

SDT Probes and Tracepoint

Before anything else, let me explain what a tracepoint is. Think of it as a breakpoint which doesn't stop the program's execution when it hits. In fact, it's a bit more than that: you can define actions associated with a tracepoint, and those actions will be performed when the tracepoint is hit. Neat, huh? :-)

There is a nice description of what a tracepoint in the GDB documentation, I recommend you give it a reading to understand the concept.

Ok, so now we have to learn how to put tracepoints in our code, and how to define actions for them. But before that, let's remember our example program:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
#include <sys/sdt.h>

int
main (int argc, char *argv[])
{
    int a = 10;

    STAP_PROBE1 (test_program, my_probe, a);

    return 0;
}

Very simple, isn't it? Ok, to the tracepoints now, my friends.

Using tracepoints inside GDB

In order to properly use tracepoints inside GDB, you will need to use gdbserver, a tiny version of GDB suitable for debugging programs remotely, over the net or serial line. In short, this is because GDB cannot put tracepoints on a program running directly under it, so we have to run it inside gdbserver and then connect GDB to it.

Running our program inside gdbserver

In our case, we will just start gdbserver in our machine, order it to listen to some high port, and connect to it through localhost, so there will be no need to have access to another computer or device.

First of all, make sure you have gdbserver installed. If you use Fedora, the package name you will have to install is gdb-gdbserver. If you have it installed, you can do:

1
2
3
$ gdbserver :3001 ./test_program
Process ./test_program created; pid = 17793
Listening on port 3001

The second argument passed to gdbserver instructs it to listen on the port 3001 of your loopback interface, a.k.a. localhost.

You will notice that gdbserver will stay there indefinitely, waiting for new connections to arrive. Don't worry, we will connect to it soon!

Connecting an instance of GDB to gdbserver

Now, go to another terminal and start GDB with our program:

1
2
3
4
5
6
7
$ gdb ./test_program
...
(gdb) target remote :3001
Remote debugging using :3001
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x0000003d60401530 in _start () from /lib64/ld-linux-x86-64.so.2

The command you have to use inside GDB is target remote. It takes as an argument the host and the port to which you want to connect. In our case, we just want it to connect to localhost, port 3001. If you saw an output like the above, great, things are working for you (don't pay attention to the messages about glibc debug information). If you didn't see it, please check to see if you're connecting to the right port, and if no other service is using it.

Ok, so now it is time to start our trace experiment!

Creating the tracepoints

Every command should be issued on GDB, not on gdbserver!

In your GDB prompt, put a tracepoint in the probe named my_probe:

1
2
(gdb) trace -probe-stap my_probe
Tracepoint 1 at 0x4005a9

As you can see, the trace command takes exactly the same arguments as the break command. Thus, you need to use the -probe-stap modified in order to instruct GDB to put the tracepoint in the probe.

And now, let's define the actions associated with this tracepoint. To do that, we use the actions command, which is an interactive command inside GDB. It takes some specific keywords, and if you want to learn more about it, please take a look at this link. For this example, we will use only the collect keyword, which tells GDB to... hm... collect something :-). In our case, it will collect the probe's first argument, or $_probe_arg0, as you may remember.

1
2
3
4
5
6
(gdb) actions 
Enter actions for tracepoint 1, one per line.
End with a line saying just "end".
>collect $_probe_arg0
>end
(gdb)

Simple as that. Finally, we have to define a breakpoint in the last instruction of our program, because it is necessary to keep it running on gdbserver in order to examine the tracepoints later. If we didn't put this breakpoint, our program would finish and gdbserver would not be able to provide information about what happened with our tracepoints. In our case, we will simply put a breakpoint on line 10, i.e., on the return 0;:

Running the trace experiment

Ok, time to run our trace experiment. First, we must issue a tstart to tell GDB to start monitoring the tracepoints. And then, we can continue our program normally.

1
2
3
4
5
6
7
8
(gdb) tstart 
(gdb) continue
Continuing.

Breakpoint 1, main (argc=1, argv=0x7fffffffde88) at /tmp/test_program.c:10
10        return 0;
(gdb) tstop
(gdb)

Remember, GDB is not going to stop your program, because tracepoints are designed to not interfere with the execution of it. Also notice that we have also stopped the trace experiment after the breakpoint hit, by using the tstop command.

Now, we will be able to examine what the tracepoint has collected. First, we will the tfind command to make sure the tracepoint has hit, and then we can inspect what we ordered it to collect:

1
2
3
4
5
(gdb) tfind start
Found trace frame 0, tracepoint 1
8         STAP_PROBE1 (test_program, my_probe, a);
(gdb) p $_probe_arg0
$1 = 10

And it works! Notice that we are printing the probe argument using the same notation as with breakpoints, even though we are not exactly executing the STAP_PROBE1 instruction. What does it mean? Well, with the tfind start command we tell GDB to actually use the trace frame collected during the program's execution, which, in this case, is the probe argument. If you know GDB, think of it as if we were using the frame command to jump back to a specific frame, where we would have access to its state.

This is a very simple example of how to use the SDT probe support in GDB with tracepoints. There is much more you can do, but I hope I could explain the basics so that you can start playing with this feature.

How the SDT probe is laid out in the binary

You might be interested in learning how the probes are created inside the binary. Other than reading the source code of /usr/include/sys/sdt.h, which is the heart of the whole feature, I also recommend this page, which explains in detail what's going on under the hood. I also recommend that you study a little about how the ELF format works, specifically about notes in the ELF file.

Conclusion

After this series of blog posts, I expect that you will now be able to use the not-so-new feature of SDT probe support on GDB. Of course, if you find some bug while using this, please feel free to report it using our bugzilla. And if you have some question, use the comment system below and I will answer ASAP :-).

See ya, and thanks for reading!


I tell you this: it is depressing when you realize that you spent more time struggling with blog engines than writing posts on your blog!

It's been a long time since I wrote the first post about this subject, and since then the patches have been accepted upstream, and GDB 7.5 now has official support for userspace SystemTap probes :-). Yay!

Well, but enough of cheap talk, let's get to the business!

Errata for my last post

Frank Ch. Eigler, one of SystemTap's maintainers, kindly mentioned something that I should say about SystemTap userspace probes.

Basically, it should be clear that SDT probes are not the only kind of userspace probing one can do with SystemTap. There is yet another kind of probe (maybe even more powerful, depending on the goals): DWARF-based function/statement probes. SystemTap supports this kind of probing mechanism for quite a while now.

It is not the goal of this post to explain it in detail, but you might want to give it a try by compiling your binary with debuginfo support (use the -g flag on GCC), and do something like:

1
2
$ stap -e 'probe process("/bin/foo").function("name") { log($$parms) }' -c /bin/foo
$ stap -e 'probe process("/bin/foo").statement("*@file.c:443") { log($$vars) }' -c /bin/foo

And that's it. You can read SystemTap's documentation, or this guide to learn how to add userspace probes.

Using GDB with SystemTap SDT Probes

Well, now let's get to the interesting part. It is time to make GDB work with the SDT probe that we have put in our example code. Let's remember it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
#include <sys/sdt.h>

int
main (int argc, char *argv[])
{
  int a = 10;

  STAP_PROBE1 (test_program, my_probe, a);

  return 0;
}

It is a very simple example, and we will have to extend it later in order to show more features. But for now, it will do.

The first thing to do is to open GDB (with SystemTap support, of course!), and check to see if it can actually see probe inserted in our example.

1
2
3
4
5
6
7
8
$ gdb ./test_program
GNU gdb (GDB) 7.5.50.20121014-cvs
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
...
(gdb) info probes
Provider     Name     Where              Semaphore Object
test_program my_probe 0x00000000004004ae           /home/sergio/work/src/git/build/gdb/test_program

Wow, it actually works! :-)

If you have seen something like the above, it means your GDB is correctly recognizing SDT probes. If you see an error, or if your GDB doesn't have the info probes command, then you'd better make sure you have a recent version of GDB otherwise you won't be able to use the SDT support.

Putting breakpoints in the code

Anyway, now it is time to start using this support. The first thing I want to show you is how to put a breakpoint in a probe.

1
2
(gdb) break -probe-stap my_probe
Breakpoint 1 at 0x4004ae

That's all! We have chosen to extend the break command in order to support the new -probe-stap parameter. If you're wondering ... why the -probe prefix?, it is because I was asked to implement a complete abstraction layer inside GDB in order to allow more types of probes to be added in the future. So, for example, if someone implements support for an hypothetical type of probe called xyz, you would have break -probe-xyz. It took me a little more time to implement this layer, but it is worth the effort.

Anyway, as you have see above, GDB recognize the probe's name and correctly put a breakpoint in it. You can also confirm that it has done the right thing by matching the address reported by info probes with the one reported by break: they should be the same.

Ok, so now, with our breakpoint in place, let's run the program and see what happens.

1
2
3
4
5
(gdb) run
Starting program: /home/sergio/work/src/git/build/gdb/test_program

Breakpoint 1, main (argc=1, argv=0x7fffffffdf68) at /tmp/example-stap.c:8
8  STAP_PROBE1 (test_program, my_probe, a);

As you can see, GDB stopped at the exact location of the probe. Therefore, you are now able to put marks (i.e., probes) in your source code which are location-independent. It means that it doesn't really matter where in the source code your probe is, and it also doesn't matter if you change the code around it, changing the line numbers, or even moving it to another file. GDB will always find your probe, and always stop at the right location. Neat!

Examining probes' arguments

But wait, there's more! Remember when I told you that you could also inspect the probe's arguments? Yes, let's do it now!

Just remember that, in SDT's parlance, the current probe's argument is a. So let's print its value.

1
2
3
4
(gdb) p $_probe_arg0
$1 = 10
(gdb) p a
$2 = 10

"Hey, captain, it seems the boat really floats!"

Check the source code above, and convince yourself that a's value is 10 :-). As you might have seen, I have used a fairly strange way of printing it. It is because the probe's arguments are available inside GDB by means of convenience variables. You can see a list of them here.

Since SDT probes can have up to 12 arguments (i.e., you can use STAP_PROBE1 ... STAP_PROBE12), we have created inside GDB 12 convenience variables, named $_probe_arg0 until $_probe_arg11. I know, it is not an easy name to remember, and even the relation between SDT naming and GDB naming is not direct (i.e., you have to subtract 1 from the SDT probe number). If you are not satisfied with this, please open a bug in our bugzilla and I promise we will discuss other options.

I would like to emphasize something here: just as you don't need debuginfo support for dealing with probes inside GDB, you also don't need debuginfo support for dealing with their arguments as well. It means that you can actually compile your code without debuginfo support, but still have access to some important variables/expressions when debugging it. Depending on how GCC optimizes your code, you may experience some difficulties with argument printing, but so far I haven't heard of anything like that.

More to come

Ok, now we have covered more things about the SDT probe support inside GDB, and I hope you understood all the concepts. It is not hard to get things going with this, specially because you don't need extra libraries to make it work.

In the next post, I intend to finish this series by explaining how to use tracepoints with SDT probes. Also, as I said in the previous post of this series, maybe I will talk a little bit about how the SDT probes are organized within the binary.

See you soon!


After a long time, here we are again :-).

With this post I will start to talk about the integration between GDB and SystemTap. This is something that Tom Tromey and I did during the last year. The patch is being reviewed as I write this post, and I expect to see it checked-in in the next few days/weeks. But let's get our hands dirty...

SystemTap Userspace Probes

You probably use (or have at least heard of) SystemTap, and maybe you think the tool is only useful for kernel inspections. If that's your case, I have a good news: you're wrong! You can actually use SystemTap to inspect userspace applications too, by using what we call SDT probes, or Static Defined Tracing probes. This is a very cheap and easy way to include probes in your application, and you can even specify arguments to those probes.

In order to use the probes (see an example below), you must include the <sys/sdt.h> header file in your source code. If you are using Fedora systems, you can obtain this header file by installing the package systemtap-sdt-devel, version equal or greater than 1.4.

Here's a simple example of an application with a one-argument probe:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
#include <sys/sdt.h>

int
main (int argc, char *argv[])
{
  int a = 10;

  STAP_PROBE1 (test_program, my_probe, a);

  return 0;
}

As you can see, this is a very simple program with one probe, which contains one argument. You can now compile the program:

1
$ gcc test_program.c -o test_program

Now you must be thinking: "Wait, wait... Didn't you just forget to link this program against some SystemTap-specific library or something?" And my answer is no. One of the spetacular things about this <sys/sdt.h> header is that it does not have any dependencies at all! As Tom said in his blog post, this is "a virtuoso display of ELF and GCC asm wizardy".

If you want to make sure your probe was inserted in the binary, you can use readelf command:

1
2
3
4
5
6
7
8
$ readelf -x .note.stapsdt ./test_program

Hex dump of section '.note.stapsdt':
  0x00000000 08000000 3a000000 03000000 73746170 ....:.......stap
  0x00000010 73647400 86044000 00000000 88054000 sdt...@.......@.
  0x00000020 00000000 00000000 00000000 74657374 ............test
  0x00000030 5f70726f 6772616d 006d795f 70726f62 _program.my_prob
  0x00000040 65002d34 402d3428 25726270 29000000 e.-4@-4(%rbp)...

(I will think about writing an explanation on how the probes are laid out on the binary, but for now you just have to care if you actually see an output from this readelf command.)

You can also use SystemTap to perform this verification:

1
2
$ stap -L 'process("./test_program").mark("*")'
process("./test_program").mark("my_probe") $arg1:long

So far, so good. If you see an output like the one above, it means your probe is correctly inserted. You could obviously use SystemTap to inspect this probe, but I won't do this right now because this is not the purpose of this post.

For now, we have learned how to:

  1. Include an SDT probe in our source code, and compile it;
  2. Verify if the probe was correctly inserted.

In the next post, I will talk about the GDB support that allows you to inspect, print arguments, and gather other information about SDT probes. I hope you like it!


This post is actually a "reply" to Gary Benson's Working on gdb post.

I have been working with GDB for quite some time now, and even though the project officially uses CVS (yes, you read it correctly, it is CVS indeed!) as its version control system, fortunately we also have a git mirror. In the end, what happens is that almost every developer uses the git mirror and just goes to CVS to commit something. But this is another discussion. Aside of this git mirror, we also have the Archer repository (which uses git by default).

My plan here is to show you how I do my daily work with GDB. The workflow is pretty simple, but maybe you will see something here that might help you.

Checking out the code

The first thing to do is to check out the code. I only have one GDB repository here, and I make branches out of it whenever I want to hack. So, to check out (or clone, in git's parlance) the code, I do (or did):

With this, we have just cloned the GDB repository, and also added another remote (i.e., repository). This is useful because we might want to hack on a branch which is on Archer, but use GDB's master branch as a base.

Create a new branch for your work

So, now it's time to create a new branch for you. Here I use one of my little "tricks" (taught to me by my friend Dodji), which is the command git-new-workdir. This is a nice command because it creates a new working directory for your project!

Maybe you're wondering why this is so cool. Well, if you ever worked with git, and more specifically, if you ever used more than one branch at a time, then maybe you will understand my excitement. In this scenario, having to constantly switch between the branches is not something rare. When you have uncommited work in your tree you can always use git stash, but that is not the ideal solution (for me). Sometimes I would forget what was on the stash, and later when I checked it, it was full of crap. Also, I like to have a separate directory for every project I am working on.

It is also important to mention that git-new-workdir is under the directory /usr/share/doc/git-VERSION/contrib/workdir/, so I created an alias that will automagically call the script for me:

So, after setting up the script, here is what I do:

Build GDB

In order to build the project, I create a build-64 directory inside my project directory (which, in the example above, is work/lazy-debuginfo-reading).

GDB fortunately supports VPATH building (i.e., build the project outside of the source tree). I strongly recommend you to use it.

As you may have noticed, I use -g3 (include debuginfo) and -O0 (do not optimize the code) in CFLAGS. Also, since some of the features I work on may affect code in other architectures, I use --enable-targets=all. It will tell configure to compile everything related to all architectures (not only x86_64, for example). At last, I specify a separate debug directory which GDB should use to search for debuginfo files.

Finalizing (for now)

After that, you will have a fresh GDB binary compiled in the build-64 directory. But that is not enough yet, since you will also want to test GDB and make sure you didn't insert a bug while hacking on it. In my next post, I will explain what is my "testflow". I hope it will be useful for someone :-).

Stay tuned!