Back in September, we had the GNU Tools Cauldron in the gorgeous city of Montréal (perhaps I should write a post specifically about it...). One of the sessions we had was the GDB BoF, where we discussed, among other things, how to improve our patch review system.

I have my own personal opinions about the current review system we use (mailing list-based, in a nutshell), and I haven't felt very confident to express it during the discussion. Anyway, the outcome was that at least 3 global maintainers have used or are currently using the Gerrit Code Review system for other projects, are happy with it, and that we should give it a try. Then, when it was time to decide who wanted to configure and set things up for the community, I volunteered. Hey, I'm already running the Buildbot master for GDB, what is the problem to manage yet another service? Oh, well.

Before we dive into the details involved in configuring and running gerrit in a machine, let me first say that I don't totally support the idea of migrating from mailing list to gerrit. I volunteered to set things up because I felt the community (or at least the its most active members) wanted to try it out. I don't necessarily agree with the choice.

Ah, and I'm writing this post mostly because I want to be able to close the 300+ tabs I had to open on my Firefox during these last weeks, when I was searching how to solve the myriad of problems I faced during the set up!

The initial plan

My very initial plan after I left the session room was to talk to the sourceware.org folks and ask them if it would be possible to host our gerrit there. Surprisingly, they already have a gerrit instance up and running. It's been set up back in 2016, it's running an old version of gerrit, and is pretty much abandoned. Actually, saying that it has been configured is an overstatement: it doesn't support authentication, user registration, barely supports projects, etc. It's basically what you get from a pristine installation of the gerrit RPM package in RHEL 6.

I won't go into details here, but after some discussion it was clear to me that the instance on sourceware would not be able to meet our needs (or at least what I had in mind for us), and that it would be really hard to bring it to the quality level I wanted. I decided to go look for other options.

The OSCI folks

Have I mentioned the OSCI project before? They are absolutely awesome. I really love working with them, because so far they've been able to meet every request I made! So, kudos to them! They're the folks that host our GDB Buildbot master. Their infrastructure is quite reliable (I never had a single problem), and Marc Dequénes (Duck) is very helpful, friendly and quick when replying to my questions :-).

So, it shouldn't come as a surprise the fact that when I decided to look for other another place to host gerrit, they were my first choice. And again, they delivered :-).

Now, it was time to start thinking about the gerrit set up.

User registration?

Over the course of these past 4 weeks, I had the opportunity to learn a bit more about how gerrit does things. One of the first things that negatively impressed me was the fact that gerrit doesn't handle user registration by itself. It is possible to have a very rudimentary user registration "system", but it relies on the site administration manually registering the users (via htpasswd) and managing everything by him/herself.

It was quite obvious to me that we would need some kind of access control (we're talking about a GNU project, with a copyright assignment requirement in place, after all), and the best way to implement it is by having registered users. And so my quest for the best user registration system began...

Gerrit supports some user authentication schemes, such as OpenID (not OpenID Connect!), OAuth2 (via plugin) and LDAP. I remembered hearing about FreeIPA a long time ago, and thought it made sense using it. Unfortunately, the project's community told me that installing FreeIPA on a Debian system is really hard, and since our VM is running Debian, it quickly became obvious that I should look somewhere else. I felt a bit sad at the beginning, because I thought FreeIPA would really be our silver bullet here, but then I noticed that it doesn't really offer a self-service user registration.

After exchanging a few emails with Marc, he told me about Keycloak. It's a full-fledged Identity Management and Access Management software, supports OAuth2, LDAP, and provides a self-service user registration system, which is exactly what we needed! However, upon reading the description of the project, I noticed that it is written in Java (JBOSS, to be more specific), and I was afraid that it was going to be very demanding on our system (after all, gerrit is also a Java program). So I decided to put it on hold and take a look at using LDAP...

Oh, man. Where do I start? Actually, I think it's enough to say that I just tried installing OpenLDAP, but gave up because it was too cumbersome to configure. Have you ever heard that LDAP is really complicated? I'm afraid this is true. I just didn't feel like wasting a lot of time trying to understand how it works, only to have to solve the "user registration" problem later (because of course, OpenLDAP is just an LDAP server).

OK, so what now? Back to Keycloak it is. I decided that instead of thinking that it was too big, I should actually install it and check it for real. Best decision, by the way!

Setting up Keycloak

It's pretty easy to set Keycloak up. The official website provides a .tar.gz file which contains the whole directory tree for the project, along with helper scripts, .jar files, configuration, etc. From there, you just need to follow the documentation, edit the configuration, and voilà.

For our specific setup I chose to use PostgreSQL instead of the built-in database. This is a bit more complicated to configure, because you need to download the JDBC driver, and install it in a strange way (at least for me, who is used to just editing a configuration file). I won't go into details on how to do this here, because it's easy to find on the internet. Bear in mind, though, that the official documentation is really incomplete when covering this topic! This is one of the guides I used, along with this other one (which covers MariaDB, but can be adapted to PostgreSQL as well).

Another interesting thing to notice is that Keycloak expects to be running on its own virtual domain, and not under a subdirectory (e.g, https://example.org instead of https://example.org/keycloak). For that reason, I chose to run our instance on another port. It is supposedly possible to configure Keycloak to run under a subdirectory, but it involves editing a lot of files, and I confess I couldn't make it fully work.

A last thing worth mentioning: the official documentation says that Keycloak needs Java 8 to run, but I've been using OpenJDK 11 without problems so far.

Setting up Gerrit

The fun begins now!

The gerrit project also offers a .war file ready to be deployed. After you download it, you can execute it and initialize a gerrit project (or application, as it's called). Gerrit will create a directory full of interesting stuff; the most important for us is the etc/ subdirectory, which contains all of the configuration files for the application.

After initializing everything, you can try starting gerrit to see if it works. This is where I had my first trouble. Gerrit also requires Java 8, but unlike Keycloak, it doesn't work out of the box with OpenJDK 11. I had to make a small but important addition in the file etc/gerrit.config:

[container]
    ...
    javaOptions = "--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED"
    ...

After that, I was able to start gerrit. And then I started trying to set it up for OAuth2 authentication using Keycloak. This took a very long time, unfortunately. I was having several problems with Gerrit, and I wasn't sure how to solve them. I tried asking for help on the official mailing list, and was able to make some progress, but in the end I figured out what was missing: I had forgotten to add the AddEncodedSlashes On in the Apache configuration file! This was causing a very strange error on Gerrit (as you can see, a java.lang.StringIndexOutOfBoundsException!), which didn't make sense. In the end, my Apache config file looks like this:

<VirtualHost *:80>
    ServerName gnutoolchain-gerrit.osci.io

    RedirectPermanent / https://gnutoolchain-gerrit.osci.io/r/
</VirtualHost>

<VirtualHost *:443>
    ServerName gnutoolchain-gerrit.osci.io

    RedirectPermanent / /r/

    SSLEngine On
    SSLCertificateFile /path/to/cert.pem
    SSLCertificateKeyFile /path/to/privkey.pem
    SSLCertificateChainFile /path/to/chain.pem

    # Good practices for SSL
    # taken from: <https://mozilla.github.io/server-side-tls/ssl-config-generator/>

    # intermediate configuration, tweak to your needs
    SSLProtocol             all -SSLv3
    SSLCipherSuite          ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
    SSLHonorCipherOrder     on
    SSLCompression          off
    SSLSessionTickets       off

    # OCSP Stapling, only in httpd 2.3.3 and later
    #SSLUseStapling          on
    #SSLStaplingResponderTimeout 5
    #SSLStaplingReturnResponderErrors off
    #SSLStaplingCache        shmcb:/var/run/ocsp(128000)

    # HSTS (mod_headers is required) (15768000 seconds = 6 months)
    Header always set Strict-Transport-Security "max-age=15768000"

    ProxyRequests Off
    ProxyVia Off
    ProxyPreserveHost On
    <Proxy *>
        Require all granted
    </Proxy>

    AllowEncodedSlashes On
        ProxyPass /r/ http://127.0.0.1:8081/ nocanon
        #ProxyPassReverse /r/ http://127.0.0.1:8081/r/
</VirtualHost>

I confess I was almost giving up Keycloak when I finally found the problem...

Anyway, after that things went more smoothly. I was finally able to make the user authentication work, then I made sure Keycloak's user registration feature also worked OK...

Ah, one interesting thing: the user logout wasn't really working as expected. The user was able to logout from gerrit, but not from Keycloak, so when the user clicked on "Sign in", Keycloak would tell gerrit that the user was already logged in, and gerrit would automatically log the user in again! I was able to solve this by redirecting the user to Keycloak's logout page, like this:

[auth]
    ...
    logoutUrl = https://keycloak-url:port/auth/realms/REALM/protocol/openid-connect/logout?redirect_uri=https://gerrit-url/
    ...

After that, it was already possible to start worrying about configure gerrit itself. I don't know if I'll write a post about that, but let me know if you want me to.

Conclusion

If you ask me if I'm totally comfortable with the way things are set up now, I can't say that I am 100%. I mean, the set up seems robust enough that it won't cause problems in the long run, but what bothers me is the fact that I'm using technologies that are alien to me. I'm used to setting up things written in Python, C, C++, with very simple yet powerful configuration mechanisms, and an easy to discover what's wrong when something bad happens.

I am reasonably satisfied with the Keycloak logs things, but Gerrit leaves a lot to be desired in that area. And both projects are written in languages/frameworks that I am absolutely not comfortable with. Like, it's really tough to debug something when you don't even know where the code is or how to modify it!

All in all, I'm happy that this whole adventure has come to an end, and now all that's left is to maintain it. I hope that the GDB community can make good use of this new service, and I hope that we can see a positive impact in the quality of the whole patch review process.

My final take is that this is all worth as long as the Free Software and the User Freedom are the ones who benefit.

P.S.: Before I forget, our gerrit instance is running at https://gnutoolchain-gerrit.osci.io.

Yes, you are reading correctly: I decided to buy a freacking Chromebook. I really needed a lightweight notebook with me for my daily hackings while waiting for my subway station, and this one seemed to be the best option available when comparing models and prices. To be fair, and before you throw me rocks, I visited the LibreBoot X60's website for some time, because I was strongly considering buying one (even considering its weight); however, they did not have it in stock, and I did not want to wait anymore, so...

Anyway, as one might expect, configuring GNU/Linux on notebooks is becoming harder as time goes by, either because the infamous Secure Boot (anti-)feature, or because they come with more and more devices that demand proprietary crap to be loaded. But fortunately, it is still possible to overcome most of those problems and still get a GNU/Linux distro running.

References

For main reference, I used the following websites:

• ACER Chromebook C720P: touchpad and touchscreen do not work, a bug related to the Fedora support for the machine.
• Fedora 20 (LXDE LiveCD) on my Acer c720p Chromebook, by Perry Myers.
• Acer C720 Chromebook, a wiki page from Arch[GNU/]Linux.
• Chromebook, a wiki page from Arch[GNU/]Linux.

I also used other references for small problems that I had during the configuration, and I will list them when needed.

Backing up ChromeOS

The first thing you will probably want to do is to make a recovery image of the ChromeOS that comes pre-installed in the machine, in case things go wrong. Unfortunately, to do that you need to have a Google account, otherwise the system will fail to record the image. So, if you want to let Google know that you bought a Chromebook, login into the system, open Chrome, and go to the special URL chrome://imageburner. You will need a 4 GiB pendrive/sdcard. It should be pretty straightforward to do the recording from there.

Screw the screw

Now comes the hard part. This notebook comes with a write-protect screw. You might be thinking: what is the purpose of this screw?

Well, the thing is: Chromebooks come with their own boot scheme, which unfortunately doesn't work to boot Linux. However, newer models also offer a “legacy boot” option (SeaBIOS), and this can boot Linux. So far, so good, but...

When you switch to SeaBIOS (details below), the system will complain that it cannot find ChromeOS, and will ask if you want to reinstall the system. This will happen every time you boot the machine, because the system is still entering the default BIOS. In order to activate SeaBIOS, you have to press CTRL-L (Control + L) every time you boot! And this is where the screw comes into play.

If you remove the write-protect screw, you will be able to make the system use SeaBIOS by default, and therefore will not need to worry about pressing CTRL-L every time. Sounds good? Maybe not so much...

The first thing to consider is that you will lose your warranty the moment you open the notebook case. As I was not very concerned about it, I decided to try to remove the screw, and guess what happened? I stripped the screw! I am still not sure why that happened, because I was using the correct screw driver for the job, but when I tried to remove the screw, it seemed like butter and started to “decompose”!

Anyway, after spending many hours trying to figure out a way to remove the screw, I gave up. My intention is to always suspend the system, so I rarely need to press CTRL-L anyway...

Well, that's all I have to say about this screwed screw. If you decide to try removing it, keep in mind that I cannot help you in any way, and that you are entirely responsible for what happens.

Now, let's install the system :-).

Enable Developer Mode

You need to enable the Developer Mode in order to be able to enable SeaBIOS. To do that, follow these steps from the Arch[GNU/]Linux wiki page.

I don't remember if this step works if you don't have activated the ChromeOS (i.e., if you don't have a Google account associated with the device). For my use, I just created a fake account to be able to proceed.

Accessing the superuser shell inside ChromeOS

Now, you will need to access the superuser (root) shell inside ChromeOS, to enable SeaBIOS. Follow the steps described in the Arch[GNU/]Linux wiki page. For this specific step, you don't need to login, which is good.

Enabling SeaBIOS

We're almost there! The last step before you boot your Fedora LiveUSB is to actually enable SeaBIOS. Just go inside your superuser shell (from the previous step) and type:

1

> crossystem dev_boot_usb=1 dev_boot_legacy=1

And that's it!

If you managed to successfuly remove the write-protect screw, you may also want to enable booting SeaBIOS by default. To do that, there is a guide, again on Arch[GNU/]Linux wiki. DO NOT DO THAT IF YOU DID NOT REMOVE THE WRITE-PROTECT SCREW!!!!

Booting Fedora

Now, we should finally be able to boot Fedora! Remember, you will have to press CTRL-L after you reboot (if you have not removed the write-protect screw), otherwise the system will just complain and not boot into SeaBIOS. So, press CTRL-L, choose the boot order (you will probably want to boot from USB first, if your Fedora is on a USB stick), choose to boot the live Fedora image, and... bum!! You will probably see a message complaining that there was not enough memory to boot (the message is “Not enough memory to load specified image”).

You can solve that by passing the mem parameter to Linux. So, when GRUB complains that it was unable to load the specified image, it will give you a command prompt (boot:), and you just need to type:

#!bash boot: linux mem=1980M

And that's it, things should work.

Installing the system

I won't guide you through the installation process; I just want to remember you that you have a 32 GiB SSD drive, so think carefully before you decide how you want to set up the partitions. What I did was to reserve 1 GB for my swap, and take all the rest to the root partition (i.e., I did not create a separate /home partition).

You will also notice that the touchpad does not work (neither does the touchscreen). So you will have to do the installation using a USB mouse for now.

Getting the touchpad to work

I strongly recommend you to read this Fedora bug, which is mostly about the touchpad/touchscreen support, but also covers other interesting topics as well.

Anyway, the bug is still being constantly updated, because the proposed patches to make the touchpad/touchscreen work were not fully integrated into Linux yet. So, depending on the version of Linux that you are running, you will probably need to run a different version of the scripts that are being kindly provided in the bug.

As of this writing, I am running Linux 3.16.2-201.fc20, and the script that does the job for me is this one. If you are like me, you will never run a script without looking at what it does, so go there and do it, I will wait :-).

OK, now that you are confident, run the script (as root, of course), and confirm that it actually installs the necessary drivers to make the devices work. In my case, I only got the touchpad working, even though the touchscreen is also covered by this script. However, since I don't want the touchscreen, I did not investigate this further.

After the installation, reboot your system and at least your touchpad should be working :-). Or kind of...

What happened to me was that I was getting strange behaviors with the touchpad. Sometimes (randomly), its sensitivity became weird, and it was very hard to move the pointer or to click on things. Fortunately, I found the solution in the same bug, in this comment by Yannick Defais. After creating this X11 configuration file, everything worked fine.

Getting suspend to work

Now comes the hard part. My next challenge was to get suspend to work, because (as I said above) I don't want to poweroff/poweron every time.

My first obvious attempt was to try to suspend using the current configuration that came with Fedora. The notebook actually suspended, but then it resumed 1 second later, and the system froze (i.e., I had to force the shutdown by holding the power button for a few seconds). Hmm, it smelled like this would take some effort, and my nose was right.

After a lot of search (and asking in the bug), I found out about a few Linux flags that I could provide in boot time. To save you time, this is what I have now in my /etc/default/grub file:

1

GRUB_CMDLINE_LINUX="tpm_tis.force=1 tpm_tis.interrupts=0 ..."

The final ... means that you should keep whatever was there before you included those parameters, of course. Also, after you edit this file, you need to regenerate the GRUB configuration file on /boot. Run the following command as root:

1

> grub2-mkconfig -o /boot/grub2/grub.cfg

Then, after I rebooted the system, I found that only adding those flags was still not enough. I saw a bunch of errors on dmesg, which showed me that there was some problem with EHCI and xHCI. After a few more research, I found the this comment on an Arch[GNU/]Linux forum. Just follow the steps there (i.e., create the necessary files, especially the /usr/lib/systemd/system-sleep/cros-sound-suspend.sh), and things should start to get better. But not yet...

Now, you will see that suspend/resume work OK, but when you suspend, the system will still resume after 1 second or so. Basically, this happens because the system is using the touchpad and the touchscreen to determine whether it should resume from suspend or not. So basically what you have to do is to disable those sources of events:

1
2

echo TPAD > /proc/acpi/wakeup
echo TSCR > /proc/acpi/wakeup

And voilà! Now everything should work as expected :-). You might want to issue those commands every time you boot the system, in order to get suspend to work every time, of course. To do that, you can create a /etc/rc.d/rc.local, which gets executed when the system starts:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

> cat /etc/rc.d/rc.local
#!/bin/bash

suspend_tricks()
{
  echo TPAD > /proc/acpi/wakeup
  echo TSCR > /proc/acpi/wakeup
}

suspend_tricks

exit 0

Don't forget to make this file executable:

1

> chmod +x /etc/rc.d/rc.local

Conclusion

Overall, I am happy with the machine. I still haven't tried installing Linux-libre on it, so I am not sure if it can work without binary blobs and proprietary craps.

I found the keyboard comfortable, and the touchpad OK. The only extra issue I had was using the Canadian/French/whatever keyboard that comes with it, because it lacks some useful keys for me, like Page Up/Down, Insert, and a few others. So far, I am working around this issue by using xbindkeys and xvkdb.

I do not recommend this machine if you are not tech-savvy enough to follow the steps listed in this post. If that is the case, then consider buying a machine that can easily run GNU/Linux, because you feel much more comfortable configuring it!

It has been a while since I dream of being able to send encrypted e-mail to everyone in my contact list. It is still a distant future, but fortunately it is getting closer with campaigns like the Reset the Net. And while I already send encrypted messages to a couple of friends, it is always good to discover (and share!) some configuration tips to make your life easy :-).

I use Gnus as my e-mail (and news!) reader for quite a while, and I can say it is a very nice piece of software (kudos to Lars and all the devs!). For those who are not aware, Gnus runs inside Emacs, which is a very nice operating system (and text editor also).

Emacs provides EasyPG for those who want to make use of cryptographic operations inside it, and Gnus also uses it to encrypt/decrypt the messages it handles. I am using it for my own messages, and it works like a charm. However, there was something that I had not had configured properly: the ability to read the encrypted messages that I was sending to my friends.

In a brief explanation, when you send an encrypted message GnuPG looks at the recipients of the message (i.e., the people that will receive it, listed in the “From:”, “Cc:” and “Bcc:” fields) and encrypts it according to each recipient's public key, which must be present in your local keyring. But when you send a message to someone, you are not (usually) present in the original recipients list, so GnuPG does not encrypt the message using your public key, and therefore you are unable to read the message later. In fact, this example can be used to illustrate how secure this system really is, when not even the sender can read his/her message again!

Anyway, this behavior was mostly unnoticed by me because I rarely look at my “Sent/” IMAP folder. Until today. And it kind of pissed me off, because I wanted to read what I wrote, damn it! So, after looking for a solution, I found a neat GnuPG setting called hidden-encrypt-to. It basically tells GnuPG to add a hidden recipient in every message it encrypts. So, all I had to do was to provide my key's ID and ask GnuPG to always encrypt the message to myself too.

You basically have to edit your $HOME/.gnupg/gpg.conf file and put this setting there:

1

hidden-encrypt-to ID

That's it. Now, whenever I send an encrypted message, GnuPG encrypts it for me as well, so I just need to go to my “Sent/” folder, and decrypt it to read.

I hope this tip helps you the same way it helped me!

After trying (and failing!) to find a guide, how-to, or anything that could help me in the migration from Jabberd2 to Prosody on my personal server, I decided to write my own version here. I hope it can help other people who want to do this somewhat painful procedure!

Struggling with Jabberd2

When I installed my personal server, I chose Jabberd2 as my Jabber server. At that time, this choice seemed the most logical to me because of a few reasons:

1. It is written in C, which is my favorite language and, therefore, would make it easier for me if I ever wanted to fix something in the software (as it really happened afterwards).
2. Looking at the page comparing different Jabber servers at Wikipedia, I found that Jabberd2 was mature enough to be used seriously.
3. I knew some people who uses it in their own servers, and they said good things about it.

So, the decision seemed pretty simple for me: Jabberd2 would be my choice! And then the problems started...

The first issue I had to solve was not Jabberd2's fault: I am using Debian Wheezy (stable) in my server, and Jabberd2 is only available for Debian Jessie (testing) or Sid (unstable). Therefore, I had to create my own version of the Jabberd2 Debian package (and all its dependencies that were not packaged) for Wheezy, which took me about 1 day. But after that, I managed to install the software in my server. Then, the configuration hell began...

Jabberd2 uses configuration files written in XML. They are well documented, with helpful comments inside. But they are confuse, as confuse as XML can be. Of course you have to take into account that it was my first time configuring a Jabber server, which added a lot to the complexity of the task. However, I feel compelled to say that the way Jabberd2 organizes its configuration files makes it a much more complex work than it should be. Nevertheless, and after lots of fails, I managed to set the server up properly. Yay!

Now, before I continue complaining, one good thing about Jabberd2: it has never crashed with me. I consider this to be something good because I am a software developer myself and I know that, despite our best efforts, bad things can happen. But Jabberd2 takes the gold medal on this one...

However... My confidence on Jabberd2's security was severily damaged when I found that the SQLite backend could not encrypt the users's passwords!!! I stumbled on this issue by myself, while naively dumping my SQLite database to check something there... You can imagine how (badly) impressed I was when I saw my password there, in plaintext. I decided to fix this issue ASAP. Hopefully next users will benefit from this fix.

After that, the bell rang in my head and I started to look for alternatives for Jabberd2. Though I still want to contribute to the project eventually (I am even working on a patch to merge all the database backends), I wanted to have a little bit more confidence in the software that I use as my Jabber server.

Meeting Prosody

Prosody came to my attention when I was setting up the server for our local Free Software group in Brazil. You can reach our wiki here (in pt_br) if you are interested. We wanted to offer a few services to our members/friends, and Jabber was obviously one of them. This happened after I discovered the bug in Jabberd2's SQLite backend, so using Jabberd2 was not a choice anymore. We had heard ejabberd, which was being used by Jabber-BR (they recently migrated to Prosody as well), but the fact that it is written in Erlang, a language that I am not familiar with, has contributed to our decision of dropping the idea. So, the only choice left was Prosody itself.

Since I am brazilian, I also feel a little bit proud of Prosody because it is writte in Lua, a programming language designed by brazilians.

We installed Prosody on our server, and it was amazingly easy to configure it! The configuration file is writte in Lua as well, which makes it a lot easier to read than XML. It is also well documented, and I felt that they were more organized too: you have small configuration files splitted by categories, instead of one big XML to edit.

The modular structure of Prosody also impressed me. You can load and unload many modules very easily, generally just by (un)commenting lines on the configuration file. Neat.

Prosody also offers a command-line program to manage the server, which is really helpful if you want to automatize some tasks and write scripts. There is a little thing that still annoys me, which is the fact that this command-line program does not have a very useful "--help" command, but I plan to propose a patch to fix that.

And at last, but definitely not least, Prosody is also very robust, and have not crashed one single time with us. It runs smoothly in the server, and although I haven't really compared the memory footprint of Jabberd2 and Prosody, I have nothing to complain about it too.

The Migration Process

Well, so after all this story, I think it is clear why I decided to migrate to Prosody. However, it was not an easy task.

Before we begin to understand the procedure needed to do the migration, I would like to say a few things. First, I would like to thank the guys at the Prosody chatroom, who were very helpful and provided several resources to make this migration possible. And I would also like to say that these instructions apply if you are running jabberd2_2.2.17-1 and prosody-0.8.2-4+deb7u2!! I have not tested with other versions of those softwares, so do it at your own risk.

The first thing you have to do is to convert Jabberd2's database to XEP-0227. This XEP is very nice: it defines a standard format to import/export user data to and from XMPP servers. Unfortunately, not every server supports this XEP, and Jabberd2 is one of those... So I started looking for ways to extract the information which was inside Jabberd2's SQLite database in a XEP-0227 compatible way. Thanks to the guys at the Prosody chatroom, I found a tool called sleekmigrate. It allowed me to generate a XEP-0227 file that could be imported into Prosody. Nice! But... I needed to extract this information from Jabberd2, and sleekmigrate could not do it. Back to the beginning...

It took me quite a while to figure out how to extract this info from Jabberd2. I was initially looking for ways (other than using sleekmigrate) that would allow me to extract this info directly from Jabberd2's SQLite database, but could not find it. Only when I read that sleekmigrate could actually work with jabberd14 data directories directly, I had the idead to find a way to convert my SQLite database into a jabberd14 data directory, and then I found this link: it teaches how to migrate from Jabberd2 to ejabberd, and has separate instructions on how to do the Jabberd2 -> Jabberd14 conversion! Sweet!

The first thing you have to do is to download the j2to1 Perl script. I had to patch the script to make it work with SQLite, and also to fix a little bug in a SQL query; you can grab my patched version here. Save the file as j2to1.pl, and run the script (don't forget to edit the source code in order to provide the database name/file):

1
2
3

$> perl j2to1.pl jabberd14-dir/
Converting user@host...
$>

This will convert the database from Jabberd2 to Jabberd14, and put the XML file of each Jabber user in the server into jabberd14-dir/host/. Now, you have a Jabberd14 version of your user data. Let's proceed with the migration.

After following the instructions on the sleekmigrate page on how to set it up, you can run it on your Jabberd14 data directory in order to finally generate a XEP-0227 XML file that will be imported into Prosody.

1

$> ./sleekmigrate.py -j /path/to/jabberd14-dir/

This should create a file called 227.xml on your current directory, which is the exported version of the Jabberd14 data directory. As a side note, it is always recommended to check those generated files in order to see if everything is OK.

Right, so now you have 227.xml, which means you can finally import it into Prosody. Fortunately, Prosody has a tool to help you with that: it is a Lua script called xep227toprosody.lua. However, if you are doing this using Debian and the same versions of the softwares that I was using, you may find it harder than it seems to run this script without errors. Here is what I had to do.

First, grab a copy of version 0.8.2 of Prosody. I had to do that because using the latest version of the script was not working. I also had to build some POSIX module of Prosody in order to make everything work. To do that, unpack the tar.gz file, go to the Prosody source code directory, and do:

1

$> apt-get build-dep prosody && ./configure --ostype=debian && make

Only after I did that I could finally run the conversion script successfully. The script is locate inside the tools/ directory. To run it:

1

$> cd tools && lua ./xep227toprosody.lua /path/to/227.xml

And yay! I finally had everything imported into Prosody!!!! Then it was just a matter of finishing the server configuration, initializing it, and everything was there: my contacts, my user, etc.

Conclusion

The migration was not very easy, especially because Jabberd2 does not support XEP-0227. I found a bug against Jabberd2 that requested this feature to be implemented, but it was not receiving any attention. Of course, if Jabberd2 implemented XEP-0227 it would make it easier for people to migrate from it, but it would also make it easier to migrate to it, so it is definitely not a bad thing to have.

Despite some difficulties, Prosody made it really easy to import my data, so kudos to it. The Prosody community is also very responsive and helpful, which made me feel very good about it. I hope I can contribute some patches to the project :-).

So, that's it. I hope this guide will be helpful to anyone who is planning to do this migration. Feel free to contact me about mistakes/comments/suggestions.

Happy migration!

Hi everybody :-).

I finally got some time to finish this series of posts, and I hope you like the overall result. For those of you who are reading this blog for the first time, you can access the first post here, and the second here.

My goal with this third post is to talk a little bit about how you can use the SDT probes with tracepoints inside GDB. Maybe this particular feature will not be so helpful to you, but I recommend reading the post either way. I will also give a brief explanation about how the SDT probes are laid out inside the binary. So, let's start!

Complementary information

In my last post, I forgot to mention that the SDT probe support present on older versions of Fedora GDB is not exactly as the way I described here. This is because Fedora GDB adopted this feature much earlier than upstream GDB itself, so while this has a great positive aspect in terms of how the distro's philosophy works (i.e., Fedora contains leading-edge features, so if you want to know how to FLOSS community will be in a few months, use it!), it also has the downside of delivering older/different versions of features in older Fedoras. But of course, this SDT feature will be fully available on Fedora 18, to be announced soon.

My suggestion is that if you use a not-so-recent Fedora (like Fedora 16, 15, etc), please upgrade it to the last version, or compile your own version of GDB yourself (it's not that hard, I will make a post about it in the next days/weeks!).

With that said, let's move on to our main topic here.

SDT Probes and Tracepoint

Before anything else, let me explain what a tracepoint is. Think of it as a breakpoint which doesn't stop the program's execution when it hits. In fact, it's a bit more than that: you can define actions associated with a tracepoint, and those actions will be performed when the tracepoint is hit. Neat, huh? :-)

There is a nice description of what a tracepoint in the GDB documentation, I recommend you give it a reading to understand the concept.

Ok, so now we have to learn how to put tracepoints in our code, and how to define actions for them. But before that, let's remember our example program:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

#include <sys/sdt.h>

int
main (int argc, char *argv[])
{
    int a = 10;

    STAP_PROBE1 (test_program, my_probe, a);

    return 0;
}

Very simple, isn't it? Ok, to the tracepoints now, my friends.

Using tracepoints inside GDB

In order to properly use tracepoints inside GDB, you will need to use gdbserver, a tiny version of GDB suitable for debugging programs remotely, over the net or serial line. In short, this is because GDB cannot put tracepoints on a program running directly under it, so we have to run it inside gdbserver and then connect GDB to it.

Running our program inside gdbserver

In our case, we will just start gdbserver in our machine, order it to listen to some high port, and connect to it through localhost, so there will be no need to have access to another computer or device.

First of all, make sure you have gdbserver installed. If you use Fedora, the package name you will have to install is gdb-gdbserver. If you have it installed, you can do:

1
2
3

$ gdbserver :3001 ./test_program
Process ./test_program created; pid = 17793
Listening on port 3001

The second argument passed to gdbserver instructs it to listen on the port 3001 of your loopback interface, a.k.a. localhost.

You will notice that gdbserver will stay there indefinitely, waiting for new connections to arrive. Don't worry, we will connect to it soon!

Connecting an instance of GDB to gdbserver

Now, go to another terminal and start GDB with our program:

1
2
3
4
5
6
7

$ gdb ./test_program
...
(gdb) target remote :3001
Remote debugging using :3001
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x0000003d60401530 in _start () from /lib64/ld-linux-x86-64.so.2

The command you have to use inside GDB is target remote. It takes as an argument the host and the port to which you want to connect. In our case, we just want it to connect to localhost, port 3001. If you saw an output like the above, great, things are working for you (don't pay attention to the messages about glibc debug information). If you didn't see it, please check to see if you're connecting to the right port, and if no other service is using it.

Ok, so now it is time to start our trace experiment!

Creating the tracepoints

Every command should be issued on GDB, not on gdbserver!

In your GDB prompt, put a tracepoint in the probe named my_probe:

1
2

(gdb) trace -probe-stap my_probe
Tracepoint 1 at 0x4005a9

As you can see, the trace command takes exactly the same arguments as the break command. Thus, you need to use the -probe-stap modified in order to instruct GDB to put the tracepoint in the probe.

And now, let's define the actions associated with this tracepoint. To do that, we use the actions command, which is an interactive command inside GDB. It takes some specific keywords, and if you want to learn more about it, please take a look at this link. For this example, we will use only the collect keyword, which tells GDB to... hm... collect something :-). In our case, it will collect the probe's first argument, or $_probe_arg0, as you may remember.

1
2
3
4
5
6

(gdb) actions 
Enter actions for tracepoint 1, one per line.
End with a line saying just "end".
>collect $_probe_arg0
>end
(gdb)

Simple as that. Finally, we have to define a breakpoint in the last instruction of our program, because it is necessary to keep it running on gdbserver in order to examine the tracepoints later. If we didn't put this breakpoint, our program would finish and gdbserver would not be able to provide information about what happened with our tracepoints. In our case, we will simply put a breakpoint on line 10, i.e., on the return 0;:

Running the trace experiment

Ok, time to run our trace experiment. First, we must issue a tstart to tell GDB to start monitoring the tracepoints. And then, we can continue our program normally.

1
2
3
4
5
6
7
8

(gdb) tstart 
(gdb) continue
Continuing.

Breakpoint 1, main (argc=1, argv=0x7fffffffde88) at /tmp/test_program.c:10
10        return 0;
(gdb) tstop
(gdb)

Remember, GDB is not going to stop your program, because tracepoints are designed to not interfere with the execution of it. Also notice that we have also stopped the trace experiment after the breakpoint hit, by using the tstop command.

Now, we will be able to examine what the tracepoint has collected. First, we will the tfind command to make sure the tracepoint has hit, and then we can inspect what we ordered it to collect:

1
2
3
4
5

(gdb) tfind start
Found trace frame 0, tracepoint 1
8         STAP_PROBE1 (test_program, my_probe, a);
(gdb) p $_probe_arg0
$1 = 10

And it works! Notice that we are printing the probe argument using the same notation as with breakpoints, even though we are not exactly executing the STAP_PROBE1 instruction. What does it mean? Well, with the tfind start command we tell GDB to actually use the trace frame collected during the program's execution, which, in this case, is the probe argument. If you know GDB, think of it as if we were using the frame command to jump back to a specific frame, where we would have access to its state.

This is a very simple example of how to use the SDT probe support in GDB with tracepoints. There is much more you can do, but I hope I could explain the basics so that you can start playing with this feature.

How the SDT probe is laid out in the binary

You might be interested in learning how the probes are created inside the binary. Other than reading the source code of /usr/include/sys/sdt.h, which is the heart of the whole feature, I also recommend this page, which explains in detail what's going on under the hood. I also recommend that you study a little about how the ELF format works, specifically about notes in the ELF file.

Conclusion

After this series of blog posts, I expect that you will now be able to use the not-so-new feature of SDT probe support on GDB. Of course, if you find some bug while using this, please feel free to report it using our bugzilla. And if you have some question, use the comment system below and I will answer ASAP :-).

See ya, and thanks for reading!

I tell you this: it is depressing when you realize that you spent more time struggling with blog engines than writing posts on your blog!

It's been a long time since I wrote the first post about this subject, and since then the patches have been accepted upstream, and GDB 7.5 now has official support for userspace SystemTap probes :-). Yay!

Well, but enough of cheap talk, let's get to the business!

Errata for my last post

Frank Ch. Eigler, one of SystemTap's maintainers, kindly mentioned something that I should say about SystemTap userspace probes.

Basically, it should be clear that SDT probes are not the only kind of userspace probing one can do with SystemTap. There is yet another kind of probe (maybe even more powerful, depending on the goals): DWARF-based function/statement probes. SystemTap supports this kind of probing mechanism for quite a while now.

It is not the goal of this post to explain it in detail, but you might want to give it a try by compiling your binary with debuginfo support (use the -g flag on GCC), and do something like:

1
2

$ stap -e 'probe process("/bin/foo").function("name") { log($$parms) }' -c /bin/foo
$ stap -e 'probe process("/bin/foo").statement("*@file.c:443") { log($$vars) }' -c /bin/foo

And that's it. You can read SystemTap's documentation, or this guide to learn how to add userspace probes.

Using GDB with SystemTap SDT Probes

Well, now let's get to the interesting part. It is time to make GDB work with the SDT probe that we have put in our example code. Let's remember it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

#include <sys/sdt.h>

int
main (int argc, char *argv[])
{
  int a = 10;

  STAP_PROBE1 (test_program, my_probe, a);

  return 0;
}

It is a very simple example, and we will have to extend it later in order to show more features. But for now, it will do.

The first thing to do is to open GDB (with SystemTap support, of course!), and check to see if it can actually see probe inserted in our example.

1
2
3
4
5
6
7
8

$ gdb ./test_program
GNU gdb (GDB) 7.5.50.20121014-cvs
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
...
(gdb) info probes
Provider     Name     Where              Semaphore Object
test_program my_probe 0x00000000004004ae           /home/sergio/work/src/git/build/gdb/test_program

Wow, it actually works! :-)

If you have seen something like the above, it means your GDB is correctly recognizing SDT probes. If you see an error, or if your GDB doesn't have the info probes command, then you'd better make sure you have a recent version of GDB otherwise you won't be able to use the SDT support.

Putting breakpoints in the code

Anyway, now it is time to start using this support. The first thing I want to show you is how to put a breakpoint in a probe.

1
2

(gdb) break -probe-stap my_probe
Breakpoint 1 at 0x4004ae

That's all! We have chosen to extend the break command in order to support the new -probe-stap parameter. If you're wondering ... why the -probe prefix?, it is because I was asked to implement a complete abstraction layer inside GDB in order to allow more types of probes to be added in the future. So, for example, if someone implements support for an hypothetical type of probe called xyz, you would have break -probe-xyz. It took me a little more time to implement this layer, but it is worth the effort.

Anyway, as you have see above, GDB recognize the probe's name and correctly put a breakpoint in it. You can also confirm that it has done the right thing by matching the address reported by info probes with the one reported by break: they should be the same.

Ok, so now, with our breakpoint in place, let's run the program and see what happens.

1
2
3
4
5

(gdb) run
Starting program: /home/sergio/work/src/git/build/gdb/test_program

Breakpoint 1, main (argc=1, argv=0x7fffffffdf68) at /tmp/example-stap.c:8
8  STAP_PROBE1 (test_program, my_probe, a);

As you can see, GDB stopped at the exact location of the probe. Therefore, you are now able to put marks (i.e., probes) in your source code which are location-independent. It means that it doesn't really matter where in the source code your probe is, and it also doesn't matter if you change the code around it, changing the line numbers, or even moving it to another file. GDB will always find your probe, and always stop at the right location. Neat!

Examining probes' arguments

But wait, there's more! Remember when I told you that you could also inspect the probe's arguments? Yes, let's do it now!

Just remember that, in SDT's parlance, the current probe's argument is a. So let's print its value.

1
2
3
4

(gdb) p $_probe_arg0
$1 = 10
(gdb) p a
$2 = 10

"Hey, captain, it seems the boat really floats!"

Check the source code above, and convince yourself that a's value is 10 :-). As you might have seen, I have used a fairly strange way of printing it. It is because the probe's arguments are available inside GDB by means of convenience variables. You can see a list of them here.

Since SDT probes can have up to 12 arguments (i.e., you can use STAP_PROBE1 ... STAP_PROBE12), we have created inside GDB 12 convenience variables, named $_probe_arg0 until $_probe_arg11. I know, it is not an easy name to remember, and even the relation between SDT naming and GDB naming is not direct (i.e., you have to subtract 1 from the SDT probe number). If you are not satisfied with this, please open a bug in our bugzilla and I promise we will discuss other options.

I would like to emphasize something here: just as you don't need debuginfo support for dealing with probes inside GDB, you also don't need debuginfo support for dealing with their arguments as well. It means that you can actually compile your code without debuginfo support, but still have access to some important variables/expressions when debugging it. Depending on how GCC optimizes your code, you may experience some difficulties with argument printing, but so far I haven't heard of anything like that.

More to come

Ok, now we have covered more things about the SDT probe support inside GDB, and I hope you understood all the concepts. It is not hard to get things going with this, specially because you don't need extra libraries to make it work.

In the next post, I intend to finish this series by explaining how to use tracepoints with SDT probes. Also, as I said in the previous post of this series, maybe I will talk a little bit about how the SDT probes are organized within the binary.

See you soon!

After a long time, here we are again :-).

With this post I will start to talk about the integration between GDB and SystemTap. This is something that Tom Tromey and I did during the last year. The patch is being reviewed as I write this post, and I expect to see it checked-in in the next few days/weeks. But let's get our hands dirty...

SystemTap Userspace Probes

You probably use (or have at least heard of) SystemTap, and maybe you think the tool is only useful for kernel inspections. If that's your case, I have a good news: you're wrong! You can actually use SystemTap to inspect userspace applications too, by using what we call SDT probes, or Static Defined Tracing probes. This is a very cheap and easy way to include probes in your application, and you can even specify arguments to those probes.

In order to use the probes (see an example below), you must include the <sys/sdt.h> header file in your source code. If you are using Fedora systems, you can obtain this header file by installing the package systemtap-sdt-devel, version equal or greater than 1.4.

Here's a simple example of an application with a one-argument probe:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

#include <sys/sdt.h>

int
main (int argc, char *argv[])
{
  int a = 10;

  STAP_PROBE1 (test_program, my_probe, a);

  return 0;
}

As you can see, this is a very simple program with one probe, which contains one argument. You can now compile the program:

1

$ gcc test_program.c -o test_program

Now you must be thinking: "Wait, wait... Didn't you just forget to link this program against some SystemTap-specific library or something?" And my answer is no. One of the spetacular things about this <sys/sdt.h> header is that it does not have any dependencies at all! As Tom said in his blog post, this is "a virtuoso display of ELF and GCC asm wizardy".

If you want to make sure your probe was inserted in the binary, you can use readelf command:

1
2
3
4
5
6
7
8

$ readelf -x .note.stapsdt ./test_program

Hex dump of section '.note.stapsdt':
  0x00000000 08000000 3a000000 03000000 73746170 ....:.......stap
  0x00000010 73647400 86044000 00000000 88054000 sdt...@.......@.
  0x00000020 00000000 00000000 00000000 74657374 ............test
  0x00000030 5f70726f 6772616d 006d795f 70726f62 _program.my_prob
  0x00000040 65002d34 402d3428 25726270 29000000 e.-4@-4(%rbp)...

(I will think about writing an explanation on how the probes are laid out on the binary, but for now you just have to care if you actually see an output from this readelf command.)

You can also use SystemTap to perform this verification:

1
2

$ stap -L 'process("./test_program").mark("*")'
process("./test_program").mark("my_probe") $arg1:long

So far, so good. If you see an output like the one above, it means your probe is correctly inserted. You could obviously use SystemTap to inspect this probe, but I won't do this right now because this is not the purpose of this post.

For now, we have learned how to:

1. Include an SDT probe in our source code, and compile it;
2. Verify if the probe was correctly inserted.

In the next post, I will talk about the GDB support that allows you to inspect, print arguments, and gather other information about SDT probes. I hope you like it!