Memory mappings, core dumps, GDB and Linux

After spending the last weeks struggling with this, I decided to write a blog post. First, what is “this” that you are talking about? The answer is: Linux kernel’s concept of memory mapping. I found it utterly confused, beyond my expectations, and so I believe that a blog post is the write way to (a) preserve and (b) share this knowledge. So, let’s do it!

First things first

First, I cannot begin this post without a few acknowledgements and “thank you’s”. The first goes to Oleg Nesterov (sorry, I could not find his website), a Linux kernel guru who really helped me a lot through the whole task. Another “thank you” goes to Jan Kratochvil, who also provided valuable feedback by commenting my GDB patch. Now, back to the point.

The task

The task was requested here: GDB needed to respect the /proc/<PID>/coredump_filter file when generating a coredump (i.e., when you use the gcore command).

Currently, GDB has his own coredump mechanism implemented which, despite its limitations and bugs, has been around for quite some time. However, and maybe you don’t know that, but the Linux kernel has its own algorithm for generating the corefile of a process. And unfortunately, GDB and Linux were not really following the same standards here…

So, in the end, the task was about synchronizing GDB and Linux. To do that, I first had to decipher the contents of the /proc/<PID>/smaps file.

The /proc/<PID>/smaps file

This special file, generated by the Linux kernel when you read it, contains detailed information about each memory mapping of a certain process. Some of the fields on this file are documented in the proc(5) manpage, but others are missing there (asking for a patch!). Here is an explanation of everything I needed:

  • The first line of each memory mapping has the following format:

    The fields here are:

    a) address is the address range, in the process’ address space, that the mapping occupies. This part was already treated by GDB, so I did not have to worry about it.

    b) perms is a set of permissions (r ead, w rite, e x ecute, s hared, p rivate [COW – copy-on-write]) applied to the memory mapping. GDB was already dealing with rwx permissions, but I needed to include the p flag as well. I also made GDB ignore the mappings that did not have the r flag active, because it does not make sense to dump something that you cannot read.

    c) offset is the offset into the applied to the file, if the mapping is file-backed (see below). GDB already handled this correctly.

    d) dev is the device (major:minor) related to the file, if there is one. GDB already handled this correctly, though I was using this field for more things (continue reading).

    e) inode is the inode on the device above. The value of zero means that no inode is associated with the memory mapping. Nothing to do here.

    f) pathname is the file associate with this mapping, if there is one. This is one of the most important fields that I had to use, and one of the most complicated to understand completely. GDB now uses this to heuristically identify whether the mapping is anonymous or not.

  • GDB is now also interested in Anonymous: and AnonHugePages: fields from the smaps file. Those fields represent the content of anonymous data on the mapping; if GDB finds that this content is greater than zero, this means that the mapping is anonymous.

  • The last, but perhaps most important field, is the VmFlags: field. It contains a series of two-letter flags that provide very useful information about the mapping. A description of the fields is: a) sh: the mapping is shared (VM_SHARED) b) dd: this mapping should not be dumped in a corefile (VM_DONTDUMP) c) ht: this is HugeTLB mapping

With that in hands, the following task was to be able to determine whether a memory mapping is anonymous or file-backed, private or shared.

Types of memory mappings

There can be four types of memory mappings:

  1. Anonymous private mapping
  2. Anonymous shared mapping
  3. File-backed private mapping
  4. File-backed shared mapping

It should be possible to uniquely identify each mapping based on the information provided by the smaps file; however, you will see that this is not always the case. Below, I will explain how to determine each of the four characteristics that define a mapping.


A mapping is anonymous if one of these conditions apply:

  1. The pathname associated with it is either /dev/zero (deleted), /SYSV%08x (deleted), or <filename> (deleted) (see below).
  2. There is content in the Anonymous: or in the AnonHugePages: fields of the mapping in the smaps file.

A special explanation is needed for the <filename> (deleted) case. It is not always guaranteed that it identifies an anonymous mapping; in fact, it is possible to have the (deleted) part for file-backed mappings as well (say, when you are running a program that uses shared libraries, and those shared libraries have been removed because of an update, for example). However, we are trying to mimic the behavior of the Linux kernel here, which checks to see if a file has no hard links associated with it (and therefore is truly deleted).

Although it may be possible for the userspace to do an extensive check (by stat ing the file, for example), the Linux kernel certainly could give more information about this.


A mapping is file-backed (i.e., not anonymous) if:

  1. The pathname associated with it contains a <filename>, without the (deleted) part.

As has been explained above, a mapping whose pathname contains the (deleted) string could still be file-backed, but we decide to consider it anonymous.

It is also worth mentioning that a mapping can be simultaneously anonymous and file-backed: this happens when the mapping contains a valid pathname (without the (deleted) part), but also contains Anonymous: or AnonHugePages: contents.


A mapping is considered to be private (i.e., not shared) if:

  1. In the absence of the VmFlags field (in the smaps file), its permission field has the flag p.
  2. If the VmFlags field is present, then the mapping is private if we do not find the sh flag there.


A mapping is shared (i.e., not private) if:

  1. In the absence of VmFlags in the smaps file, the permission field of the mapping does not have the p flag. Not having this flag actually means VM_MAYSHARE and not necessarily VM_SHARED (which is what we want), but it is the best approximation we have.
  2. If the VmFlags field is present, then the mapping is shared if we find the sh flag there.

The patch

With all that in mind, I hacked GDB to improve the coredump mechanism for GNU/Linux operating systems. The main function which decides the memory mappings that will or will not be dumped on GNU/Linux is linux_find_memory_regions_full; the Linux kernel obviously uses its own function, vma_dump_size, to do the same thing.

Linux has one advantage: it is a kernel, and therefore has much more knowledge about processes’ internals than a userspace program. For example, inside Linux it is trivial to check if a file marked as “(deleted)” in the output of the smaps file has no hard links associated with it (and therefore is not really deleted); the same operation on userspace, however, would require root access to inspect the contents of the /proc/<PID>/map_files/ directory.

The case described above, if you remember, is something that impacts the ability to tell whether a mapping is anonymous or not. I am talking to the Linux kernel guys to see if it is possible to export this information directly via the smaps file, instead of having to do the current heuristic.

While doing this work, some strange behaviors were found in the Linux kernel. Oleg is working on them, along with other Linux hackers. From our side, there is still room for improvement on this code. The first thing I can think of is to improve the heuristics for finding anonymous mappings. Another relatively easy thing to do would be to let the user specify a value for coredump_filter on the command line, without editing the /proc file. And of course, keep this code always updated with its counterpart in the Linux kernel.

Upstream discussions and commit

If you are interested, you can see the discussions that happened upstream by going to this link. This is the fourth (and final) submission of the patch; you should be able to find the other submissions in the archive.

The final commit can be found in the official repository.

Fazendo a Diferença

Deu saudade de escrever em português :-). E deu saudade, também, de fazer algum post mais “filosófico”.

Não sei dizer o porquê, mas às vezes tenho uma mania besta: gosto de ficar procurando “sarna pra me coçar”. Em outras palavras, eu fico procurando coisas que me deixam mal, mesmo sabendo que vou ficar mal depois de vê-las.

Não tenho explicação pra esse comportamento. É algo meio sabotador, meio sofredor, meio… Não sei. Às vezes, quando me vejo novamente nesse ciclo vicioso, consigo parar. No entanto, na maioria das vezes, eu entro num estado estranho: é como se eu estivesse me observando, estudando quais consequências aquele ato traz para mim. Fico me perguntando se sou a única pessoa desse mundo que faz isso…

Acho que um exemplo bom desse tipo de comportamento é o que tenho feito ultimamente. Às vezes, por algum motivo que me é estranho, leio coisas ruins escritas por pessoas extremamente insensatas. E, talvez pelo mesmo motivo misterioso, eu fico mal com o que leio, mesmo sabendo que, colocando na balança o que essas pessoas fazem e o que eu faço, a diferença é gigantesca. Então por que raios eu fico mal quando leio as besteiras que são praticamente vomitadas por essas pessoas?

Talvez algumas pessoas (eu incluso) tenham um radar pra sentimentos fortes. Por exemplo, um gesto de altruísmo é algo que consegue tocar o fundo da alma, e merece ser apreciado como um vinho raro. Mas, em contrapartida, uma expressão de raiva, desprezo ou incompreensão também capta a atenção de uma forma quase inevitável. O mistério que esse gesto, muitas vezes incoerente, esconde é algo que me deixa quase aficcionado, como se eu estivesse lendo um livro e não quisesse parar antes de chegar no final. Por que uma pessoa se coloca num papel por vezes ridículo, apenas por conta de uma opinião? Por que essa pessoa, na ânsia de criticar um comportamento, um pensamento, ou uma ideologia, muitas vezes exibe exatamente as mesmas características que repudia? O que faz um ser humano, cheio de falhas e limitações, subir num (muitas vezes falso) pedestal e esquecer que já esteve lá embaixo?

Felizmente, as questões acima, por mais intrigantes que sejam, não têm me prendido por muito tempo. Acho que, nesse processo de aprendizagem a que chamamos de “vida”, estou num ponto em que percebo claramente o caos que reina na cabeça dessas pessoas, e tento me afastar dele. Mas, mais importante que isso, acho que me dou conta de você pode escolher ser a mudança que quer ver no mundo (Gandhi), ou ficar ladrando enquanto a caravana passa… E eu definitivamente não quero perder meu tempo comparando códigos pra dizer quem é melhor.

The GNU Radical

A friend of mine, Blaise, once told me not to introduce myself as “… what you would call a radical…”. He had listened to me talking to a person who were questioning what a Free Software activist does. My friend’s rationale, to which I totally agree, is that you must let the other person decide whether she thinks you are a “radical” or not. In other words, if you say you are a “radical” from the beginning, it will probably induce the other person to a pre-judgement about you, which is not good for you and for her.

As I said, I agree with him. But I am going through a lot of situations in my life that are constantly reminding me that, maybe, I am that “radical” after all. I do not know whether this is good or bad, and I can say I have been questioning myself for a while now. This post, by the way, is going to be a lot about self-questioning.

Maybe the problem is that I am expecting too much from those that have the same beliefs that I do. Or maybe the cause is that I do not know what to expect from them in certain situations, and I am disappointed when I see that they do not follow what I think is best sometimes. On the other hand, when I look myself in the mirror, I do not know whether I am totally following what I think is best; and if I am not, then how can I even consider telling others to do that? And even if I am following my own advices, how can I be sure that they are good enough for others?

One good example of this is my opinion about FSF’s use of Twitter. The opinion is public, and has been criticized by many people already, including Free Software supporters. Shortly after I wrote the post, I mentioned it to Richard Stallman, and he told me he was not going to read it because he considered it “too emotional”. I felt deeply sad because of his reaction, especially because it came from someone who often appeals to emotions in order to teach what he has to say. But I also started questioning myself about the topic.

Is it really bad to use Twitter? This is what I ask myself sometimes. I see so many people using it, including those who defend Free Software as I do (like Matt Lee), or those who stand against privacy abuses (like Jacob Appelbaum), or who are worried about social causes, or… Yeah, you got the point. I refuse to believe that they did not think about Twitter’s issues, or about how they would be endorsing its use by using it themselves. Yet, they are there, and a lot of people is following their posts and discussing their opinions and ideas for a better world. As much as I try to understand their motivation for using Twitter (or even Facebook), I cannot convince myself that what they are doing is good for their goals. Am I being too narrow minded? Am I missing something?

Another example are my thoughts about Free Software programs that support (and sometimes even promote) unethical services. They (the thoughts) are also public. And it seems that this opinion, which is about something I called “Respectful Software”, is too strong (or “radical”?) for the majority of the developers, even considering Free Software developers. I saw very good arguments on why Free Software should support unethical services, and it is hard to disagree with them. I guess the best of those arguments is that when you support unethical services like Facebook, you are offering a Free Software option for those who want or need to use the service. In other words, you are helping them to slowly get rid of the digital handcuffs.

It seems like all those arguments (about Twitter, about implementing support for proprietary systems on Free Software, and others) are ultimately about reaching users that would otherwise remain ignorant of the Free Software philosophy. And how can someone have counter-arguments for this? It is impossible to argue that we do not need to take the Free Software message to everybody, because when someone does not use Free Software, she is doing harm to her community (thus, we want more people using Free Software, of course). When the Free Software Foundation makes use of Twitter to bring more people to the movement, and when I see that despite talking to people all around me I can hardly convince them to try GNU/Linux, who am I to criticize the FSF?

So, I have been thinking to myself whether it is time to change. What I am realizing more and more is that my fight for coherence perhaps is flawed. We are incoherent by nature. And the truth is that, no matter what we do, people change according to their own time, their own will, and their own beliefs (or to the lack of them). I remembered something that I once heard: changing is not binary, changing is a process. So, after all, maybe it is time to stop being a “GNU radical” (in the sense that I am radical even for the GNU project), and become a new type of activist.

Brasil em Conserva

As eleições brasileiras já acabaram, e talvez eu devesse me sentir mais à vontade pra falar do assunto do que realmente me sinto. Não sei, mas tenho a impressão de que, dessa vez, as coisas aconteceram de um modo um pouco diferente do que o de costume. Aliás, não acho que tenha sido “coisa de momento”, e tampouco acho que seja uma exclusividade brasileira: as pessoas estão ficando mais conservadoras, mais “endireitadas”. E eu vou tentar explicar, talvez pretensiosamente, por que eu não acho que isso seja bom.

Nunca votei em candidato algum, em nenhuma eleição até hoje. Sempre me vi descrente das propostas apresentadas, ainda mais quando percebia que aqueles que davam rostos às propostas eram basicamente os mesmos. Por isso, nas primeiras eleições em que pude “exercer a cidadania em sua plenitude” (uma mentira deslavada contada pela imprensa, que talvez mereça outro post), lá no longínquo ano de 2002, decidi por anular meus votos. Depois disso, mudei-me de cidade, e não transferi meu título de eleitor porque, no final das contas, iria acabar votando nulo novamente. No entanto, e de uma maneira aparentemente contraditória, sempre interessei-me por política.

Pode mesmo parecer contradição, mas eu nunca entendi como existiam pessoas (e são muitas!) que não queriam saber de política, e do que estava acontecendo no próprio país. Obviamente, essas mesmas pessoas em geral são as primeiras que reclamam do governo, ou que criticam um político, mesmo sem saber exatamente o porquê de fazerem isso. E quando vamos falar sobre política com elas, aquela velha máxima “Política não se discute!” vem à tona, e você de repente perde qualquer motivação para continuar conversando. Mas mesmo com toda essa minha “descrença-crente” com a política, nessas últimas eleições eu tive uma enorme vontade de votar.

Meu voto não iria para o Aécio. E no primeiro turno, provavelmente eu não votasse em nenhum dos candidatos, como sempre fiz. Mas no segundo turno, senti que eu não podia deixar de ajudar a Dilma a ser reeleita, mesmo que isso não necessariamente signifique que eu a apóie e concorde com tudo o que seu governo tem feito. Mas, usando uma outra velha máxima, “dos males, o menor”.

Considero-me uma pessoa com fortes tendências para as questões sociais. Não à toa defendo o Software Livre com empenho, porque acredito que se olharmos para todos, avançamos mais. Daí deriva minha antipatia pela maioria das causas individualistas, por acreditar que, apesar de ser totalmente plausível admitir que o ser humano é egoísta, não acho que devamos nos acomodar com essa constatação. E isso vai de encontro com o que o governo da Dilma (e o antigo governo do Lula) tem feito para o Brasil: avançar nas causas sociais. O crescimento que o país experimentou nos últimos anos foi, sim, muito perceptível para mim. E, quando tive a oportunidade de visitar o Nordeste brasileiro há alguns anos, pude ver que a situação por lá, apesar de ainda não ser a ideal, também melhorou bastante. Ou seja, o bolo finalmente está sendo dividido mais igualmente para todos, ainda que falte muito para que a divisão possa ser considerada boa.

Mas não foi só na área social que eu vi mudanças. Apesar de já estar nascido na época da inflação galopante da década de 1980, não tenho uma noção muito grande do que era viver naquela época. Eu era criança, e crianças não se preocupam com o preço das coisas. No entanto, lembro-me de que a vida, naquela época, não era fácil. O planejamento familiar era tarefa ingrata, porque como planejar se você não sabe o preço das coisas amanhã? E a desigualdade social era muito mais acentuada, porque (por exemplo) era impossível pensar em viajar de avião, mesmo para uma família de classe média (Europa, então, era outro mundo). Mas daí veio o plano Real, e as coisas melhoraram… E obviamente muito do crédito por essa melhora vai para o ex-presidente Fernando Henrique Cardoso e por sua política econômica que, pelo menos no começo do seu governo, conseguiu estabilizar as coisas de forma louvável. Mas, como pôde-se ver depois, Lula refinou a economia do antigo governo, e atrelou-a ao lado social, que por tanto tempo ficou esquecido.

Um outro argumento que ouço e vejo muito, principalmente por aqueles que são anti-petistas declarados, é o da corrupção. Confesso que não entendo o motivo desse ódio tão grande a apenas uma parcela do governo brasileiro (o PT não toma conta do Brasil, ao contrário do que muita gente insiste em dizer). Esquecem-se que denúncias de corrupção sempre existiram, em todos os nossos governos, e que é contra ela (a corrupção) que devemos lutar, ao invés de escolhermos um partido específico? Quando escolhemos um alvo, estamos, de certa forma, dando salvo-conduto para os outros que cometem o mesmo crime. Para mim, quando ouço um “argumento” desse tipo, a vontade de discutir cai exponencialmente.

Infelizmente, o argumento anti-petista passa por tantos outros absurdos (Venezuela, Cuba, bolivarianismo, comunismo), e é tão estúpido, que parece-me que a pré-condição para ser ouvinte dele é estar imerso na completa ignorância, principalmente a respeito desses termos. Se você souber o que é bolivarianismo ou comunismo, por exemplo, você já não pode ouvir o argumento, porque aí não vai acreditar nas conclusões. É algo tão impressionante e infantil que, novamente, fica difícil ter qualquer tipo de conversa com pessoas que repetem essas falácias como se fossem obviedades que estão aí, para qualquer um que queira vê-las.

Todo esse tipo de conversa, ao meu ver, leva a apenas uma conclusão: o conservadorismo está se alastrando no mundo. No Brasil, ele está tomando proporções perigosamente grandes. Às vezes penso qual seria a melhor maneira de combatê-lo: educar não me parece ser uma solução muito efetiva, ainda mais quando estamos falando de pessoas que possuem uma condição social mais favorável, e que julgam-se instruídas e informadas. Por enquanto, a solução tem sido ignorar o problema, o que também não vem surtindo efeitos práticos, haja visto a escalada dos conservadores nessas últimas eleições. Contraditoriamente, talvez a solução fosse parar de lutar e deixar a água correr. Parece-me às vezes que a humanidade precisa mesmo repetir seus erros de modo cíclico, para reaprender o motivo pelo qual eles já foram corrigidos outras vezes.

Respectful Software

To what extent should Free Software respect its users?

The question, strange as it may sound, is not only valid but also becoming more and more important these days. If you think that the four freedoms are enough to guarantee that the Free Software will respect the user, you are probably being oversimplistic. The four freedoms are essential, but they are not sufficient. You need more. I need more. And this is why I think the Free Software movement should have been called the Respectful Software movement.

I know I will probably hear that I am too radical. And I know I will hear it even from those who defend Free Software the way I do. But I need to express this feeling I have, even though I may be wrong about it.

It all began as an innocent comment. I make lots of presentations and talks about Free Software, and, knowing that the word “Free” is ambiguous in English, I started joking that Richard Stallman should have named the movement “Respectful Software”, instead of “Free Software”. If you think about it just a little, you will see that “respect” is a word that brings different interpretations to different people, just as “free” does. It is a subjective word. However, at least it does not have the problem of referring to completely unrelated things such as “price” and “freedom”. Respect is respect, and everybody knows it. What can change (and often does) is what a person considers respectful or not.

(I am obviously not considering the possible ambiguity that may exist in another language with the word “respect”.)

So, back to the software world. I want you to imagine a Free Software. For example, let’s consider one that is used to connect to so-called “social networks” like GNU Social or I do not want to use a specific example here; I am more interested in the consequences of a certain decision. Which decision? Keep reading :-).

Now, let’s imagine that this Free Software is just beginning its life, probably in some code repository under the control of its developer(s), but most likely using some proprietary service like GitHub (which is an issue by itself). And probably the developer is thinking: “Which social network should my software support first?”. This is an extremely valid and important question, but sometimes the developer comes up with an answer that may not be satisfactory to its users. This is where the “respect” comes into play.

In our case, this bad answer would be “Facebook”, “Twitter”, “Linkedin”, or any other unethical social network. However, those are exactly the easiest answers for many and many Free Software developers, either because those “vampiric” services are popular among users, or because the developer him/herself uses them!! By now, you should be able to see where I am getting at. My point, in a simple question, is: “How far should we, Free Software developers, allow users to go and harm themselves and the community?”. Yes, this is not just a matter of self-inflicted restrictions, as when the user chooses to use a non-free software to edit a text file, for example. It is, in most cases, a matter of harming the community too. (I have written a post related to this issue a while ago, called “Privacy as a Collective Good <post/2014-05-15-privacy-collective-good>”.)

It should be easy to see that it does not matter if I am using Facebook through my shiny Free Software application on my computer or cellphone. What really matters is that, when doing so, you are basically supporting the use of those unethical social networks, to the point that perhaps some of your friends are also using them because of you. What does it matter if they are using Free Software to access them or not? Is the benefit offered by the Free Software big enough to eliminate (or even soften) the problems that exist when the user uses an unethical service like Linkedin?

I wonder, though, what is the limit that we should obey. Where should we draw the line and say “I will not pass beyond this point”? Should we just “abandon” the users of those unethical services and social networks, while we lock ourselves in our not-very-safe world? After all, we need to communicate with them in order to bring them to our cause, but it is hard doing so without getting our hands dirty. But that is a discussion to another post, I believe.

Meanwhile, I could give plenty of examples of existing Free Softwares that are doing a disservice to the community by allowing (and even promoting) unethical services or solutions for their users. They are disrespecting their users, sometimes exploiting the fact that many users are not fully aware of privacy issues that come as a “gift” when you use those services, without spending any kind of effort to teach the users. However, I do not want this post to become a flamewar, so I will not mention any software explicitly. I think it should be quite easy for the reader to find examples out there.

Perhaps this post does not have a conclusion. I myself have not made my mind completely about the subject, though I am obviously leaning towards what most people would call the “radical” solution. But it is definitely not an easy topic to discuss, or to argument about. Nonetheless, we are closing our eyes to it, and we should not do so. The future of Free Software depends also on what kinds of services we promote, and what kinds of services we actually warn the users against. This is my definition of respect, and this is why I think we should develop Free and Respectful Software.