Sunday, November 30, 2014

Make system library source code available to gdb on Ubuntu

While debugging an application crash issue, I worked out the following steps that allow gdb to step through source code of system libraries. The steps are:
  1. Run gdb to find out the name of the system libraries whose source code we want to step through.
  2. Install the corresponding debug package of the system library.
  3. Download the source package of the system library, and tell gdb the path to find the source code.
  4. (Optional) Store the source path setup commands in a gdb command script file, and reuse the commands in the future.
The Ubuntu package management tool tool apt-get is used in these steps. A apt-get tutorial can be found here.

The following two examples shows how the steps are used to access source code of standarc C and C++ libraries.

Example 1 - libc6

$ gdb program -c core
...
Program terminated with signal 11, Segmentation fault.
 0xb7347876 in ?? () from /lib/i386-linux-gnu/libc.so.6
(gdb) bt
#0  0xb7347876 in ?? () from /lib/i386-linux-gnu/libc.so.6
#1  0x00008090 in ?? ()
So gdb provides useful information that segmentation fault happened in libc at address 0xb7347876. But I would rather read source code than memory address. After some research I found that I could install debug package for libc:
$ sudo apt-get install libc6-dbg
With the libc debug package installed, gdb now knows the filename and line numbers of the memory address that caused segmentation fault:
$ gdb program -c core
...
Program terminated with signal 11, Segmentation fault.
#0  free_check (mem=0x40, caller=0x40) at hooks.c:246
246 hooks.c: No such file or directory.
Next step is to help gdb access the source code. The source code of libc6 can be downloaded by apt-get source libc6 command:
$ mkdir /var/tmp/source/libc6/
$ cd /var/tmp/source/libc6/
$ apt-get source libc6
$ find . -name hooks.c
./eglibc-2.15/malloc/hooks.c
So we need to tell gdb the path to hooks.c is /var/tmp/source/libc6/eglibc-2.15/malloc/. This is done with the dir command. After setting the source directory, gdb is able to list the source code:
$ gdb program -c core
...
Program terminated with signal 11, Segmentation fault.
#0  free_check (mem=0x40, caller=0x40) at hooks.c:246
246 hooks.c: No such file or directory.
(gdb) dir /var/tmp/sources/libc6/eglibc-2.15/malloc/
Source directories searched: /var/tmp/sources/libc6/eglibc-2.15/malloc:$cdir:$cwd
(gdb) list
241   mchunkptr p;
242 
243   if(!mem) return;
244   (void)mutex_lock(&main_arena.mutex);
245   p = mem2chunk_check(mem, NULL);
246   if(!p) {
247     (void)mutex_unlock(&main_arena.mutex);
248 
249     malloc_printerr(check_action, "free(): invalid pointer", mem);
250     return;
To spare the effort of entering the source directory in the future, I place the command in the text file gdb.setup, and ask gdb to execute it before starting the interactive debugging with the -x option. Notice that gdb executes the command file after it complains about not knowing where the source file is, so it might look like the command didn't work. But gdb is able to list the source file content OK:
$ echo dir /var/tmp/sources/libc6/eglibc-2.15/malloc/ > gdb.setup
$ gdb program -c core -x gdb.setup
...
Program terminated with signal 11, Segmentation fault.
#0  free_check (mem=0x40, caller=0x40) at hooks.c:246
246 hooks.c: No such file or directory.
(gdb) list
241   mchunkptr p;
242 
243   if(!mem) return;
244   (void)mutex_lock(&main_arena.mutex);
245   p = mem2chunk_check(mem, NULL);
246   if(!p) {
247     (void)mutex_unlock(&main_arena.mutex);
248 
249     malloc_printerr(check_action, "free(): invalid pointer", mem);
250     return;

Example 2 - libstdc++

$ gdb program -c core
...
(gdb) frame 3
#3  0xb752a51f in operator delete(void*) () from /usr/lib/i386-linux-gnu/libstdc++.so.6
Install the libstdc++ debug package. First we find out what libstdc++ package is installed:
$ dpkg -l | grep libstdc++
ii  libstdc++6                             4.6.3-1ubuntu5                          GNU Standard C++ Library v3
ii  libstdc++6-4.6-dev                     4.6.3-1ubuntu5                          GNU Standard C++ Library v3 (development files)
So we are looking for source code for version 4.6.3-ubuntu5. Next we check what is available:
$ apt-cache search libstdc++|grep 4.6
libstdc++6-4.6-dbg - GNU Standard C++ Library v3 (debugging files)
libstdc++6-4.6-dev - GNU Standard C++ Library v3 (development files)
libstdc++6-4.6-doc - GNU Standard C++ Library v3 (documentation files)
lib64stdc++6-4.6-dbg - GNU Standard C++ Library v3 (debugging files)
libhfstdc++6-4.6-dbg-armel-cross - GNU Standard C++ Library v3 (debugging files)
libsfstdc++6-4.6-dbg-armhf-cross - GNU Standard C++ Library v3 (debugging files)
libstdc++6-4.6-dbg-armel-cross - GNU Standard C++ Library v3 (debugging files)
libstdc++6-4.6-dbg-armhf-cross - GNU Standard C++ Library v3 (debugging files)
libstdc++6-4.6-dev-armel-cross - GNU Standard C++ Library v3 (development files)
libstdc++6-4.6-dev-armhf-cross - GNU Standard C++ Library v3 (development files)
libstdc++6-4.6-pic - GNU Standard C++ Library v3 (shared library subset kit)
libstdc++6-4.6-pic-armel-cross - GNU Standard C++ Library v3 (shared library subset kit)
libstdc++6-4.6-pic-armhf-cross - GNU Standard C++ Library v3 (shared library subset kit)
By matching the names of the installed libstdc++ development package and the debug package, we know the package we need is libstdc++6-4.6-dbg. So we install this package:
$ sudo apt-get install libstdc++6-4.6-dbg
For sanity check, verify the package version is indeed 4.6.3-1ubuntu5:
$ dpkg -l|grep libstdc++6-4.6-dbg
ii  libstdc++6-4.6-dbg                     4.6.3-1ubuntu5                          GNU Standard C++ Library v3 (debugging files)
Installing the libstdc++ debug package provides gdb the filename and line number information:
$ gdb program -c core
...
(gdb) frame 4
#4  0xb751199b in deallocate (__p=0xb1f05488 "\303\001", this=) at /build/buildd/gcc-4.6-4.6.3/build/i686-linux-gnu/libstdc++-v3/include/ext/new_allocator.h:98
98 /build/buildd/gcc-4.6-4.6.3/build/i686-linux-gnu/libstdc++-v3/include/ext/new_allocator.h: No such file or directory.
Next we fetch the source package of libstdc+.
$ mkdir /var/tmp/sources/libstdc+/
$ cd /var/tmp/sources/libstdc++/
$ apt-get source libstdc++6
$ find . -name new_allocator.h
./gcc-4.6-4.6.3/gcc-4.6-4.6.3/gcc-4.6.3/libstdc++-v3/include/ext/new_allocator.h
Since the libstdc++ debug package records the location in the source tree of the source file, such as /build/buildd/gcc-4.6-4.6.3/build/i686-linux-gnu/libstdc++-v3/include/ext/new_allocator.h, we could use gdb's substitute-path variable to specify the source tree location:
$ gdb program -c core
...
(gdb) set substitute-path /build/buildd/gcc-4.6-4.6.3/build/i686-linux-gnu/libstdc++-v3/ /var/tmp/sources/libstdc++/gcc-4.6-4.6.3/gcc-4.6-4.6.3/gcc-4.6.3/libstdc++-v3/
(gdb) frame 4
#4  0xb751199b in deallocate (__p=0xb1f05488 "\303\001", this=) at /build/buildd/gcc-4.6-4.6.3/build/i686-linux-gnu/libstdc++-v3/include/ext/new_allocator.h:98
98       { ::operator delete(__p); }
(gdb) list
93       }
94 
95       // __p is not permitted to be a null pointer.
96       void
97       deallocate(pointer __p, size_type)
98       { ::operator delete(__p); }
99 
100       size_type
101       max_size() const throw() 
102       { return size_t(-1) / sizeof(_Tp); }