While debugging an application crash issue, I worked out the following steps that allow gdb to step through source code of system libraries. The steps are:
- Run gdb to find out the name of the system libraries whose source code we want to step through.
- Install the corresponding debug package of the system library.
- Download the source package of the system library, and tell gdb the path to find the source code.
- (Optional) Store the source path setup commands in a gdb command script file, and reuse the commands in the future.
The following two examples shows how the steps are used to access source code of standarc C and C++ libraries.
Example 1 - libc6
$ gdb program -c core ... Program terminated with signal 11, Segmentation fault. 0xb7347876 in ?? () from /lib/i386-linux-gnu/libc.so.6 (gdb) bt #0 0xb7347876 in ?? () from /lib/i386-linux-gnu/libc.so.6 #1 0x00008090 in ?? ()
So gdb provides useful information that segmentation fault happened in libc at address 0xb7347876. But I would rather read source code than memory address. After some research I found that I could install debug package for libc:
$ sudo apt-get install libc6-dbg
With the libc debug package installed, gdb now knows the filename and line numbers of the memory address that caused segmentation fault:
$ gdb program -c core ... Program terminated with signal 11, Segmentation fault. #0 free_check (mem=0x40, caller=0x40) at hooks.c:246 246 hooks.c: No such file or directory.
Next step is to help gdb access the source code. The source code of libc6 can be downloaded by apt-get source libc6 command:
$ mkdir /var/tmp/source/libc6/ $ cd /var/tmp/source/libc6/ $ apt-get source libc6 $ find . -name hooks.c ./eglibc-2.15/malloc/hooks.c
So we need to tell gdb the path to hooks.c is /var/tmp/source/libc6/eglibc-2.15/malloc/. This is done with the dir command. After setting the source directory, gdb is able to list the source code:
$ gdb program -c core
...
Program terminated with signal 11, Segmentation fault.
#0 free_check (mem=0x40, caller=0x40) at hooks.c:246
246 hooks.c: No such file or directory.
(gdb) dir /var/tmp/sources/libc6/eglibc-2.15/malloc/
Source directories searched: /var/tmp/sources/libc6/eglibc-2.15/malloc:$cdir:$cwd
(gdb) list
241 mchunkptr p;
242
243 if(!mem) return;
244 (void)mutex_lock(&main_arena.mutex);
245 p = mem2chunk_check(mem, NULL);
246 if(!p) {
247 (void)mutex_unlock(&main_arena.mutex);
248
249 malloc_printerr(check_action, "free(): invalid pointer", mem);
250 return;
To spare the effort of entering the source directory in the future, I place the command in the text file gdb.setup, and ask gdb to execute it before starting the interactive debugging with the -x option. Notice that gdb executes the command file after it complains about not knowing where the source file is, so it might look like the command didn't work. But gdb is able to list the source file content OK:
$ echo dir /var/tmp/sources/libc6/eglibc-2.15/malloc/ > gdb.setup
$ gdb program -c core -x gdb.setup
...
Program terminated with signal 11, Segmentation fault.
#0 free_check (mem=0x40, caller=0x40) at hooks.c:246
246 hooks.c: No such file or directory.
(gdb) list
241 mchunkptr p;
242
243 if(!mem) return;
244 (void)mutex_lock(&main_arena.mutex);
245 p = mem2chunk_check(mem, NULL);
246 if(!p) {
247 (void)mutex_unlock(&main_arena.mutex);
248
249 malloc_printerr(check_action, "free(): invalid pointer", mem);
250 return;
Example 2 - libstdc++
$ gdb program -c core ... (gdb) frame 3 #3 0xb752a51f in operator delete(void*) () from /usr/lib/i386-linux-gnu/libstdc++.so.6
Install the libstdc++ debug package. First we find out what libstdc++ package is installed:
$ dpkg -l | grep libstdc++ ii libstdc++6 4.6.3-1ubuntu5 GNU Standard C++ Library v3 ii libstdc++6-4.6-dev 4.6.3-1ubuntu5 GNU Standard C++ Library v3 (development files)
So we are looking for source code for version 4.6.3-ubuntu5. Next we check what is available:
$ apt-cache search libstdc++|grep 4.6 libstdc++6-4.6-dbg - GNU Standard C++ Library v3 (debugging files) libstdc++6-4.6-dev - GNU Standard C++ Library v3 (development files) libstdc++6-4.6-doc - GNU Standard C++ Library v3 (documentation files) lib64stdc++6-4.6-dbg - GNU Standard C++ Library v3 (debugging files) libhfstdc++6-4.6-dbg-armel-cross - GNU Standard C++ Library v3 (debugging files) libsfstdc++6-4.6-dbg-armhf-cross - GNU Standard C++ Library v3 (debugging files) libstdc++6-4.6-dbg-armel-cross - GNU Standard C++ Library v3 (debugging files) libstdc++6-4.6-dbg-armhf-cross - GNU Standard C++ Library v3 (debugging files) libstdc++6-4.6-dev-armel-cross - GNU Standard C++ Library v3 (development files) libstdc++6-4.6-dev-armhf-cross - GNU Standard C++ Library v3 (development files) libstdc++6-4.6-pic - GNU Standard C++ Library v3 (shared library subset kit) libstdc++6-4.6-pic-armel-cross - GNU Standard C++ Library v3 (shared library subset kit) libstdc++6-4.6-pic-armhf-cross - GNU Standard C++ Library v3 (shared library subset kit)
By matching the names of the installed libstdc++ development package and the debug package, we know the package we need is libstdc++6-4.6-dbg. So we install this package:
$ sudo apt-get install libstdc++6-4.6-dbg
For sanity check, verify the package version is indeed 4.6.3-1ubuntu5:
$ dpkg -l|grep libstdc++6-4.6-dbg ii libstdc++6-4.6-dbg 4.6.3-1ubuntu5 GNU Standard C++ Library v3 (debugging files)
Installing the libstdc++ debug package provides gdb the filename and line number information:
$ gdb program -c core ... (gdb) frame 4 #4 0xb751199b in deallocate (__p=0xb1f05488 "\303\001", this=) at /build/buildd/gcc-4.6-4.6.3/build/i686-linux-gnu/libstdc++-v3/include/ext/new_allocator.h:98 98 /build/buildd/gcc-4.6-4.6.3/build/i686-linux-gnu/libstdc++-v3/include/ext/new_allocator.h: No such file or directory.
Next we fetch the source package of libstdc+.
$ mkdir /var/tmp/sources/libstdc+/ $ cd /var/tmp/sources/libstdc++/ $ apt-get source libstdc++6 $ find . -name new_allocator.h ./gcc-4.6-4.6.3/gcc-4.6-4.6.3/gcc-4.6.3/libstdc++-v3/include/ext/new_allocator.h
Since the libstdc++ debug package records the location in the source tree of the source file, such as /build/buildd/gcc-4.6-4.6.3/build/i686-linux-gnu/libstdc++-v3/include/ext/new_allocator.h, we could use gdb's substitute-path variable to specify the source tree location:
$ gdb program -c core ... (gdb) set substitute-path /build/buildd/gcc-4.6-4.6.3/build/i686-linux-gnu/libstdc++-v3/ /var/tmp/sources/libstdc++/gcc-4.6-4.6.3/gcc-4.6-4.6.3/gcc-4.6.3/libstdc++-v3/ (gdb) frame 4 #4 0xb751199b in deallocate (__p=0xb1f05488 "\303\001", this=) at /build/buildd/gcc-4.6-4.6.3/build/i686-linux-gnu/libstdc++-v3/include/ext/new_allocator.h:98 98 { ::operator delete(__p); } (gdb) list 93 } 94 95 // __p is not permitted to be a null pointer. 96 void 97 deallocate(pointer __p, size_type) 98 { ::operator delete(__p); } 99 100 size_type 101 max_size() const throw() 102 { return size_t(-1) / sizeof(_Tp); }
