gcore produces core file without backtrace capability

Bug #680588 reported by tom anderson
34
This bug affects 3 people
Affects Status Importance Assigned to Milestone
gdb (Ubuntu)
Fix Released
Low
Kees Cook
Lucid
Won't Fix
Low
Kees Cook
Maverick
Fix Released
Low
Kees Cook
Natty
Fix Released
Low
Kees Cook

Bug Description

Binary package hint: gdb

I am unable to get a meaningful backtrace after loading a core file. I do the following
1. compile singlethread2.c
2. ./singlethread2
3. gdb singlethread2
4. attach <pid>
5. gcore
6. gdb singlethread2 <core>
details below:
========================= compile singlethread2.c
g++ -Wall -g -D REEENTRANT -o singlethread2.g singlethread2.c
========================= singlethread2.c
sh-3.00$ cat singlethread2.c
#include <stdio.h>
#include <stdlib.h>
int x = 0;
int main(void) {
   while(1) {
      x++;
      printf("X is now %d.\n", x);
   }
   return(1);
}
========================= attaching to single thread processes
0 %> gdb singlethread2.g
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...
(gdb) attach 11575
Attaching to program: /import/singlethread2.g, process 11575
Reading symbols from /usr/lib/libstdc++.so.6...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x00007fc8881e1fc0 in write () from /lib/libc.so.6
(gdb) bt
#0 0x00007fc8881e1fc0 in write () from /lib/libc.so.6
#1 0x00007fc88817d87a in _IO_file_write () from /lib/libc.so.6
#2 0x00007fc88817e759 in _IO_do_write () from /lib/libc.so.6
#3 0x00007fc88817d4e9 in _IO_file_xsputn () from /lib/libc.so.6
#4 0x00007fc88814ffb3 in vfprintf () from /lib/libc.so.6
#5 0x00007fc888159eca in printf () from /lib/libc.so.6
#6 0x0000000000400614 in main () at singlethread2.c:7
(gdb) gcore core.singlethead2.g.11575
Saved corefile core.singlethead2.g.11575
========================= loading core file
0 %> gdb singlethread2.g core.singlethead2.g.11575
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...
Core was generated by `/import/core_files/singlethread2.g'.
[New process 11575]
#0 0x00007fc8881e1fc0 in ?? ()
(gdb) bt
#0 0x00007fc8881e1fc0 in ?? ()
#1 0x00007fc88817d87a in ?? ()
#2 0x0000000000000000 in ?? ()
(gdb)

I have tried this on fedora 14 and it works properly. Why doesnt this work on ubunut? I tried on ubuntu 8.1, 9.04 and 10.10. It doesn't work on any of those ubuntu desktop releases.
Here is the output from fedora 14
[root@fedora14 ~]# gdb singlethread2.g core.10244
GNU gdb (GDB) Fedora (7.2-23.fc14)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /root/singlethread2.g...done.
[New Thread 10244]
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/25/7d0d6467b9f4374470e0384a3e602be99b058e
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Core was generated by `./singlethread2.g'.
#0 0x005b1416 in __kernel_vsyscall ()
Missing separate debuginfos, use: debuginfo-install glibc-2.12.90-19.i686
(gdb) info share
From To Syms Read Shared Object Library
0x00e5bbf0 0x00f7e1d4 Yes (*) /lib/libc.so.6
0x00a18850 0x00a30e2f Yes (*) /lib/ld-linux.so.2
(*): Shared library is missing debugging information.
(gdb) bt
#0 0x005b1416 in __kernel_vsyscall ()
#1 0x00f111e3 in __write_nocancel () from /lib/libc.so.6
#2 0x00eb03ac in _IO_new_file_write () from /lib/libc.so.6
#3 0x00eaffff in new_do_write () from /lib/libc.so.6
#4 0x00eb193e in _IO_new_do_write () from /lib/libc.so.6
#5 0x00eb1c7a in _IO_new_file_overflow () from /lib/libc.so.6
#6 0x00eb0145 in _IO_new_file_xsputn () from /lib/libc.so.6
#7 0x00e85d7a in vfprintf () from /lib/libc.so.6
#8 0x00e900a0 in printf () from /lib/libc.so.6
#9 0x080483f1 in main () at singlethread2.c:7
(gdb)

I have also tried this on the latest debian and it works. Here is the output from debian
debian:/home/install# gdb ./singlethread2.g core.3008
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu"...
warning: Can't read pathname for load map: Input/output error.
Reading symbols from /usr/lib/libstdc++.so.6...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /lib/i686/cmov/libm.so.6...done.
Loaded symbols for /lib/i686/cmov/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/i686/cmov/libc.so.6...done.
Loaded symbols for /lib/i686/cmov/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 Core was generated by `/home/install/singlethread2.g'.
[New process 3008]
#0 0xb7710416 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7710416 in __kernel_vsyscall ()
#1 0xb75553a3 in write () from /lib/i686/cmov/libc.so.6
#2 0xb74eeeac in _IO_file_write () from /lib/i686/cmov/libc.so.6
#3 0xb74f0027 in _IO_do_write () from /lib/i686/cmov/libc.so.6
#4 0xb74efa00 in _IO_file_overflow () from /lib/i686/cmov/libc.so.6
#5 0xb74eeb15 in _IO_file_xsputn () from /lib/i686/cmov/libc.so.6
#6 0xb74c4c97 in vfprintf () from /lib/i686/cmov/libc.so.6
#7 0xb74ce4e0 in printf () from /lib/i686/cmov/libc.so.6
#8 0x080484b7 in main () at singlethread2.c:8
(gdb)
install@debian:~$ cat /proc/version
Linux version 2.6.26-2-686 (Debian 2.6.26-25lenny1) (<email address hidden>) (gcc version 4.1.3 20080704 (prerelease) (Debian 4.1.2-25)) #1 SMP Thu Sep 16 19:35:51 UTC 2010
install@debian:~$ cat /etc/issue
Debian GNU/Linux 5.0 \n \l
install@debian:~$

Revision history for this message
Kees Cook (kees) wrote : Re: [Bug 680588] [NEW] gcore produces core file without backtrace capability

On Tue, Nov 23, 2010 at 06:30:23PM -0000, tom anderson wrote:
> 0 %> gdb singlethread2.g
> GNU gdb 6.8-debian

Fails (Ubuntu gdb 6.8)

> [root@fedora14 ~]# gdb singlethread2.g core.10244
> GNU gdb (GDB) Fedora (7.2-23.fc14)

Works (Fedora gdb 7.2)

> debian:/home/install# gdb ./singlethread2.g core.3008
> GNU gdb 6.8-debian

Works (Debian gdb 6.8)

Have you tried Debian unstable's 7.2 gdb yet? The Ubuntu and Debian versions are
almost identical there. Debian and Ubuntu's 6.8 gdbs have some large differences.
Fedora and Debian's 7.2 gdb are pretty different as well, since Fedora carries a
large patchset that they haven't pushed upstream yet. I would assume that whatever
was working in a relatively stock gdb 6.8 regressed in 7.2, but Fedora may have
patched it to work again.

I would start by looking through the Fedora patches for gdb 7.2 and try applying them
until the fix is found.

-Kees

--
Kees Cook
Ubuntu Security Team

Revision history for this message
tom anderson (xentoma) wrote :

I download the source for debian and ubuntu 6.8. They were not that different. And I copied ubuntu source to debian and compiled on debian and gdb gcore produced a core that generated a good backtrace. I did the same copying debian source to ubuntu and compiling and it did not produce a workable core.

I noticed that if on ubuntu I compile with --static I can get a backtrace

gcc -Wall -lo singelthread2e.g singlethread.c --static

-rwxr-x-r-x 1 root root 8331 2010-11-23 01:49 singlethread.g
-rwxr-x-r-x 1 root root 6119312010-11-30 22:10 singlethread2e.g

root@install:~/ubuntu_kernel# gdb singlethread2e.g core.2e
GNU gdb (GDB) 7.2-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /root/ubuntu_kernel/singlethread2e.g...(no debugging symbols found)...done.
[New Thread 15412]
Core was generated by `/root/ubuntu_kernel/singlethread2e.g'.
Program terminated with signal 3, Quit.
#0 0xb7fff416 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7fff416 in __kernel_vsyscall ()
#1 0x080530d2 in __write_nocancel ()
#2 0x08069141 in _IO_new_file_write ()
#3 0x08068e2c in new_do_write ()
#4 0x080690ed in _IO_new_do_write ()
#5 0x08069ad5 in _IO_new_file_overflow ()
#6 0x08068f67 in _IO_new_file_xsputn ()
#7 0x0805bfe5 in vfprintf ()
#8 0x08048e01 in printf ()
#9 0x080482ed in main ()
(gdb)

Any ideas of why dynamic linking is not producing good backtraces on ubuntu but does on debian?

Revision history for this message
tom anderson (xentoma) wrote :

I built gcc (4.4.4) from the original source without applying the patches and was able to compile my test program, generate a core and then reload the core in gdb and get a backtrace. I then had two different versions of my compiled program. I did "objdump -s" on each and noticed this difference:

The version which doesn't produce a backtrace has two extra sections ".gnu.hash" and ".note.gnu.build-id".

If I do "objcopy -R .gnu.hash singlethread singlethread.nognuhash" to remove the .gnu.hash section from the version which "didn't work" I am able to produce a core file where I can get a backtrace.

gdb ./singlethread.nognuhash
attach 2187
gcore core.nognuhash

gdb ./singlethread.nognuhash core.nognuhash
...
#0 0x00007fbf157c6eaf in _IO_new_file_xsputn (f=0x7fbf15ad1780, data=0x400655, n=9) at fileops.c:1383
1383 fileops.c: No such file or directory.
        in fileops.c
(gdb) bt
#0 0x00007fbf157c6eaf in _IO_new_file_xsputn (f=0x7fbf15ad1780, data=0x400655, n=9) at fileops.c:1383
#1 0x00007fbf157994ec in _IO_vfprintf_internal (s=0x7fbf15ad1780, format=0x40064c "X is now %d.\n", ap=0x7fffcfbffa40) at vfprintf.c:1333
#2 0x00007fbf157a453a in __printf (format=0x400655 "%d.\n") at printf.c:35
#3 0x0000000000400551 in main () at singlethread2.c:7

I notice there are two patches and am wondering if one of these broke the gdb core functionality
   gcc-hash-style-both.diff
   gcc-hash-style-gnu.diff

I will remove those from the series and rebuild the ubuntu patched gcc package.

Revision history for this message
tom anderson (xentoma) wrote :

After running readelf -a on various version of the executable I noticed that the version created by doing objcopy -R do *not* contain the GNU_RELRO type under Program Headers.

Is it possible that the GNU_RELRO program header is the reason I am unable to get a backtrace when creating and then loading a core file with gdb?

How do I compile an executable without the GNU_RELRO program header?

Revision history for this message
tom anderson (xentoma) wrote :

I found this patch

 https://build.opensuse.org/package/view_file?file=gdb-7.2-bnc596646.patch&package=gdb&project=openSUSE%3AFactory&srcmd5=fa6606a08c53d3cff59d9b6bf6991afb

and build the latest gnu gdb (ftp://sourceware.org/pub/gdb/snapshots/current/gdb-weekly-7.2.50.20101207.tar.bz2)
with and without hand merging the patch. The patched version gcore command produces a core which provides a meaningful backtrace. the unpatched version does not.

This gore patch needs to be applied to for ubuntu.

Revision history for this message
Kees Cook (kees) wrote :

Ah-ha! Excellent find. Can you attach the patch you ended up using? I wonder if this making its way into the upstream gdb or not. Some details on relro options are here: https://wiki.ubuntu.com/CompilerFlags specifically, -Wl,-z,norelro will turn it off.

Revision history for this message
tom anderson (xentoma) wrote :

Thanks for the info on the compiler flags. I was searching for a way to turn it off and did not find that.

I will generate a patch and post it.

Revision history for this message
Kees Cook (kees) wrote :

Actually, I pulled the proposed upstream patches (they apply cleanly), and will upload it shortly.

Kees Cook (kees)
Changed in gdb (Ubuntu Natty):
assignee: nobody → Kees Cook (kees)
status: New → Fix Committed
Changed in gdb (Ubuntu Maverick):
status: New → Confirmed
Changed in gdb (Ubuntu Natty):
importance: Undecided → Low
Changed in gdb (Ubuntu Maverick):
importance: Undecided → Low
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package gdb - 7.2-1ubuntu6

---------------
gdb (7.2-1ubuntu6) natty; urgency=low

  * Add support to "gcore" command for executables with GNU_RELRO
    sections (Debian bug 606667, LP: #680588).
 -- Kees Cook <email address hidden> Thu, 09 Dec 2010 19:47:22 -0800

Changed in gdb (Ubuntu Natty):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in gdb (Ubuntu Lucid):
status: New → Confirmed
Kees Cook (kees)
Changed in gdb (Ubuntu Lucid):
importance: Undecided → Low
Kees Cook (kees)
Changed in gdb (Ubuntu Maverick):
status: Confirmed → Fix Committed
Changed in gdb (Ubuntu Lucid):
status: Confirmed → In Progress
Changed in gdb (Ubuntu Maverick):
assignee: nobody → Kees Cook (kees)
Changed in gdb (Ubuntu Lucid):
assignee: nobody → Kees Cook (kees)
Revision history for this message
Martin Pitt (pitti) wrote :

This is quite an intrusive patch, maverick's gdb is not totally broken, and maverick certainly isn't a very active development platform any more. Given the regression potential and that this is not a serious issue for users, I have some doubts that we should SRU this for maverick.

Revision history for this message
Kees Cook (kees) wrote : Re: [Bug 680588] Re: gcore produces core file without backtrace capability

On Thu, Sep 22, 2011 at 04:57:13AM -0000, Martin Pitt wrote:
> This is quite an intrusive patch, maverick's gdb is not totally broken,
> and maverick certainly isn't a very active development platform any
> more. Given the regression potential and that this is not a serious
> issue for users, I have some doubts that we should SRU this for
> maverick.

What about for Lucid? It's actually a larger patch for Lucid due to need to
backport some additional infrastructure that didn't exist yet...

Lucid is still going to be supported for a while...

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

This is some interest to have this back ported to Lucid. Is that a possibility?

Revision history for this message
Clint Byrum (clint-fewbar) wrote :

I have to agree with Martin that making this work on Maverick isn't nearly as important as making it work on Lucid.. given that Maverick will be unsupported in 7 months vs. lucid's 1.5 - 3.5 more years.

Are there enough affected users who would test the change that we should let it go through to maverick anyway?

Revision history for this message
Kees Cook (kees) wrote :

I feel like it's a pretty straight forward thing to test. Either gdb has
sensible output or it doesn't for this patch. :) That said, I was hoping to
see the Maverick SRU finished first so that the even more invasive Lucid
patch would then have the Maverick SRU to stand on for supporting evidence
of sanity.

Revision history for this message
Martin Pitt (pitti) wrote : Please test proposed package

Hello tom, or anyone else affected,

Accepted gdb into maverick-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

tags: added: verification-needed
Revision history for this message
Louis Bouchard (louis) wrote :

As joe mentionned it previously, is there any ETA on having this fixed for Lucid ? This is the version with the longest support timespan ?

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Can someone please test the package in maverick-proposed? If it gets verified and moved to updates, it might be possible to make it available to Lucid in some fashion (eg, a PPA).

Revision history for this message
Steve Beattie (sbeattie) wrote :

I've reproduced the failure to generate an accurate backtrace with the version of gdb in maverick-updates, 7.2-1ubuntu3, and can confirm that the version in maverick-proposed, 7.2-1ubuntu3.1, fixes the issue (on both i386 and amd64). A cursory but nowhere near comprehensive exercise of gdb features didn't turn up any obvious regressions. Marking verification-done.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package gdb - 7.2-1ubuntu3.1

---------------
gdb (7.2-1ubuntu3.1) maverick-proposed; urgency=low

  * Add support to "gcore" command for executables with GNU_RELRO
    sections (Debian bug 606667, LP: #680588).
 -- Kees Cook <email address hidden> Fri, 16 Sep 2011 11:17:31 -0700

Changed in gdb (Ubuntu Maverick):
status: Fix Committed → Fix Released
Revision history for this message
Steve Beattie (sbeattie) wrote :

I've uploaded the maverick-updates version of gdb that incorporates the fix for this into a ppa for lucid (10.04 LTS) at https://launchpad.net/~sbeattie/+archive/lp680588/ after testing with a local build that it fixes the issue there.

(Unfortunately, the PPA buildds are behind, so packages probably won't be available for another few hours or so.)

The changes between gdb 7.1-1ubuntu2 and 7.2-1ubuntu3.1 are significant enough that in my opinion it's not appropriate for an SRU, though it could conceivably go into backports.

Revision history for this message
Steve Beattie (sbeattie) wrote :

Closing the lucid (SRU) task as Won't Fix; if it's desired to go through the Backports process (https://wiki.ubuntu.com/UbuntuBackports) we can work through that process. Thanks.

Changed in gdb (Ubuntu Lucid):
status: In Progress → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.