Ubuntu
crash package

Unable to analyse vmcore/dump via crash due to bad kernel debug info build

Bug #1607894 reported by bugproxy on 2016-07-29

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	crash (Ubuntu)	Fix Released	High	Robert Hooker
	Trusty	Fix Released	High	Robert Hooker

Bug Description

[ Test case ]
* trigger a kernel crash dump
* use crash tool to analyse it
* ... on ppc64el architecture

== Comment: #0 - NAVEED A. UPPINANGADY SALIH - 2016-07-01 02:41:07 ==
---Problem Description---
Unable to analyse vmcore/kernel dumpfile generated by 4.4.0-29-generic kernel with ddeb kernel debuginfo available in http://ddebs.ubuntu.com/pool/main/l/linux/

http://ddebs.ubuntu.com/pool/main/l/linux/linux-image-4.4.0-29-generic-dbgsym_4.4.0-29.48_ppc64el.ddeb

---uname output---
4.4.0-29-generic #48~14.04.1-Ubuntu SMP Wed Jun 29 19:55:03 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

Machine Type = model : 8247-22L machine : PowerNV 8247-22L

---Steps to Reproduce---
# crash /usr/lib/debug/boot/vmlinux-4.4.0-29-generic /var/crash/201606300840/dump.201606300840
crash 7.0.3
...
This GDB was configured as "powerpc64le-unknown-linux-gnu"...

crash: invalid kernel virtual address: 61c2555c09c17994 type: "possible"
WARNING: cannot read cpu_possible_map
crash: invalid kernel virtual address: 8a0f6ddee20c4116 type: "present"
WARNING: cannot read cpu_present_map
crash: invalid kernel virtual address: 5a09344e320a1886 type: "online"
WARNING: cannot read cpu_online_map
WARNING: cannot read linux_banner string
crash: /usr/lib/debug/boot/vmlinux-4.4.0-29-generic and /var/crash/201606300840/dump.201606300840 do not match!

Userspace rpm: linux-image-4.4.0-29-generic-dbgsym_4.4.0-29.48_ppc64el.ddeb

== Comment: #5 - Naresh Bannoth - 2016-07-27 04:12:09 ==

The Latest kernel I am having is as follows

root@ltcalpine-lp6:~# uname -a
Linux ltcalpine-lp6 4.4.0-31-generic #50~14.04.1-Ubuntu SMP Wed Jul 13 01:03:56 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux
root@ltcalpine-lp6:~#

====>>> the issue same as posted originally.

...
This GDB was configured as "powerpc64le-unknown-linux-gnu"...

crash: invalid kernel virtual address: 61c2555c09c17994 type: "possible"
WARNING: cannot read cpu_possible_map
crash: invalid kernel virtual address: 8a0f6ddee20c4116 type: "present"
WARNING: cannot read cpu_present_map
crash: invalid kernel virtual address: 5a09344e320a1886 type: "online"
WARNING: cannot read cpu_online_map
WARNING: cannot read linux_banner string
crash: /usr/lib/debug/boot/vmlinux-4.4.0-31-generic and /var/crash/201607270248/dump.201607270248 do not match!

== Comment: #6 - Hari Krishna Bathini - 2016-07-28 11:49:31 ==
The kernel is compiled with gcc version 4.8.4 while the vmlinux pulled
from http://ddebs.ubuntu.com/pool/main/l/linux/ is compiled with gcc
version 5.3.1 which is the source of the issue.

Resolved this by pulling the kernel debug symbols package from:

deb http://ddebs.ubuntu.com/ -updates main

which indeed has the vmlinux compiled with gcc version 4.8.4.
But I got this error:

crash: invalid structure member offset: module_num_symtab
FILE: kernel.c LINE: 3049 FUNCTION: module_init()

after the update owning to couple of missing patches listed below:

  commit 6f1f78e33474d00d5f261d7ed9d835c558b34d61
  Author: Dave Anderson <email address hidden>
  Date: Wed Jan 20 09:56:36 2016 -0500

Fix for the changes made to the kernel module structure introduced by
this kernel commit for Linux 4.5 and later kernels:

commit 7523e4dc5057e157212b4741abd6256e03404cf1
module: use a structure to encapsulate layout.

    Without the patch, the crash session fails during initialization
    with the error message: "crash: invalid structure member offset:
    module_init_text_size".
    (<email address hidden>)

  commit 098cdab16dfa6a85e9dad2cad604dee14ee15f66
  Author: Dave Anderson <email address hidden>
  Date: Fri Feb 12 14:32:53 2016 -0500

Fix for the changes made to the kernel module structure introduced by
this kernel commit for Linux 4.5 and later kernels:

commit 8244062ef1e54502ef55f54cced659913f244c3e
modules: fix longstanding /proc/kallsyms vs module insertion race.

    Without the patch, the crash session fails during initialization
    with the error message: "crash: invalid structure member offset:
    module_num_symtab".
    (<email address hidden>)

Applying the above patches on top of crash tool version 7.0.3-3ubuntu4.4,
was able to analyze the dump. The patches apply cleanly..

Thanks
Hari

== Comment: #8 - Naresh Bannoth - 2016-07-29 06:59:23 ==
Verified with 4.4.0-31-generic kernel.
on Applying the patches mentioned in the #C 6, it is working fine.
I took the Compiled Binary attached in the BUG and able to analyse the Dumpfile without any issues.

Thanks
Naresh

== Comment: #9 - Hari Krishna Bathini - 2016-07-29 07:47:25 ==

To sum this up, two things to make this work:

1. Install debug symbols package from here

    deb http://ddebs.ubuntu.com/ trusty main restricted universe multiverse
    deb http://ddebs.ubuntu.com/ trusty-security main restricted universe multiverse
    deb http://ddebs.ubuntu.com/ trusty-updates main restricted universe multiverse
    deb http://ddebs.ubuntu.com/ trusty-proposed main restricted universe multiverse

instead of http://ddebs.ubuntu.com/pool/main/l/linux/ as the gcc versions don't match.

2. Use a crash binary that has the upstream patches 6f1f78 & 098cda

Thanks
Hari

See original description

Tags:

Revision history for this message

bugproxy (bugproxy) wrote on 2016-07-29: Compressed working crash binary - compiled with crash_7.0.3-3ubuntu4.4 along with commits 6f1f78 & 098cda

Compressed working crash binary - compiled with crash_7.0.3-3ubuntu4.4 along with commits 6f1f78 & 098cda Edit (3.1 MiB, application/gzip)

Default Comment by Bridge

tags:	added: architecture-ppc64le bugnameltc-143307 severity-critical targetmilestone-inin14045
Changed in ubuntu:
assignee:	nobody → Taco Screen team (taco-screen-team)
affects:	ubuntu → linux (Ubuntu)

Tim Gardner (timg-tpi) on 2016-07-29

affects:

linux (Ubuntu) → crash (Ubuntu)

Revision history for this message

bugproxy (bugproxy) wrote on 2016-08-25: Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2016-08-25 06:05 EDT-------
Can we expect upstream crash utility commits 6f1f78e33474 & 098cdab16dfa
included in crash package for the next update?

Thanks
Hari

Revision history for this message

Breno Leitão (breno-leitao) wrote on 2016-11-17:

Hi Canonical, any update on this bug? It is an important fix for 14.04 IMO

Robert Hooker (sarvatt) on 2016-12-01

Changed in crash (Ubuntu):
assignee:	Taco Screen team (taco-screen-team) → Robert Hooker (sarvatt)

Revision history for this message

Robert Hooker (sarvatt) wrote on 2016-12-07:

debdiff containing the fix Edit (5.3 KiB, text/plain)

crash (7.0.3-3ubuntu4.5) trusty; urgency=medium

* Fix for: Unable to analyse vmcore/dump via crash due to bad kernel
debug info build (LP: #1607894).

-- Robert Hooker <email address hidden> Thu, 01 Dec 2016 12:03:17 -0500

Revision history for this message

Ubuntu Foundations Team Bug Bot (crichton) wrote on 2016-12-07:

The attachment "debdiff containing the fix" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags:

added: patch

Dimitri John Ledkov (xnox) on 2016-12-08

description:	updated
Changed in crash (Ubuntu):
status:	New → Fix Released
Changed in crash (Ubuntu Trusty):
status:	New → In Progress
assignee:	nobody → Robert Hooker (sarvatt)
importance:	Undecided → High

Revision history for this message

Dimitri John Ledkov (xnox) wrote on 2016-12-08:

https://launchpad.net/ubuntu/trusty/+queue?queue_state=1&queue_text=crash

In trusty unapproved queue.

Revision history for this message

Brian Murray (brian-murray) wrote on 2016-12-09: Please test proposed package

Hello bugproxy, or anyone else affected,

Accepted crash into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/crash/7.0.3-3ubuntu4.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in crash (Ubuntu Trusty):
status:	In Progress → Fix Committed
tags:	added: verification-needed

Mathew Hodson (mhodson) on 2017-01-08

Changed in crash (Ubuntu):
importance:	Undecided → High

Revision history for this message

bugproxy (bugproxy) wrote on 2017-03-09: Comment bridged from LTC Bugzilla

Download full text (3.4 KiB)

------- Comment From <email address hidden> 2017-03-09 08:06 EDT-------
This issue is resolved in latest 14.04 build.. closing this bug as fixed
.13.0-112-generic dump.20170309072503090725# crash /usr/lib/debug/boot/vmlinux-3.

crash 7.0.3
Copyright (C) 2002-2013 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu"...

KERNEL: /usr/lib/debug/boot/vmlinux-3.13.0-112-generic
DUMPFILE: dump.201703090725 [PARTIAL DUMP]
CPUS: 80
DATE: Thu Mar 9 07:23:46 2017
UPTIME: 00:11:05
LOAD AVERAGE: 0.00, 0.04, 0.04
TASKS: 640
NODENAME: ltciofvtr-hab2
RELEASE: 3.13.0-112-generic
VERSION: #159-Ubuntu SMP Fri Mar 3 15:29:14 UTC 2017
MACHINE: ppc64le (2926 Mhz)
MEMORY: 256 GB
PANIC: "Oops: Kernel access of bad area, sig: 11 [#1]" (check log for details)
PID: 1845
COMMAND: "bash"
TASK: c000003f66100000 [THREAD_INFO: c000003f78200000]
CPU: 20
STATE: TASK_RUNNING (PANIC)

crash> bt
PID: 1845 TASK: c000003f66100000 CPU: 20 COMMAND: "bash"
#0 [c000003f782034a0] .crash_kexec at c00000000013875c
#1 [c000003f78203690] .die at c000000000022618
#2 [c000003f78203730] .bad_page_fault at c00000000004a580
#3 [c000003f782037b0] handle_page_fault at c000000000009584
Data Access error [300] exception frame:
R0: c0000000005962ac R1: c000003f78203aa0 R2: c000000001661000
R3: 0000000000000063 R4: c000000004b85bd0 R5: c000000004b964c8
R6: 00000000000002b0 R7: c0000000018575c0 R8: c000000000e21000
R9: 0000000000000001 R10: 0000000000000000 R11: 00000000000000c2
R12: 0000000042422844 R13: c000000007e44600 R14: 0000010039206d68
R15: 000000001016e048 R16: 000000001013ab40 R17: 00000100392069a8
R18: 000000001016ff10 R19: 000000001016e048 R20: 00000000100c2f80
R21: 0000000000000000 R22: 0000000010172008 R23: 000000001016f880
R24: 0000000000000004 R25: 0000000000000000 R26: 0000000000000001
R27: c0000000014e8370 R28: c0000000019a4eb8 R29: 0000000000000063
R30: c0000000014bda68 R31: c0000000014e8730
NIP: c00000000059534c MSR: 9000000000009033 OR3: c000000000009318
CTR: c000000000595320 LR: c0000000005962ac XER: 0000000020000000
CCR: 0000000042422822 MQ: 0000000000000000 DAR: 0000000000000000
DSISR: 00000000420000...

------- Comment From naveedaus@in.ibm.com 2017-03-09 08:06 EDT-------
This issue is resolved in latest 14.04 build.. closing this bug as fixed
.13.0-112-generic dump.20170309072503090725# crash /usr/lib/debug/boot/vmlinux-3.

crash 7.0.3
Copyright (C) 2002-2013  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu"...

KERNEL: /usr/lib/debug/boot/vmlinux-3.13.0-112-generic
DUMPFILE: dump.201703090725  [PARTIAL DUMP]
CPUS: 80
DATE: Thu Mar  9 07:23:46 2017
UPTIME: 00:11:05
LOAD AVERAGE: 0.00, 0.04, 0.04
TASKS: 640
NODENAME: ltciofvtr-hab2
RELEASE: 3.13.0-112-generic
VERSION: #159-Ubuntu SMP Fri Mar 3 15:29:14 UTC 2017
MACHINE: ppc64le  (2926 Mhz)
MEMORY: 256 GB
PANIC: "Oops: Kernel access of bad area, sig: 11 [#1]" (check log for details)
PID: 1845
COMMAND: "bash"
TASK: c000003f66100000  [THREAD_INFO: c000003f78200000]
CPU: 20
STATE: TASK_RUNNING (PANIC)

crash> bt
PID: 1845   TASK: c000003f66100000  CPU: 20  COMMAND: "bash"
#0 [c000003f782034a0] .crash_kexec at c00000000013875c
#1 [c000003f78203690] .die at c000000000022618
#2 [c000003f78203730] .bad_page_fault at c00000000004a580
#3 [c000003f782037b0] handle_page_fault at c000000000009584
Data Access error  [300] exception frame:
R0:  c0000000005962ac    R1:  c000003f78203aa0    R2:  c000000001661000
R3:  0000000000000063    R4:  c000000004b85bd0    R5:  c000000004b964c8
R6:  00000000000002b0    R7:  c0000000018575c0    R8:  c000000000e21000
R9:  0000000000000001    R10: 0000000000000000    R11: 00000000000000c2
R12: 0000000042422844    R13: c000000007e44600    R14: 0000010039206d68
R15: 000000001016e048    R16: 000000001013ab40    R17: 00000100392069a8
R18: 000000001016ff10    R19: 000000001016e048    R20: 00000000100c2f80
R21: 0000000000000000    R22: 0000000010172008    R23: 000000001016f880
R24: 0000000000000004    R25: 0000000000000000    R26: 0000000000000001
R27: c0000000014e8370    R28: c0000000019a4eb8    R29: 0000000000000063
R30: c0000000014bda68    R31: c0000000014e8730
NIP: c00000000059534c    MSR: 9000000000009033    OR3: c000000000009318
CTR: c000000000595320    LR:  c0000000005962ac    XER: 0000000020000000
CCR: 0000000042422822    MQ:  0000000000000000    DAR: 0000000000000000
DSISR: 0000000042000000     Syscall Result: 0000000000000000
#4 [c000003f78203aa0] .sysrq_handle_crash at c00000000059534c
[Link Register ]  [c000003f78203aa0] .__handle_sysrq at c0000000005962ac
-- MORE --  forward: <SPACE>, <ENTER> or j  backward: b or k  quit: q
..

Revision history for this message

Louis Bouchard (louis) wrote on 2017-03-14:

I can confirm that this version fixes the following issue :

crash: invalid structure member offset: module_num_symtab
FILE: kernel.c LINE: 3049 FUNCTION: module_init()

using the latest linux-image-lts-xenial kernel

tags:

added: verification-done
removed: verification-needed

Revision history for this message

Launchpad Janitor (janitor) wrote on 2017-03-15:

#10

This bug was fixed in the package crash - 7.0.3-3ubuntu4.5

---------------
crash (7.0.3-3ubuntu4.5) trusty; urgency=medium

* Fix for: Unable to analyse vmcore/dump via crash due to bad kernel
debug info build (LP: #1607894).

-- Robert Hooker <email address hidden> Thu, 01 Dec 2016 12:03:17 -0500

Changed in crash (Ubuntu Trusty):
status:	Fix Committed → Fix Released

Revision history for this message

Brian Murray (brian-murray) wrote on 2017-03-15: Update Released

#11

The verification of the Stable Release Update for crash has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.