Ubuntu
crash package

fails on arm64 w/ 5.4 kernel: crash: cannot determine VA_BITS_ACTUAL: please use /proc/kcore

Bug #1858958 reported by dann frazier
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
crash (Ubuntu)
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Undecided
Unassigned
Eoan
Fix Released
Undecided
Unassigned

Bug Description

[impact]

autopkgtest failures

[test case]

crash autopkgtests have began to fail on arm64, as seen here:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/arm64/c/crash/20191223_190616_50c73@/log.gz

The significant difference between this test and the last successful one is that the host kernel has moved from 5.3 to 5.4. As crash needs to understand some internals of the kernel memory, it often needs updates to deal with new kernels.

I've verified that the existing crash still works fine w/ a 5.3 kernel[1], but does not w/ 5.4:

$ sudo crash -st /usr/lib/debug/boot/vmlinux-$(uname -r)

crash: cannot determine VA_BITS_ACTUAL: please use /proc/kcore

However, note hte "please use /proc/kcore" - if I point crash at that instead of whatever it normally uses (/dev/mem?) it is able to start [2]. The same is also true w/ current upstream as of commit 7c2d41e btw - it fails w/o /proc/kcore and works with it.

[1]
ubuntu@d05-3:~$ sudo crash /usr/lib/debug/boot/vmlinux-5.3.0-26-generic

crash 7.2.6
Copyright (C) 2002-2019 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-unknown-linux-gnu"...

      KERNEL: /usr/lib/debug/boot/vmlinux-5.3.0-26-generic
    DUMPFILE: /proc/kcore
        CPUS: 64
        DATE: Wed Jan 8 22:32:40 2020
      UPTIME: 4 days, 20:46:37
LOAD AVERAGE: 1.09, 0.76, 0.46
       TASKS: 836
    NODENAME: d05-3
     RELEASE: 5.3.0-26-generic
     VERSION: #28-Ubuntu SMP Wed Dec 18 05:39:18 UTC 2019
     MACHINE: aarch64 (unknown Mhz)
      MEMORY: 127.7 GB
         PID: 19415
     COMMAND: "crash"
        TASK: ffff801464623b00 [THREAD_INFO: ffff801464623b00]
         CPU: 13
       STATE: TASK_RUNNING (ACTIVE)

crash> quit

[2]
ubuntu@d05-3:~$ sudo crash -st /usr/lib/debug/boot/vmlinux-$(uname -r) /proc/kcore
Thu Jan 9 00:20:20 2020

ubuntu@d05-3:~$ sudo crash /usr/lib/debug/boot/vmlinux-$(uname -r) /proc/kcore

crash 7.2.6
Copyright (C) 2002-2019 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-unknown-linux-gnu"...

      KERNEL: /usr/lib/debug/boot/vmlinux-5.4.0-9-generic
    DUMPFILE: /proc/kcore
        CPUS: 64
        DATE: Thu Jan 9 00:20:57 2020
      UPTIME: 03:36:15
LOAD AVERAGE: 0.56, 0.31, 0.26
       TASKS: 823
    NODENAME: d05-3
     RELEASE: 5.4.0-9-generic
     VERSION: #12-Ubuntu SMP Mon Dec 16 22:33:31 UTC 2019
     MACHINE: aarch64 (unknown Mhz)
      MEMORY: 127.7 GB
         PID: 44363
     COMMAND: "crash"
        TASK: ffff00179ae0da00 [THREAD_INFO: ffff00179ae0da00]
         CPU: 11
       STATE: TASK_RUNNING (ACTIVE)

crash>

[regression potential]

autopkgtest change only; any regression would involve false positive/negative tests.

See original description

Related branches

~mfo/ubuntu/+source/crash:merge-lp2047861-noble
Sergio Durigan Junior (community): Approve
Diff: 253 lines (+166/-4)
4 files modified
debian/changelog (+145/-0)
debian/control (+2/-1)
debian/tests/control (+2/-2)
debian/tests/live (+17/-1)
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package crash - 7.2.6-1ubuntu3

---------------
crash (7.2.6-1ubuntu3) focal; urgency=medium

  * Rebuild with updated binutils, fixing an "illegal insruction" crash
    on arm64. LP: #1859226.
  * Update the maintainer field for Ubuntu.

 -- dann frazier <email address hidden> Sat, 11 Jan 2020 09:36:27 -0700

Changed in crash (Ubuntu):
status: New → Fix Released
Dan Streetman (ddstreet)
description: updated
Revision history for this message
Andy Whitcroft (apw) wrote : Please test proposed package

Hello dann, or anyone else affected,

Accepted crash into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/crash/7.2.8-1ubuntu0.19.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in crash (Ubuntu Eoan):
status: New → Fix Committed
tags: added: verification-needed verification-needed-eoan
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello dann, or anyone else affected,

Accepted crash into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/crash/7.2.8-1ubuntu0.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in crash (Ubuntu Bionic):
status: New → Fix Committed
tags: added: verification-needed-bionic
Revision history for this message
dann frazier (dannf) wrote :
Download full text (27.4 KiB)

eoan verification:
ubuntu@crash-eoan:~/crash-7.2.8$ sudo sh ./debian/tests/live
Adding linux-image debug symbols.
Executing: /tmp/apt-key-gpghome.7LWILREpSP/gpg.1.sh --keyserver keyserver.ubuntu.com --recv-keys C8CAB6595FDFF622
gpg: key C8CAB6595FDFF622: 5 signatures not checked due to missing keys
gpg: key C8CAB6595FDFF622: "Ubuntu Debug Symbol Archive Automatic Signing Key (2016) <email address hidden>" 3 new signatures
gpg: Total number processed: 1
gpg: new signatures: 3
Hit:1 http://ports.ubuntu.com/ubuntu-ports eoan InRelease
Hit:2 http://ports.ubuntu.com/ubuntu-ports eoan-updates InRelease
Hit:3 http://archive.canonical.com/ubuntu eoan InRelease
Hit:4 http://ppa.launchpad.net/dannf/dannf/ubuntu eoan InRelease
Hit:5 http://ports.ubuntu.com/ubuntu-ports eoan-backports InRelease
Ign:6 http://ddebs.ubuntu.com eoan InRelease
Hit:7 http://ports.ubuntu.com/ubuntu-ports eoan-security InRelease
Ign:8 http://ddebs.ubuntu.com eoan-updates InRelease
Hit:9 http://ports.ubuntu.com/ubuntu-ports eoan-proposed InRelease
Ign:10 http://ddebs.ubuntu.com eoan-proposed InRelease
Hit:11 http://ddebs.ubuntu.com eoan Release
Hit:12 http://ddebs.ubuntu.com eoan-updates Release
Hit:13 http://ddebs.ubuntu.com eoan-proposed Release
Reading package lists... Done
W: Target Packages (main/binary-arm64/Packages) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/ddebs.list:1
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/ddebs.list:1
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/ddebs.list:1
W: Target CNF (main/cnf/Commands-arm64) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/ddebs.list:1
W: Target CNF (main/cnf/Commands-all) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/ddebs.list:1
W: Target Packages (main/binary-arm64/Packages) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/ddebs.list:1
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/ddebs.list:1
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/ddebs.list:1
W: Target CNF (main/cnf/Commands-arm64) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/ddebs.list:1
W: Target CNF (main/cnf/Commands-all) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/ddebs.list:1
Hit:1 http://ppa.launchpad.net/dannf/dannf/ubuntu eoan InRelease
Hit:2 http://ports.ubuntu.com/ubuntu-ports eoan InRelease
Hit:3 http://archive.canonical.com/ubuntu eoan InRelease
Ign:4 http://ddebs.ubuntu.com eoan InRelease
Ign:5 http://ddebs.ubuntu.com eoan-updates InRelease
Hit:6 http://ports.ubuntu.com/ubuntu-ports eoan-updates InRelease
Hit:7 http://ports.ubun...

tags: added: verification-done-eoan
removed: verification-needed-eoan
Revision history for this message
dann frazier (dannf) wrote :
Download full text (28.2 KiB)

bionic verification:

ubuntu@crash-bionic:~/crash-7.2.8$ sudo sh ./debian/tests/live
Adding linux-image debug symbols.
Executing: /tmp/apt-key-gpghome.vuS9jI2yc6/gpg.1.sh --keyserver keyserver.ubuntu.com --recv-keys C8CAB6595FDFF622
gpg: key C8CAB6595FDFF622: 5 signatures not checked due to missing keys
gpg: key C8CAB6595FDFF622: "Ubuntu Debug Symbol Archive Automatic Signing Key (2016) <email address hidden>" 3 new signatures
gpg: Total number processed: 1
gpg: new signatures: 3
Hit:1 http://archive.canonical.com/ubuntu bionic InRelease
Hit:2 http://ports.ubuntu.com/ubuntu-ports bionic InRelease
Hit:3 http://ports.ubuntu.com/ubuntu-ports bionic-updates InRelease
Ign:4 http://ddebs.ubuntu.com bionic InRelease
Hit:5 http://ppa.launchpad.net/dannf/dannf/ubuntu bionic InRelease
Hit:6 http://ports.ubuntu.com/ubuntu-ports bionic-backports InRelease
Ign:7 http://ddebs.ubuntu.com bionic-updates InRelease
Hit:8 http://ports.ubuntu.com/ubuntu-ports bionic-security InRelease
Ign:9 http://ddebs.ubuntu.com bionic-proposed InRelease
Hit:10 http://ports.ubuntu.com/ubuntu-ports bionic-proposed InRelease
Hit:11 http://ddebs.ubuntu.com bionic Release
Hit:12 http://ddebs.ubuntu.com bionic-updates Release
Hit:13 http://ddebs.ubuntu.com bionic-proposed Release
Reading package lists... Done
W: Target Packages (main/binary-arm64/Packages) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/ddebs.list:1
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/ddebs.list:1
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/ddebs.list:1
W: Target CNF (main/cnf/Commands-arm64) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/ddebs.list:1
W: Target CNF (main/cnf/Commands-all) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/ddebs.list:1
W: Target Packages (main/binary-arm64/Packages) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/ddebs.list:1
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/ddebs.list:1
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/ddebs.list:1
W: Target CNF (main/cnf/Commands-arm64) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/ddebs.list:1
W: Target CNF (main/cnf/Commands-all) is configured multiple times in /etc/apt/sources.list:58 and /etc/apt/sources.list.d/ddebs.list:1
Hit:1 http://archive.canonical.com/ubuntu bionic InRelease
Hit:2 http://ports.ubuntu.com/ubuntu-ports bionic InRelease
Ign:3 http://ddebs.ubuntu.com bionic InRelease
Hit:4 http://ppa.launchpad.net/dannf/dannf/ubuntu bionic InRelease
Hit:5 http://ports....

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package crash - 7.2.8-1ubuntu0.19.10.1

---------------
crash (7.2.8-1ubuntu0.19.10.1) eoan; urgency=medium

  * Version bump from Focal, based on upstream release 7.2.8 (LP: #1865041):
    - Multiple fixes up to kernel 5.5-rc1 (e.g., fixes for kmem/timer/log
      commands, for kernels with symbol namespaces and with KASLR enabled).
    - Improve handling of /proc/kcore (live memory) for x86_64 kernels with
      CONFIG_RANDOMIZE_BASE enabled.
    - Fix ARM64 page size handling on kernel 4.4 and before.
    - Improve offset gathering and ELF support on KASLR kernels in S390X.
    - Support xz compressed vmlinux.
    - Fallback to log buffer parsing in case panic task can't be determined
      by normal means in the dumpfile.
  * Remaining changes:
    - Update the maintainer field for Ubuntu.
    - If the "live" autopkgtest fails with a recommendation to try
      /proc/kcore instead of the default, attempt that before failing
      the test. LP: #1858958.
  * Fix test, as if will return 0 when no cases were true.

 -- <email address hidden> (Guilherme G. Piccoli) Thu, 27 Feb 2020 11:12:51 -0300

Changed in crash (Ubuntu Eoan):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for crash has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package crash - 7.2.8-1ubuntu0.18.04.1

---------------
crash (7.2.8-1ubuntu0.18.04.1) bionic; urgency=medium

  * Version bump from Focal, based on upstream release 7.2.8 (LP: #1865041):
    - Multiple fixes up to kernel 5.5-rc1 (e.g., fixes for kmem/timer/log
      commands, for kernels with symbol namespaces and with KASLR enabled).
    - Improve handling of /proc/kcore (live memory) for x86_64 kernels with
      CONFIG_RANDOMIZE_BASE enabled.
    - Fix ARM64 page size handling on kernel 4.4 and before.
    - Improve offset gathering and ELF support on KASLR kernels in S390X.
    - Support xz compressed vmlinux.
    - Fallback to log buffer parsing in case panic task can't be determined
      by normal means in the dumpfile.
  * Remaining changes:
    - Update the maintainer field for Ubuntu.
    - If the "live" autopkgtest fails with a recommendation to try
      /proc/kcore instead of the default, attempt that before failing
      the test. LP: #1858958.
  * Fix test, as if will return 0 when no cases were true.

 -- <email address hidden> (Guilherme G. Piccoli) Thu, 27 Feb 2020 11:12:51 -0300

Changed in crash (Ubuntu Bionic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.