Ubuntu
kexec-tools package

[UBUNTU 22.04] Kdump broken for s390 on Ubuntu 22.04.2

Bug #2030476 reported by bugproxy
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Fix Released
High
Skipper Bug Screeners
kexec-tools (Ubuntu)
Fix Released
High
Frank Heimes
Jammy
Fix Released
High
Frank Heimes
Lunar
Fix Released
High
Frank Heimes
Mantic
Fix Released
High
Frank Heimes

Bug Description

SRU Justification:
==================

[ Impact ]

 * The 'kdump-config load' step fails on s390x,
   while trying to create a dump with:
   $ kdump-config load
     * Creating symlink /var/lib/kdump/vmlinuz
     * Creating symlink /var/lib/kdump/initrd.img
     Unknown rela relocation: 0x14 0x6bc0901c
     * failed to load kdump kernel

 * Hence it's not possible to take a dump on jammy/s390x
   using kdump (which makes use of kexec).

 * The relocation entries of type R_390_PLT32DBL, that are generated
   by gcc 11.3 (and higher) for the kdump purgatory code are silently
   ignored, which leads to this kdump kernel load failures.

 * The solution is simply to handle all R_390_PLT32DBL relocation entries
   just like the R_390_PC32DBL ones.

[ Test Plan ]

 * Have an affected s390x system (LPAR, z/VM or KVM) installed
   with jammy or newer.

 * sudo apt install kdump-tools kexec-tools
   (ensure the patched kexec-tools package is pulled-in)

 * Select 'yes' on "Should kdump-tools be enabled by default?"

 * In case of a notification about a "Modified configuration file"
   chose "install the package maintainer's version" to get
   the new zipl.conf with the crashkernel argument.

 * sudo reboot

 * optional: systemctl status kdump-tools

 * sudo kdump-config load # see particularly here if it fails.
   Before reboot a valid output is:
   " * no crashkernel= parameter in the kernel cmdline"
   after reboot its:
   " * Cannot change symbolic links when kdump is loaded"
   but no crash.

 * optional: kdump-config show
   (look for line "current state: ready to kdump"

 * and to complete the overall case, trigger a dump:

 * cat /proc/sys/kernel/sysrq
   127
   (anything else than "0" is good here and means enabled, but better do:)

 * sudo sysctl -w kernel.sysrq=1

 * sudo -s
   echo c > /proc/sysrq-trigger
   (you'll likely see "client_loop: send disconnect: Broken pipe")

 * Now watch the console output and wait until system is back up again
   and you'll find the dump in /var/crash:
   202308091605
   kdump_lock
   kexec_cmd
   linux-image-6.2.0-26-generic-202308091605.crash

[ Where problems could occur ]

 * First of all the changes are purely s390x related,
   only arch/s390/kexec-elf-rel-s390.c is touched in kexec-tools.

 * In kexec-tools all R_390_PLT32DBL relocation entries are
   handled in the same way like the R_390_PC32DBL entries:
   - else if (r_type == R_390_PC32DBL)
   + else if (r_type == R_390_PC32DBL || r_type == R_390_PLT32DBL)

 * This is btw. similar on how this is handled in the kernel.
   All relocation entries (r_type) are simply changed from
   R_390_PLT32DBL to R_390_PC32DBL.
   + if (r_type == R_390_PLT32DBL)
   + r_type = R_390_PC32DBL;

 * This could become problematic _if_ a special handling of
   R_390_PLT32DBL would be needed (which is not the case here).

 * So this works as long as no further differentiation between both
   relocation entries are needed.

[ Other Info ]

 * Jammy does not have the kernel nor the kexec-tools patch included.

 * lunar (with kernel 6.2) and mantic (with kernel 6.5, let's say > 6.2)
   have the kernel patch already in, since it got upstream accepted with 5.19.
   So these only need the kexec-tools patch, because both are still on
   kexec-tools v2.0.22, but the patch is only upstream with v2.0.24 and newer.

 * And even for Jammy the commit
   "s390/kexec: handle R_390_PLT32DBL rela in arch_kexec_apply_relocations_add()"
   got already cherry-picked (here: 8ca23f0691841e4d83722ec5b2020259dd5d786e
   and incl. in jammy since Ubuntu-5.15.0-19.19).
   This was done as part of "Jammy update: v5.15.15 upstream stable release"
   based on LP#1958418, because the upstream commit
   abf0e8e4ef25478a4390115e6a953d589d1f9ffd was tagged for stable updates.

 * So we can ignore the kernels of the affected Ubuntu releases,
   since they all have the needed commits included,
   and can focus on the exec-tools SRU only.
__________

== Reported by <email address hidden> - 2023-08-07 03:10:48 ==

---Problem Description---
Kdump broken for s390

---uname output---
Linux m836037 5.15.0-78-generic #85-Ubuntu SMP Fri Jul 7 15:25:49 UTC 2023 s390x s390x s390x GNU/Linux

Machine Type = 3906-704

---Debugger---
A debugger is not configured

---Steps to Reproduce---
1. apt install kdump-tools
   enable kexec and kdump
2. reboot
3. kdump-config load
->
 # kdump-config load
  * Creating symlink /var/lib/kdump/vmlinuz
  * Creating symlink /var/lib/kdump/initrd.img
 Unknown rela relocation: 0x14 0x6bc0901c
  * failed to load kdump kernel

See original description
bugproxy (bugproxy)
tags: added: architecture-s39064 bugnameltc-203344 severity-high targetmilestone-inin---
Changed in ubuntu:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu → linux (Ubuntu)
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2023-08-07 04:48 EDT-------
A fix for this problem is already available. We are working on the backport to Jammy.

Frank Heimes (fheimes)
affects: linux (Ubuntu) → kdump-tools (Ubuntu)
Changed in kdump-tools (Ubuntu):
importance: Undecided → High
Changed in ubuntu-z-systems:
importance: Undecided → Medium
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
Changed in kdump-tools (Ubuntu):
assignee: Skipper Bug Screeners (skipper-screen-team) → nobody
Changed in ubuntu-z-systems:
status: New → Incomplete
Changed in kdump-tools (Ubuntu):
status: New → Incomplete
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2023-08-08 05:25 EDT-------
There are two things that need to be fixed for this problem.

@Alexander: Thanks for providing the fixes as follows:

--- Part 1 ------------------
Description: s390/kexec: handle R_390_PLT32DBL rela in arch_kexec_apply_relocations_add()
Symptom: Loading of kdump kernel fails.
Problem: The relocation entries of type R_390_PLT32DBL generated by gcc 11.3 and higher for kdump purgatory code are silently ignored.
This leads to kdump kernel load failures.
Solution: Handle all R_390_PLT32DBL just like R_390_PC32DBL ones.
Reproduction: Install kdump-tools, reboot and try to activate the kdump service.
# apt-get install kdump-tools
# reboot
# systemctl enable kdump-tools
# systemctl status kdump-tools
...
Unknown rela relocation: 0x14 0x3a000901c
...
Upstream-ID: 41967a37b8eedfee15b81406a9f3015be90d3980
abf0e8e4ef25478a4390115e6a953d589d1f9ffd

--- Part 2 ------------------
There is a corresponding kexec-tools bugfix which is required too:

https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/commit/?id=186e7b0752d8fce1618fa37519671c834c46340e

Revision history for this message
Frank Heimes (fheimes) wrote :

Since both patches (kexec-tools and kernel) are needed to fix this,
not only jammy is affected and requires both fixes (kernel and kexec-tools),
but also lunar and mantic are affected.

But since lunar comes with kernel 6.2 and mantic with 6.5,
the kernel patch is already in (since it got upstream accepted with 5.19),
so both require the kexec-tools patch only
(since it got upstream accepted with 2.0.24 and lunar and mantic are still on 2.0.22).

affects: kdump-tools (Ubuntu) → kexec-tools (Ubuntu)
Changed in kexec-tools (Ubuntu):
status: Incomplete → New
Changed in ubuntu-z-systems:
status: Incomplete → New
no longer affects: linux (Ubuntu Lunar)
no longer affects: linux (Ubuntu Mantic)
Revision history for this message
Frank Heimes (fheimes) wrote :

Test builds are running at:
https://launchpad.net/~fheimes/+archive/ubuntu/lp2030476
(for now mantic only)

And debdiff for mantic attached ...

Changed in kexec-tools (Ubuntu Mantic):
status: New → In Progress
Changed in ubuntu-z-systems:
status: New → In Progress
Ubuntu Foundations Team Bug Bot (crichton)
tags: added: patch
Frank Heimes (fheimes)
description: updated
Frank Heimes (fheimes)
Changed in kexec-tools (Ubuntu Lunar):
status: New → In Progress
Changed in kexec-tools (Ubuntu Jammy):
status: New → In Progress
Changed in kexec-tools (Ubuntu Lunar):
importance: Undecided → High
Changed in kexec-tools (Ubuntu Jammy):
importance: Undecided → High
Changed in ubuntu-z-systems:
importance: Medium → High
Changed in kexec-tools (Ubuntu Mantic):
assignee: nobody → Frank Heimes (fheimes)
Changed in kexec-tools (Ubuntu Lunar):
assignee: nobody → Frank Heimes (fheimes)
Changed in kexec-tools (Ubuntu Jammy):
assignee: nobody → Frank Heimes (fheimes)
Revision history for this message
Frank Heimes (fheimes) wrote :

Updated packages for all three affected Ubuntu releases were successfully build and are available here:
https://launchpad.net/~fheimes/+archive/ubuntu/lp2030476
and pre-tested (following the test plan above).

I'm attaching a tgz with all 3 debdiffs (please ignore the one in comment #4, I modified the changelog a bit).

description: updated
Revision history for this message
Frank Heimes (fheimes) wrote :

I'm subscribing 'ubuntu-sponsors' start to work on the kernel team submission in parallel ...

Revision history for this message
Frank Heimes (fheimes) wrote :

To my (joyfully) surprise I've just found that commit "s390/kexec: handle R_390_PLT32DBL rela in arch_kexec_apply_relocations_add()" got already cherry-picked for jammy (here: 8ca23f0691841e4d83722ec5b2020259dd5d786e) as part of "Jammy update: v5.15.15 upstream stable release" at LP#1958418, since the upstream commit abf0e8e4ef25478a4390115e6a953d589d1f9ffd was tagged for stable updates. That's how it should be.

I'm removing the "affects kernel" entry again and will update the justification accordingly.

no longer affects: linux (Ubuntu)
no longer affects: linux (Ubuntu Jammy)
Frank Heimes (fheimes)
description: updated
Revision history for this message
Brian Murray (brian-murray) wrote :

I've sponsored all three of your debdiffs, thanks!

Revision history for this message
Robie Basak (racb) wrote : Please test proposed package

Hello bugproxy, or anyone else affected,

Accepted kexec-tools into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/kexec-tools/1:2.0.22-2ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in kexec-tools (Ubuntu Lunar):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-lunar
Changed in kexec-tools (Ubuntu Jammy):
status: In Progress → Fix Committed
tags: added: verification-needed-jammy
Revision history for this message
Robie Basak (racb) wrote :

Hello bugproxy, or anyone else affected,

Accepted kexec-tools into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/kexec-tools/1:2.0.22-2ubuntu2.22.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Frank Heimes (fheimes) wrote :

Successfully verified on jammy - see attachment.
Adjusting the tags accordingly...

tags: added: verification-done-jammy
removed: verification-needed-jammy
Revision history for this message
Frank Heimes (fheimes) wrote :

And also successfully verified on lunar - see attachment.
Adjusting the tags again...

tags: added: verification-done verification-done-lunar
removed: verification-needed verification-needed-lunar
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2023-08-23 07:23 EDT-------
I can confirm that kdump works with the kexec-tools from jammy-proposed. Thanks.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package kexec-tools - 1:2.0.22-2ubuntu3.2

---------------
kexec-tools (1:2.0.22-2ubuntu3.2) lunar; urgency=medium

  * Add d/p/lp2030476-s390-handle-R_390_PLT32DBL-reloc-entries-in-machine.patch
    to fix kdump kernel load failure on s390x. (LP: #2030476)

 -- Frank Heimes <email address hidden> Wed, 09 Aug 2023 15:06:43 +0200

Changed in kexec-tools (Ubuntu Lunar):
status: Fix Committed → Fix Released
Revision history for this message
Robie Basak (racb) wrote : Update Released

The verification of the Stable Release Update for kexec-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package kexec-tools - 1:2.0.22-2ubuntu2.22.04.2

---------------
kexec-tools (1:2.0.22-2ubuntu2.22.04.2) jammy; urgency=medium

  * Add d/p/lp2030476-s390-handle-R_390_PLT32DBL-reloc-entries-in-machine.patch
    to fix kdump kernel load failure on s390x. (LP: #2030476)

 -- Frank Heimes <email address hidden> Wed, 09 Aug 2023 15:12:40 +0200

Changed in kexec-tools (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2023-08-24 07:37 EDT-------
Verified with package from jammy-updates. The bug can be closed from my point of view. Thanks!

bugproxy (bugproxy)
tags: added: targetmilestone-inin2204
removed: targetmilestone-inin---
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package kexec-tools - 1:2.0.22-2ubuntu5

---------------
kexec-tools (1:2.0.22-2ubuntu5) mantic; urgency=medium

  * Add d/p/lp2030476-s390-handle-R_390_PLT32DBL-reloc-entries-in-machine.patch
    to fix kdump kernel load failure on s390x. (LP: #2030476)

 -- Frank Heimes <email address hidden> Tue, 08 Aug 2023 21:14:16 +0200

Changed in kexec-tools (Ubuntu Mantic):
status: In Progress → Fix Released
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.