Ubuntu
linux package

kexec enable to load/kdump zstd compressed zimg

Bug #2037398 reported by Philip Roche
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
kdump-tools (Ubuntu)
Invalid
Undecided
Unassigned
kexec-tools (Ubuntu)
Triaged
Wishlist
Unassigned
linux (Ubuntu)
Fix Released
Critical
Unassigned
Ubuntu ubuntu-23.10

Bug Description

While testing the 6.5.0-6-generic proposed arm64 generic kernel I encountered issues being able to use kdump.

After enabling -proposed and installing the -proposed 6.5.0-6 kernel and rebooting I encountered the following:

`kdump-config show` shows `current state: Not ready to kdump` and looking at the status of the kdump-tools services I see `Cannot determine the file type of /var/lib/kdump/vmlinuz`

Full output:

```
ubuntu@cloudimg:~$ sudo kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_COREDIR: /var/crash
crashkernel addr: 0xde000000
   /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-6.5.0-6-generic
kdump initrd:
   /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-6.5.0-6-generic
current state: Not ready to kdump

kexec command:
  no kexec command recorded
ubuntu@cloudimg:~$ sudo service kdump-tools status
● kdump-tools.service - Kernel crash dump capture service
     Loaded: loaded (/lib/systemd/system/kdump-tools.service; enabled; preset: enabled)
     Active: active (exited) since Tue 2023-09-26 09:21:44 UTC; 5min ago
    Process: 515 ExecStart=/etc/init.d/kdump-tools start (code=exited, status=0/SUCCESS)
   Main PID: 515 (code=exited, status=0/SUCCESS)
        CPU: 4min 21.329s

Sep 26 09:16:14 cloudimg systemd[1]: Starting kdump-tools.service - Kernel crash dump capture service...
Sep 26 09:16:24 cloudimg kdump-tools[515]: Starting kdump-tools:
Sep 26 09:16:24 cloudimg kdump-tools[537]: * Creating symlink /var/lib/kdump/vmlinuz
Sep 26 09:16:32 cloudimg kdump-tools[580]: kdump-tools: Generating /var/lib/kdump/initrd.img-6.5.0-6-generic
Sep 26 09:21:42 cloudimg kdump-tools[537]: * Creating symlink /var/lib/kdump/initrd.img
Sep 26 09:21:43 cloudimg kdump-tools[5538]: Cannot determine the file type of /var/lib/kdump/vmlinuz
Sep 26 09:21:44 cloudimg kdump-tools[537]: * failed to load kdump kernel
Sep 26 09:21:44 cloudimg kdump-tools[5539]: failed to load kdump kernel
Sep 26 09:21:44 cloudimg systemd[1]: Finished kdump-tools.service - Kernel crash dump capture service.

ubuntu@cloudimg:~$ ls -al /var/lib/kdump/vmlinuz
lrwxrwxrwx 1 root root 29 Sep 26 09:16 /var/lib/kdump/vmlinuz -> /boot/vmlinuz-6.5.0-6-generic
ubuntu@cloudimg:~$ file /var/lib/kdump/vmlinuz
/var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-6.5.0-6-generic
ubuntu@cloudimg:~$ sudo file /boot/vmlinuz-6.5.0-6-generic
/boot/vmlinuz-6.5.0-6-generic: PE32+ executable (EFI application) Aarch64 (stripped to external PDB), for MS Windows, 2 sections
```

The reboot with 6.5.0-6 was successful and the reboot after linux-crashdump install was successful too.
I used https://ubuntu.com/server/docs/kernel-crash-dump guide for installing linux-crashdump and attempting to trigger a dump.

I used arm64 qcow cloud image from http://cloud-images.ubuntu.com/mantic/20230925/mantic-server-cloudimg-arm64.img to test the above emulated on amd64.

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 2037398

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Dimitri John Ledkov (xnox)
Changed in kdump-tools (Ubuntu):
status: New → Incomplete
Changed in kexec-tools (Ubuntu):
status: New → Triaged
Revision history for this message
Dimitri John Ledkov (xnox) wrote : Re: Unable to capture kernel crash dump using arm64 mantic 6.5.0-6-generic -proposed kernel

$ sudo /tmp/tmp.u3jP6LAlnj/usr/local/sbin/kexec --debug -l -c /boot/vmlinuz-6.5.0-6-generic
arch_process_options:178: command_line: (null)
arch_process_options:180: initrd: (null)
arch_process_options:182: dtb: (null)
arch_process_options:185: console: (null)
Try gzip decompression.
kernel: 0xffff8e4ef010 kernel_size: 0xda0988
set_phys_offset: phys_offset : 0000000040000000 (method : vmcoreinfo pt_note)
get_memory_ranges:+[0] 0000000040000000 - 000000023c58ffff
get_memory_ranges:- 00000000fa000000 - 00000000ffffffff
get_memory_ranges:- 00000001001dc000 - 00000001001dcfff
get_memory_ranges:- 0000000100210000 - 000000010021ffff
get_memory_ranges:- 0000000100220000 - 000000010022ffff
get_memory_ranges:- 0000000100230000 - 000000010023ffff
get_memory_ranges:- 0000000109540000 - 000000010a0fffff
get_memory_ranges:- 000000022fb50000 - 000000022fb50fff
get_memory_ranges:- 000000022fd5d000 - 000000022fd64fff
get_memory_ranges:- 0000000234400000 - 000000023c3fffff
get_memory_ranges:- 000000023c4a6000 - 000000023c4a6fff
get_memory_ranges:- 000000023c580000 - 000000023c58ffff
get_memory_ranges:+[9] 000000023c730000 - 000000023fbfffff
get_memory_ranges:- 000000023dbf1000 - 000000023e7f1fff
get_memory_ranges:- 000000023eaff000 - 000000023ebbefff
get_memory_ranges:- 000000023ebbf000 - 000000023ebbffff
get_memory_ranges:- 000000023ebc0000 - 000000023ebfffff
get_memory_ranges:- 000000023ec02000 - 000000023ec03fff
get_memory_ranges:- 000000023ec04000 - 000000023ec04fff
get_memory_ranges:- 000000023ec05000 - 000000023fbfffff
get_memory_ranges:+[12] 000000023ffe0000 - 000000023fffffff
get_memory_ranges:- 000000023ffe0000 - 000000023fffffff
System RAM ranges;
0000000040000000-00000000f9ffffff (0)
0000000100000000-00000001001dbfff (0)
00000001001dd000-000000010020ffff (0)
0000000100240000-000000010953ffff (0)
000000010a100000-000000022fb4ffff (0)
000000022fb51000-000000022fd5cfff (0)
000000022fd65000-00000002343fffff (0)
000000023c400000-000000023c4a5fff (0)
000000023c4a7000-000000023c57ffff (0)
000000023c730000-000000023dbf0fff (0)
000000023e7f2000-000000023eafefff (0)
000000023ec00000-000000023ec01fff (0)
elf_arm64_probe: Not an ELF executable.
image_arm64_probe: Bad arm64 image header.
pez_arm64_probe: PROBE.
pez_prepare: kexec can only decompress gziped and lzma images.
Cannot determine the file type of /boot/vmlinuz-6.5.0-6-generic

Even git snapshot of kexec-tools cannot handle zstd commpressed zimg

summary: - Unable to capture kernel crash dump using arm64 mantic 6.5.0-6-generic
- -proposed kernel
+ kexec enable to load/kdump zstd compressed zimg
Changed in kdump-tools (Ubuntu):
status: Incomplete → Invalid
Changed in linux (Ubuntu):
status: Incomplete → Triaged
Changed in kexec-tools (Ubuntu):
importance: Undecided → Wishlist
Changed in linux (Ubuntu):
importance: Undecided → Critical
tags: added: block-proposed-mantic
Revision history for this message
Philip Roche (philroche) wrote :
Download full text (3.2 KiB)

I have confirmed that this issue with not being able to capture kernel dump with a mantic arm64 kernel is not new. using the arm64 6.5 kernel (6.5.0-5) in the release pocket I captured the following during test.

```
ubuntu@cloudimg:~$ sudo kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_COREDIR: /var/crash
crashkernel addr: 0xde000000
   /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-6.5.0-5-generic
kdump initrd:
   /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-6.5.0-5-generic
current state: Not ready to kdump

kexec command:
  no kexec command recorded
ubuntu@cloudimg:~$ uname --all
Linux cloudimg 6.5.0-5-generic #5-Ubuntu SMP PREEMPT_DYNAMIC Wed Sep 6 15:36:23 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
ubuntu@cloudimg:~$ sudo cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-6.5.0-5-generic root=UUID=e7604ab2-200c-4f34-ab11-47e78ac4b8bd ro console=tty1 console=ttyS0 crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M
ubuntu@cloudimg:~$ sudo dmesg | grep -i crash
[ 0.000000] crashkernel reserved: 0x00000000de000000 - 0x00000000fe000000 (512 MB)
[ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-6.5.0-5-generic root=UUID=e7604ab2-200c-4f34-ab11-47e78ac4b8bd ro console=tty1 console=ttyS0 crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M
[ 87.054802] pstore: Using crash dump compression: deflate
ubuntu@cloudimg:~$ sudo cat /proc/sys/kernel/sysrq
176
ubuntu@cloudimg:~$ sudo service kdump-tools status
● kdump-tools.service - Kernel crash dump capture service
     Loaded: loaded (/lib/systemd/system/kdump-tools.service; enabled; preset: enabled)
     Active: active (exited) since Tue 2023-09-26 13:44:22 UTC; 7min ago
    Process: 514 ExecStart=/etc/init.d/kdump-tools start (code=exited, status=0/SUCCESS)
   Main PID: 514 (code=exited, status=0/SUCCESS)
        CPU: 4min 46.461s

Sep 26 13:38:21 cloudimg systemd[1]: Starting kdump-tools.service - Kernel crash dump capture service...
Sep 26 13:38:32 cloudimg kdump-tools[514]: Starting kdump-tools:
Sep 26 13:38:32 cloudimg kdump-tools[535]: * Creating symlink /var/lib/kdump/vmlinuz
Sep 26 13:38:41 cloudimg kdump-tools[577]: kdump-tools: Generating /var/lib/kdump/initrd.img-6.5.0-5-generic
Sep 26 13:44:11 cloudimg kdump-tools[535]: * Creating symlink /var/lib/kdump/initrd.img
Sep 26 13:44:22 cloudimg kdump-tools[535]: * failed to load kdump kernel
Sep 26 13:44:22 cloudimg kdump-tools[5536]: failed to load kdump kernel
Sep 26 13:44:22 cloudimg systemd[1]: Finished kdump-tools.service - Kernel crash dump capture service.
ubuntu@cloudimg:~$ sudo ls -al /var/lib/kdump/vmlinuz
lrwxrwxrwx 1 root root 29 Sep 26 13:38 /var/lib/kdump/vmlinuz -> /boot/vmlinuz-6.5.0-5-generic
ubuntu@cloudimg:~$ sudo file /boot/vmlinuz-6.5.0-5-generic
/boot/vmlinuz-6.5.0-5-generic: gzip compressed data, was "vmlinuz-6.5.0-5-generic.efi.signed", last modified: Thu Sep 7 10:43:26 2023, max compression, from Unix, original size modulo 2^32 54989192
```

.. but the `Cannot determine the file type of /var/lib/kdump/vmlinuz` in the `sudo service kdump-tools status` as @xnox has also reproduced with `kexec` is new.

I have also confirm...

Read more...

Dimitri John Ledkov (xnox)
Changed in linux (Ubuntu):
milestone: none → ubuntu-23.10
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

-6 kernel is respun, migration of -7 or higher is good

tags: removed: block-proposed-mantic
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 6.5.0-7.7

---------------
linux (6.5.0-7.7) mantic; urgency=medium

  * mantic/linux: 6.5.0-7.7 -proposed tracker (LP: #2037611)

  * kexec enable to load/kdump zstd compressed zimg (LP: #2037398)
    - [Packaging] Revert arm64 image format to Image.gz

  * Mantic minimized/minimal cloud images do not receive IP address during
    provisioning (LP: #2036968)
    - [Config] Enable virtio-net as built-in to avoid race

  * Miscellaneous Ubuntu changes
    - SAUCE: Add mdev_set_iommu_device() kABI
    - [Config] update gcc version in annotations

 -- Andrea Righi <email address hidden> Thu, 28 Sep 2023 10:19:24 +0200

Changed in linux (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-oem-6.5/6.5.0-1005.5 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-oem-6.5' to 'verification-done-jammy-linux-oem-6.5'. If the problem still exists, change the tag 'verification-needed-jammy-linux-oem-6.5' to 'verification-failed-jammy-linux-oem-6.5'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-jammy-linux-oem-6.5-v2 verification-needed-jammy-linux-oem-6.5
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-azure-6.5/6.5.0-1007.7~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-azure-6.5' to 'verification-done-jammy-linux-azure-6.5'. If the problem still exists, change the tag 'verification-needed-jammy-linux-azure-6.5' to 'verification-failed-jammy-linux-azure-6.5'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-jammy-linux-azure-6.5-v2 verification-needed-jammy-linux-azure-6.5
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-aws-6.5/6.5.0-1008.8~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-aws-6.5' to 'verification-done-jammy-linux-aws-6.5'. If the problem still exists, change the tag 'verification-needed-jammy-linux-aws-6.5' to 'verification-failed-jammy-linux-aws-6.5'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-jammy-linux-aws-6.5-v2 verification-needed-jammy-linux-aws-6.5
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.