Merge crash 8.0.4-1 into Noble

Work in progress.

Merge is in git branch
https://code.launchpad.net/~mfo/ubuntu/+source/crash/+git/crash/+ref/merge-lp2047861-noble

Building/testing from PPA
https://launchpad.net/~mfo/+archive/ubuntu/lp2047861-noble-crash

Still in progress.

Autopkgtests PASS from PPA:

$ DIR='https://autopkgtest.ubuntu.com/results/autopkgtest-noble-mfo-lp2047861-noble-crash'

$ curl -sf "$DIR/?format=plain" | grep result.tar | while read TAR; do echo "ARCH: $(echo $TAR | cut -d/ -f2)"; curl -sf "$DIR/$TAR" | tar xO summary; done
ARCH: amd64
live PASS
ARCH: amd64
live PASS
ARCH: arm64
live PASS
ARCH: armhf
live SKIP Test requires machine-level isolation but testbed does not provide that
ARCH: ppc64el
live PASS
ARCH: s390x
live PASS

Autopkgtests above are in live system form.
Tested in dump file form in amd64; working.

 # crash ./usr/lib/debug/boot/vmlinux-*-generic /var/crash/*/dump.*
 ...
       KERNEL: ./usr/lib/debug/boot/vmlinux-6.6.0-14-generic
     DUMPFILE: /var/crash/202401081708/dump.202401081708 [PARTIAL DUMP]
  CPUS: 2
  DATE: Mon Jan 8 17:08:44 UTC 2024
       UPTIME: 00:01:58
 LOAD AVERAGE: 0.18, 0.15, 0.06
        TASKS: 168
     NODENAME: mfo-noble-vm
      RELEASE: 6.6.0-14-generic
      VERSION: #14-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 30 10:27:29 UTC 2023
      MACHINE: x86_64 (1997 Mhz)
       MEMORY: 2 GB
        PANIC: "Kernel panic - not syncing: sysrq triggered crash"
   PID: 4587
      COMMAND: "bash"
  TASK: ffff888005c60000 [THREAD_INFO: ffff888005c60000]
   CPU: 1
        STATE: TASK_RUNNING (PANIC)

 crash> bt
 PID: 4587 TASK: ffff888005c60000 CPU: 1 COMMAND: "bash"
  #0 [ffffc90000fb7ba0] machine_kexec at ffffffff862b05eb
  #1 [ffffc90000fb7c00] __crash_kexec at ffffffff8641abe3
  #2 [ffffc90000fb7cc8] panic at ffffffff862f8be4
  #3 [ffffc90000fb7d48] sysrq_handle_crash at ffffffff86ca64ca
  #4 [ffffc90000fb7d58] __handle_sysrq at ffffffff86ca6ca3
  #5 [ffffc90000fb7da0] write_sysrq_trigger at ffffffff86ca74e8
  #6 [ffffc90000fb7db8] proc_reg_write at ffffffff8678060c
  #7 [ffffc90000fb7dd8] vfs_write at ffffffff866c4472
  #8 [ffffc90000fb7e78] ksys_write at ffffffff866c4b73
  #9 [ffffc90000fb7eb8] __x64_sys_write at ffffffff866c4c29
 #10 [ffffc90000fb7ec8] do_syscall_64 at ffffffff8734a64c
 #11 [ffffc90000fb7f50] entry_SYSCALL_64_after_hwframe at ffffffff874000e6
     RIP: 0000747d04f1b214 RSP: 00007ffd128231c8 RFLAGS: 00000202
     RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 0000747d04f1b214
     RDX: 0000000000000002 RSI: 000062040a0a75a0 RDI: 0000000000000001
     RBP: 000062040a0a75a0 R8: 0000000000000073 R9: 0000000000000000
     R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000002
     R13: 0000747d04fff5c0 R14: 0000747d04ffcf20 R15: 0000000000000000
     ORIG_RAX: 0000000000000001 CS: 0033 SS: 002b
 crash> quit

Tested in dump file form in arm64; working.

arm64
-----

vm

 SERIES=noble
 ARCH=arm64
 DISK=${SERIES}_${ARCH}.qcow2

 wget https://cloud-images.ubuntu.com/$SERIES/current/${SERIES}-server-cloudimg-${ARCH}.img
 qemu-img create -F qcow2 -b ${SERIES}-server-cloudimg-${ARCH}.img -f qcow2 $DISK 8G

 sudo apt install -y qemu-system-arm qemu-efi

 dd if=/dev/zero of=flash1.img bs=1M count=64
 dd if=/dev/zero of=flash0.img bs=1M count=64
 dd if=/usr/share/qemu-efi/QEMU_EFI.fd of=flash0.img conv=notrunc

 qemu-system-aarch64 \
 -machine virt -cpu cortex-a57 \
 -pflash flash0.img -pflash flash1.img \
 \
 -smp cpus=2 -m 2048 \
 -nodefaults -no-user-config \
 -nographic -serial stdio \
 \
 -drive file=$DISK,if=none,id=drive0 \
 -device virtio-blk-device,drive=drive0 \
 \
 -drive file=test-cidata.iso,media=cdrom \
 \
 -netdev user,hostfwd=::22222-:22,id=net0 \
 -device virtio-net-device,netdev=net0

 $ ssh ubuntu@127.0.0.1 -p 22222

 $ lsb_release -cs
 No LSB modules are available.
 noble

 $ uname -m
 aarch64

 $ uname -rv
 6.6.0-14-generic #14-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 30 10:53:08 UTC 2023

kdump-tools

 sudo apt update && sudo apt install -y linux-crashdump # No, Yes
 sudo sed 's/crashkernel=[^" ]\+/crashkernel=512M/' -i /etc/default/grub.d/kdump-tools.cfg
 sudo update-grub && sudo reboot

 $ sudo dmesg | grep 'crashkernel reserved:'
 [ 0.000000] crashkernel reserved: 0x0000000089800000 - 0x00000000a9800000 (512 MB)

 $ sudo kdump-config status
 current state : ready to kdump

crashdump

 $ echo c | sudo tee /proc/sysrq-trigger

 $ find /var/crash
 /var/crash
 /var/crash/kexec_cmd
 /var/crash/202401082101
 /var/crash/202401082101/dmesg.202401082101
 /var/crash/202401082101/dump.202401082101
 /var/crash/kdump_lock
 /var/crash/linux-image-6.6.0-14-generic-202401082101.crash

debug symbols

 dpkg -l | awk '$2 ~ /linux-image-[0-9.-]+-generic/ { print $2, $3}' \
   | while read pkg version; do \
       dbgpkg="linux-image-unsigned-${pkg#linux-image-}-dbgsym"; \
       arch=$(dpkg --print-architecture); \
       wget "https://launchpad.net/ubuntu/+archive/primary/+files/${dbgpkg}_${version}_${arch}.ddeb"; \
     done

 ar p linux-image-unsigned-*-dbgsym_*.ddeb data.tar.xz | tar xJ --wildcards './usr/lib/debug/boot/vmlinux-*-generic'

crash

 sudo add-apt-repository -y ppa:mfo/lp2047861-noble-crash && sudo apt install -y crash

 $ dpkg -s crash | grep Version:
 Version: 8.0.4-1ubuntu1

 $ crash ./usr/lib/debug/boot/vmlinux-*-generic /var/crash/*/dump.*
 ...
       KERNEL: ./usr/lib/debug/boot/vmlinux-6.6.0-14-generic
     DUMPFILE: /var/crash/202401082101/dump.202401082101 [PARTIAL DUMP]
  CPUS: 2
  DATE: Mon Jan 8 20:58:35 UTC 2024
       UPTIME: 00:24:51
 LOAD AVERAGE: 0.08, 0.15, 0.67
        TASKS: 127
     NODENAME: test
      RELEASE: 6.6.0-14-generic
      VERSION: #14-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 30 10:53:08 UTC 2023
      MACHINE: aarch64 (unknown Mhz)
       MEMORY: 2 GB
        PANIC: "Kernel panic - not syncing: sysrq triggered crash"
   PID: 4928
      COMMAND: "tee"
  TASK: ffff000000d13300 [THREAD_INFO: ffff000000d13300]
   CPU: 0
        STATE: TASK_RUNNING (PANIC)

 crash> bt
 PID: 4928 TASK: ffff000000d13300 CP...

Tested in dump file form in ppc64el; working.

ppc64el
-------

vm

 SERIES=noble
 ARCH=ppc64el
 DISK=${SERIES}_${ARCH}.qcow2

 wget https://cloud-images.ubuntu.com/$SERIES/current/${SERIES}-server-cloudimg-${ARCH}.img
 qemu-img create -F qcow2 -b ${SERIES}-server-cloudimg-${ARCH}.img -f qcow2 $DISK 8G

 sudo apt install -y qemu-system-ppc

 qemu-system-ppc64 \
   -machine pseries -cpu POWER10 \
 \
 -smp cpus=2 -m 2048 \
 -nodefaults -no-user-config \
 -nographic -serial stdio \
 \
 -drive file=$DISK,if=none,id=drive0 \
 -device virtio-blk,drive=drive0 \
 \
 -drive file=test-cidata.iso,media=cdrom \
 \
 -netdev user,hostfwd=::22223-:22,id=net0 \
 -device virtio-net,netdev=net0

 $ ssh ubuntu@127.0.0.1 -p 22223

 $ lsb_release -cs
 No LSB modules are available.
 noble

 $ uname -m
 ppc64le

 $ uname -rv
 6.5.0-9-generic #9-Ubuntu SMP Fri Oct 6 20:21:54 UTC 2023

kdump-tools

 sudo apt update && sudo apt install -y linux-crashdump # No, Yes
 sudo sed 's/crashkernel=[^" ]\+/crashkernel=512M/' -i /etc/default/grub.d/kdump-tools.cfg
 sudo update-grub && sudo reboot

 ...

 $ sudo dmesg | grep 'crashkernel.*RAM'
 [ 0.000000] Reserving 512MB of memory at 512MB for crashkernel (System RAM: 2048MB)

 $ sudo kdump-config status
 current state : ready to kdump

crashdump

 $ echo c | sudo tee /proc/sysrq-trigger

 $ find /var/crash
 /var/crash
 /var/crash/202401082101
 /var/crash/202401082101/dmesg.202401082101
 /var/crash/202401082101/dump.202401082101
 /var/crash/kexec_cmd
 /var/crash/kdump_lock

debug symbols

 dpkg -l | awk '$2 ~ /linux-image-[0-9.-]+-generic/ { print $2, $3}' \
   | while read pkg version; do \
       dbgpkg="${pkg}-dbgsym"; \
       arch=$(dpkg --print-architecture); \
       wget "https://launchpad.net/ubuntu/+archive/primary/+files/${dbgpkg}_${version}_${arch}.ddeb"; \
     done

 ar p linux-image-*-dbgsym_*.ddeb data.tar.xz | tar xJ --wildcards './usr/lib/debug/boot/vmlinux-*-generic'

crash

 sudo add-apt-repository -y ppa:mfo/lp2047861-noble-crash && sudo apt install -y crash

 $ dpkg -s crash | grep Version:
 Version: 8.0.4-1ubuntu1

 $ crash ./usr/lib/debug/boot/vmlinux-*-generic /var/crash/*/dump.*
 ...
       KERNEL: ./usr/lib/debug/boot/vmlinux-6.6.0-14-generic
     DUMPFILE: /var/crash/202401082101/dump.202401082101 [PARTIAL DUMP]
  CPUS: 2
  DATE: Mon Jan 8 21:00:05 UTC 2024
       UPTIME: 00:20:30
 LOAD AVERAGE: 0.12, 0.15, 0.39
        TASKS: 140
     NODENAME: test
      RELEASE: 6.6.0-14-generic
      VERSION: #14-Ubuntu SMP Thu Nov 30 10:29:25 UTC 2023
      MACHINE: ppc64le (1000 Mhz)
       MEMORY: 2 GB
        PANIC: "Kernel panic - not syncing: sysrq triggered crash"
   PID: 1174
      COMMAND: "tee"
  TASK: c000000047f89900 [THREAD_INFO: c000000047f89900]
   CPU: 1
        STATE: TASK_RUNNING (PANIC)

 crash> bt
 PID: 1174 TASK: c000000047f89900 CPU: 1 COMMAND: "tee"
  R0: c000000000305af8 R1: c000000047df7890 R2: c000000002162b00
  R3: c000000047df78a8 R4: 0000000000000000 R5: 0000000000000000
  R6: c000000047df7a58 R7: 0000000000000000 R8: 0000000000000001
  R9: c00000000ec87000 R10: 0000000000000000 R11: 0000000000000000
  R12: 0000000000000000 R13: ...

Tested in dump file form in s390x; working.

s390x
-----

vm

 SERIES=noble
 ARCH=s390x
 DISK=${SERIES}_${ARCH}.qcow2

 wget https://cloud-images.ubuntu.com/$SERIES/current/${SERIES}-server-cloudimg-${ARCH}.img
 qemu-img create -F qcow2 -b ${SERIES}-server-cloudimg-${ARCH}.img -f qcow2 $DISK 8G

 sudo apt install -y qemu-system-s390x

 qemu-system-s390x \
   -machine s390-ccw-virtio -cpu qemu \
 \
 -smp cpus=2 -m 2048 \
 -nodefaults -no-user-config \
 -nographic -serial stdio \
 \
 -drive file=$DISK,if=none,id=drive0 \
 -device virtio-blk,drive=drive0 \
 \
 -drive file=test-cidata.iso,media=cdrom \
 \
 -netdev user,hostfwd=::22224-:22,id=net0 \
 -device virtio-net,netdev=net0

 $ ssh ubuntu@127.0.0.1 -p 22224

 $ lsb_release -cs
 No LSB modules are available.
 noble

 $ uname -m
 s390x

 $ uname -rv
 6.6.0-14-generic #14-Ubuntu SMP Thu Nov 30 09:46:34 UTC 2023

kdump-tools

 sudo apt update && sudo apt install -y linux-crashdump # No, Yes
 sudo sed '/^parameters =/ s/$/ crashkernel=512M/' -i /etc/zipl.conf
 sudo zipl && sudo reboot

 $ sudo dmesg | grep 'crashkernel.*RAM'
 [ 1.176382] setup: Reserving 512MB of memory at 1509MB for crashkernel (System RAM: 1536MB)

 $ sudo kdump-config status
 current state : ready to kdump

crashdump

 $ echo c | sudo tee /proc/sysrq-trigger

 $ find /var/crash
 /var/crash
 /var/crash/kexec_cmd
 /var/crash/kdump_lock
 /var/crash/linux-image-6.6.0-14-generic-202401091639.crash
 /var/crash/202401091639
 /var/crash/202401091639/dmesg.202401091639
 /var/crash/202401091639/dump.202401091639

debug symbols

 dpkg -l | awk '$2 ~ /linux-image-[0-9.-]+-generic/ { print $2, $3}' \
   | while read pkg version; do \
       dbgpkg="linux-image-${pkg#linux-image-}-dbgsym"; \
       arch=$(dpkg --print-architecture); \
       wget "https://launchpad.net/ubuntu/+archive/primary/+files/${dbgpkg}_${version}_${arch}.ddeb"; \
     done

 ar p linux-image-*-dbgsym_*.ddeb data.tar.xz | tar xJ --wildcards './usr/lib/debug/boot/vmlinux-*-generic'

crash

 sudo add-apt-repository -y ppa:mfo/lp2047861-noble-crash && sudo apt install -y crash

 # dpkg -s crash | grep Version:
 Version: 8.0.4-1ubuntu1

 # crash ./usr/lib/debug/boot/vmlinux-*-generic /var/crash/*/dump.*
 ...

       KERNEL: ./usr/lib/debug/boot/vmlinux-6.6.0-14-generic
     DUMPFILE: /var/crash/202401091639/dump.202401091639 [PARTIAL DUMP]
  CPUS: 2
  DATE: Tue Jan 9 16:36:47 UTC 2024
       UPTIME: 00:06:08
 LOAD AVERAGE: 2.56, 2.20, 1.11
        TASKS: 148
     NODENAME: test
      RELEASE: 6.6.0-14-generic
      VERSION: #14-Ubuntu SMP Thu Nov 30 09:46:34 UTC 2023
      MACHINE: s390x (unknown Mhz)
       MEMORY: 2 GB
        PANIC: "Kernel panic - not syncing: sysrq triggered crash"
   PID: 1615
      COMMAND: "tee"
  TASK: 4bd2400 [THREAD_INFO: 4bd2400]
   CPU: 0
        STATE: TASK_RUNNING (PANIC)

 crash> bt
 PID: 1615 TASK: 4bd2400 CPU: 0 COMMAND: "tee"
  LOWCORE INFO:
   -psw : 0x0400c00180000000 0x0000000000126220
   -function : __do_machine_kdump at 126220
   -prefix : 0x026e8000
   -cpu timer: 0x7ffffea07b16f64d
   -clock cmp: 0x796343ea7b0db800
   -general registers:
      0x00000000026bd400 0x00000000026bd400
      0x0000000...

Autopkgtests (live system form) PASS in amd64, arm64, ppc64el, s390x (armhf skipped) [comment #3]
Manual tests (dump file form) PASS in amd64, arm64, ppc64el, s390x [comments #4, #5, #6, #7]

This bug was fixed in the package crash - 8.0.4-1ubuntu1

---------------
crash (8.0.4-1ubuntu1) noble; urgency=medium

  * Merge with Debian unstable. Remaining changes: (LP: #2047861)
    - d/t/live: If the "live" autopkgtest fails with a recommendation to
      try /proc/kcore instead of the default, attempt that before
      failing the test. LP 1858958.
    - d/t/live: Fix logic issue in "live" autopkgtest introduced in the
      last upload.
    - d/t/live: Fix test, as if will return 0 when no cases were true.
    - d/t/control: Add linux-libc-dev dependency for the autopkg test.
      This package gets usually broken with kernel upgrades, so let it
      already show in the autopkg test.
    - d/t/control: Run autopkg test with allow-stderr.
  * Drop changes:
    - d/p/0002-Fix-compilation-error-due-to-new-strlcpy-function.patch
      [ included upstream in 8.0.4 ]
    - d/p/0003-Support-module-memory-layout-change-on-Linux-6.4.patch
      [ included upstream in 8.0.4 ]

 -- Mauricio Faria de Oliveira <email address hidden> Tue, 02 Jan 2024 19:09:28 -0300