[UBUNTU 18.04] Ubuntu 18.04 kernel 4.15.0-194 crashes on IPL

Default Comment by Bridge

Status changed to 'Confirmed' because the bug affects multiple users.

Thanks for reporting this.
What is the version of the Ubuntu host where you want to install the 18.04 guest onto?
So that I can try to properly recreate this.

------- Comment From <email address hidden> 2022-10-26 11:12 EDT-------
I have the strong suggestion that this is the cause:

commit 6edb63a7b6cd57825e47cf6a8600b694a19f0d90
Author: Jason A. Donenfeld <email address hidden>
Date: Sat Jun 11 00:20:23 2022 +0200

s390/archrandom: simplify back to earlier design and initialize earlier
BugLink: https://bugs.launchpad.net/bugs/1989625
commit e4f74400308cb8abde5fdc9cad609c2aba32110c upstream.

...

because that's the only patch which has changes in the s390 start kernel/arch/random areas.

------- Comment From <email address hidden> 2022-10-26 11:15 EDT-------
Recreate is totally simple: just install a Ubuntu 18.04 - does not matter if it is on LPAR or zVM or as a KVM guest.
If my suggestion is right, this happens since kernel 4.15.0-194.

Trying to recreate, I was just able:
- to successfully complete an 18.04 installation on a 22.10 host (that what I just had at hand) and
- a successful 18.04 installation on a z/VM 6.4 host
I selected in both cases the strongly recommended option 'Install security updates automatically'.
(And don't be surprised that the installed system reports itself as 18.04.6, even if the 18.04.5 ISO image was used - there was another update needed for other architectures that led to a .6.)

Please can you share more details about the system you are using, especially the KVM host?

Btw. for the KVM guest install I used:

sudo qemu-img create -f qcow2 /var/lib/libvirt/images/bionic.qcow2 8G

sudo virt-install --name bionic --vcpus 2 --ram 1024 --disk path=/var/lib/libvirt/images/bionic.qcow2,size=5,bus=virtio,format=qcow2 --os-type linux --os-variant generic --network network=default,model=virtio --graphics none --console pty -c /var/lib/libvirt/images/ubuntu-18.04.5-server-s390x.iso

uname -a
Linux ubuntu 4.15.0-194-generic #205-Ubuntu SMP Fri Sep 16 19:53:54 UTC 2022 s390x s390x s390x GNU/Linux

Again, 18.04.5 installations for for me, I used this (latest) 18.04.5 ISO:
https://cdimage.ubuntu.com/releases/18.04/release/ubuntu-18.04.5-server-s390x.iso

If there is a reason to revert 's390/archrandom: simplify back to earlier design and initialize earlier' we could consider that, but works here (z13 with z/VM 6.4 and z13 with Ubuntu 22.10 KVM host).

Since the Ubuntu kernel team does for every kernel update and SRU regression testing, a non-bootable kernel would have been identified and the kernel would not have left -proposed, it would even not have landed in -proposed.
That makes me think that the issue is elsewhere - at the KVM host side?!

I may try on z15, too (at least a KVM guest install).

If there is a reason to revert 's390/archrandom: simplify back to earlier design and initialize earlier' we could consider that, but works here (z13 with z/VM 6.4 and z13 with Ubuntu 22.10 KVM ost).

------- Comment From <email address hidden> 2022-10-26 11:48 EDT-------
Digging into details gives:

The reason seems to be that the patch
moves the decission about if arch randomness
is available to setup.c function setup_randomness().
This code uses a static_key s390_arch_random_available.
But in the Canonical kernel the initialization function
for the jump labels (where the static keys are based on)
jump_label_init() is called in generic start_kernel()
wheres in the upstream kernel the init function is
called early in setup_arch().
So maybe another patch which is moving the jump_label_init()
from start_kernel() to setup_arch() is missing here.

I did NOT install any security fixes ... will try.

Ah - interesting.
I could recreate the situation on z15 / L1III:

$ virsh start bionic --console
Domain 'bionic' started
Connected to domain 'bionic'
Escape character is ^] (Ctrl + ])
ubuntu@z15test:~$

There is obviously a difference in the behavior of the z13 and z15.

I can create a test kernel with the mentioned patch reverted...

------- Comment From <email address hidden> 2022-10-26 12:09 EDT-------
Well, my fresh installed kvm guest from https://cdimage.ubuntu.com/releases/18.04/release/ubuntu-18.04.5-server-s390x.iso crashes even with security updates enabled.

Hi Harald, see my previous comment - I could recreate this on z15 now (but it's fine on z13).
So seems to be related to (z14) z15 and newer?!

------- Comment From <email address hidden> 2022-10-26 12:54 EDT-------
My configuration is OpenSUSE (KVM host) on a z14- we used the latest cloud build from: https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-s390x.img

A test kernel is being build at this PPA:
https://launchpad.net/~fheimes/+archive/ubuntu/test
(will be available in some minutes)

------- Comment From <email address hidden> 2022-10-26 14:29 EDT-------
(In reply to comment #21)
> Hi Harald, see my previous comment - I could recreate this on z15 now (but
> it's fine on z13).
> So seems to be related to (z14) z15 and newer?!

We do patch the kernel depending on hardware level and enable newer instructions. Given Haralds finding of using static key before it is initialized might indicate a problem with that.

Frank, can you build a kernel with
commit 95e61b1b5d6394b53d147c0fcbe2ae70fbe09446
Author: Vasily Gorbik <email address hidden>
AuthorDate: Thu Jun 18 17:17:19 2020 +0200
Commit: Heiko Carstens <email address hidden>
CommitDate: Mon Jun 29 16:28:39 2020 +0200

s390/setup: init jump labels before command line parsing

and check if this helps?

Another build is running right now at https://launchpad.net/~fheimes/+archive/ubuntu/test2
with 's390/archrandom: simplify back to earlier design and initialize earlier' still in and 's390/setup: init jump labels before command line parsing' on top.
This will take a while - will test it tomorrow.

Hi Christian, all,
both kernels are now ready:

1) with "s390/archrandom: simplify back to earlier design and initialize earlier" reverted:
f98553359746 (HEAD -> master-next) Revert "s390/archrandom: simplify back to earlier design and initialize earlier"
(https://launchpad.net/~fheimes/+archive/ubuntu/test/)
I was able to successfully test this kernel, a KVM guest on a z15 LPAR boots after having this modified kernel installed.

2) with "s390/archrandom: simplify back to earlier design and initialize earlier",
but "s390/setup: init jump labels before command line parsing" on top:
4ab78427995e (HEAD -> master-next) s390/setup: init jump labels before command line parsing"
(https://launchpad.net/~fheimes/+archive/ubuntu/test2/)
I was NOT able to successfuly test this, a KVM guest on a z15 LPAR was still NOT able to boot with this modified kernel.

Now, should we proceed with the revert (which btw. did not revert cleanly, I had to massage it a little bit), or do you want to fix upstream "s390/archrandom: simplify back to earlier design and initialize earlier"?

(Just fyi: the upcoming last day for commit for the next Ubuntu kernel SRU cycle is Nov, 2nd - so we should have decided for a fix by Nov 1st to get it in w/o missing one SRU cycle.)

I just tried 20.04 (focal legacy image), just to be sure, but it's not affected and worked fine.

------- Comment From <email address hidden> 2022-10-27 07:25 EDT-------
> Now, should we proceed with the revert (which btw. did not revert cleanly, I
> had to massage it a little bit), or do you want to fix upstream
> "s390/archrandom: simplify back to earlier design and initialize earlier"?

If Viktor can confirm that the revert works, I would go with a revert. Obviously the upstream fix is not enough and we might need more patches which are not known by now.

Separately recent rework in random causes hangs and inability to boot on
x86 virtio-rnd machines too.

I feel like I need to escalate this issue to upstream kernel,
together/separately from the virtio-rnd one.

It seems like rnd rewrites are not quite as robust as the old status quo.

------- Comment From <email address hidden> 2022-10-27 08:49 EDT-------
With Frank's revert [build https://launchpad.net/~fheimes/+archive/ubuntu/test/]
kernel boots up OK.

Kernel SRU request submitted for bionic:
https://lists.ubuntu.com/archives/kernel-team/2022-October/134331.html
changing status to 'In Progress'.

Ubuntu 18.04 kernel 4.15.0-196 (currently in -proposed) incl. this fix.
https://launchpad.net/ubuntu/+source/linux/4.15.0-196.207

This bug was fixed in the package linux - 4.15.0-196.207

---------------
linux (4.15.0-196.207) bionic; urgency=medium

  * bionic/linux: 4.15.0-196.207 -proposed tracker (LP: #1994992)

  * [UBUNTU 18.04] Ubuntu 18.04 kernel 4.15.0-194 crashes on IPL (LP: #1994601)
    - SAUCE: Revert "s390/archrandom: simplify back to earlier design and
      initialize earlier"

 -- Luke Nowakowski-Krijger <email address hidden> Thu, 27 Oct 2022 13:56:02 -0700

------- Comment From <email address hidden> 2022-11-10 11:26 EDT-------
I re-ran the installation as described above and now I was able to boot the VM. Thanks!

This bug is awaiting verification that the linux-azure-4.15/4.15.0-1162.177 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug doesn't affect the azure kernel or any other non-linux-generic kernels.
Hence updating the tags to unblock the process...