Userspace break as a result of missing patch backport

Status changed to 'Confirmed' because the bug affects multiple users.

See https://marc.info/?t=154883032500001&r=1&w=2 for LKML discussion and replies.

We will pick the follow-up patch up for the next cycle.

This bug was fixed in the package linux - 4.18.0-15.16

---------------
linux (4.18.0-15.16) cosmic; urgency=medium

  * Ubuntu boot failure. 4.18.0-14 boot stalls. (does not boot) (LP: #1814555)
    - Revert "drm/i915/ringbuffer: Delay after EMIT_INVALIDATE for gen4/gen5"

  * Userspace break as a result of missing patch backport (LP: #1813873)
    - tty: Don't hold ldisc lock in tty_reopen() if ldisc present

 -- Stefan Bader <email address hidden> Thu, 07 Feb 2019 11:24:34 +0100

Hi,

Still present in latest i386 Bionic 18.04.02 LTS

Hi Joe, can you please paste the output of "uname -a" from your system that is presenting this issue?

Thanks in advance

No package containing the fix for 18.04 has been released yet, only 18.10.
It is therefore expected that 18.04.X still displays the issue.

I am running 18.04 with linux-image-generic-hwe-18.04 installed. I can confirm that the issue has been resolved with the latest kernel which is 4.18.0-15-generic.

Great, thanks Rehan!

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

I tried xenial-proposed, but the fix doesn't seem to be available yet. It's still pulling 4.15.0 from xenial-updates (and that one still has the problem).

I double checked that xenial-proposed is properly configured by installing 'linux-generic':
...
Get:24 http://be.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 linux-generic amd64 4.4.0.143.150 [1,786 B]
Get:25 http://be.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 linux-image-generic amd64 4.4.0.143.150 [2,478 B]
...

W.

Putatieve,

Seems like you have the bionic HWE installed, so that is why your server is defaulting to 4.15.
Your apt log shows 4.4 downloading so you should be able to manually boot into it from grub...

Personally I'm hoping an update drops to bionic proposed as Xenial was not part of my original testing....

I most definitly do have linux-generic-hwe-16.04 installed. I installed linux-generic to test if xenial-proposed was properly configured, as 'apt-get install -t xenial-proposed linux-genric-hwe-16.04' didn't install anything.

PS: linux-genric-hwe-16.04 is a typo in this post

And I am running ubuntu 16.04

Right, the fixed kernel mentioned in -proposed is NOT the 4.15 HWE kernel, it's the 4.4.0 kernel.

You need to select the correct kernel if you wish to test the fix.

You don't seem to understand what I said: I DO HAVE 4.15.0 installed (linux-generic-hwe-16.04):
https://packages.ubuntu.com/search?keywords=linux-generic-hwe-16.04&searchon=names&suite=xenial-updates&section=all

running 'apt-get install -t xenial-proposed linux-generic-hwe-16.04' does NOT pull any new/updated 4.15.0 kernel from xenial-proposed.

To double check that I configured xenial-proposed correctly, I installed linux-generic, wich DOES have an update waiting (4.4.0-143 in xenial-proposed over 4.4.0-142 from xenial/main):

I'm under unbuntu 16.04 kernel 4.4.0 and the xenial-proposed solved problem reading /dev/tty for me

Of course, the bug/regression only appeared in 4.15.0-45 (4.15.0-43 is fine).

In post #11, Brad Figg specifically asked us to test a fix, released in the 'proposed' repository, ASAP and report back with the result. Otherwise the proposed fix would be dropped.

But on Ubuntu 16.04 (xenial-proposed) the fix is not available (yet?) or am I missing something?

Hi Putatieve, thanks for you attention here. Let me try to clarify what happens here: the issue is present in kernel 4.4.0-142, 4.15.0-45 and 4.18.0-14. The three versions were already released,
but the fix for the regression observed in this LP is present in the new versions that are getting released in this cycle.

For Ubuntu 16.04/Xenial (and Trusty-HWE), the candidate version is: 4.4.0-143. This version is already available through the -proposed pocket, nubu validated that it fixes the issue (thanks nubu!).

For Ubuntu 18.04/Bionic (and Xenial-HWE), the candidate version for kernel 4.15 wasn't released yet, so we can't validate yet. This was the reason you couldn't test - your Ubuntu is configured to use 4.15 series, the called HWE kernel. Given the proposed version of 4.15 series for this cycle wasn't released yet, you couldn't test. You *can* install 4.4.0-143 from -proposed if you want, but you'll need to manually set grub to boot that instead of 4.15 (as per md_5 comment).

Finally, for Ubuntu 18.10/Cosmic, the fix was released in -updates already, in kernel 4.18.0-15, so the -proposed test isn't even necessary, the fix was released normally by now.

Please, let me know if you have questions on Ubuntu kernel cycles in order we can clarify it for you.
Cheers,

Guilherme

Hi Guilherme,

Thanks for your response. I was under the impression that there would also be fix in 'proposed' for Xenial-HWE.

I can confirm that the issue is fixed in 4.4.0-143 (and indeed present in 4.4.0-142).

Best,

W;-)

Hi all.

I'm following your discussion, but I couldn't realize if I can fix or just waiting the fix.

In my case I have two stations with ubuntu 18.04(bionic), both with linux-image-4.15.0-45-generic and linux-image-4.15.0-43-generic.

So, I wait or can I apply somehow the fix it?

All this is very new to me, sorry

Regards,

Samuel

You're welcome Putatieve, thanks for testing the Xenial 4.4.0-143 version!

And you're right, we will have a fix for Xenial-HWE - as soon it is available, you will observe a new message from Brad Figg here, this time adding a tag "verification-needed-bionic".
And then, likely the version for Xenial-HWE will be ready to test.

Notice the "likely" word above: happens that the build process is usually parallel, with Bionic 4.15 and Xenial 4.15~16.04 at same time; but there are small code changes (mostly toolchain related) between Bionic 4.15 and Xenial-HWE, so it may happen that Bionic version is ready before Xenial-HWE. It's likely to be at same time (or with some hours of difference) though.

If you're interested in more knowledge about Ubuntu kernel cycles, I'd like to suggest you this interesting wiki page: https://wiki.ubuntu.com/Kernel/FAQ#Kernel.2FSupport.Ubuntu_Kernel_Release_Schedule
You can check the releases and also, how the HWE works (18.04 kernel 4.15 is present as a long-term supported HWE kernel for 16.04.5 version; same relation we observe between 16.04 and 14.04.5).

Feel free to expose any remaining questions you have - also, your testing when the 4.15 version gets released to -proposed is much appreciated!
Cheers,

Guilherme

On Thu 2019-02-14 (10:46), SAMUEL TEIXEIRA SANTOS wrote:

> In my case I have two stations with ubuntu 18.04(bionic), both with
> linux-image-4.15.0-45-generic and linux-image-4.15.0-43-generic.
>
> So, I wait or can I apply somehow the fix it?

Install linux-image-generic-hwe-18.04

This package has the bug-fix,

--
Ullrich Horlacher Server und Virtualisierung
Rechenzentrum TIK
Universitaet Stuttgart E-Mail: <email address hidden>
Allmandring 30a Tel: ++49-711-68565868
70569 Stuttgart (Germany) WWW: http://www.tik.uni-stuttgart.de/
REF:<email address hidden>

Samuel, for now the suggestion is to keep kernel 4.15.0-43, as it doesn't show the issue reported in this LP.

Soon (probably later today or tomorrow) the new version for 4.15 will show up in -proposed pocket, so you can update to that version for testing purposes, since that version will have the fix for the issue reported here.

Cheers,

Guilherme

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

I enabled proposed, did an apt-get update, it got the index file:

Get:9 http://archive.ubuntu.com/ubuntu bionic-proposed InRelease [242 kB]

Then I did an apt-get dist-upgrade and no kernel showed up in the list of pacages to upgrade. What do I need to do to test the new kernel.

Error seems to be fixed. I have made the following solution:

# apt install --install-recommends linux-generic-hwe-18.04 xserver-xorg-hwe-18.04

Login as user/root in the text console (tty) with kernel 4.18.0-15 was possible without any problems.

My system: KDE Neon Useredition

The kernel in -proposed solves the problem, I've tested it !

I don't know how to add the flag "verification-done-bionic" to this bug though.

I think I may have run into another issue related to the same bug. While the login loop problem appears to have been fixed, the fix does not seem to have cascaded down to other prompts that one may encounter in special circumstances. In my case, I had an external USB drive connected to my server and this happened to have been powered off (physically) when booting. As expected, the system did not complete the boot process and stopped (momentarily) at the "Enter Ctrl-D etc" prompt to allow repair. Well, it just kept going. I was unable to get in at all and eventually just had to physically reset the box and reconnect power to the USB drive. So the looping that used to happen at the console login persists when dropping to the emergency shell. I will be grateful if anyone else can confirm. Just add a spurious entry to fstab and try to reboot.

Hello, I mistakenly change the status of Bionic to Fix released, I didn't know that I could change something like this since I'm not a maitainer of some sort, I'm just an end user. I tried to cancel the change but I don't have the right to do so.

Hi Edgar, I've changed it back =)
It should be Fix Committed if the patch is present/merged in the kernel, but kernel wasn't released yet. Once it gets released, it'll get changed to Fix Released.

Cheers,

Guilherme

Confirming the login loop bug for 18.04.2 with 4.15.0-45-generic
And also confirming the fix in proposed with 4.15.0-46-generic

Confirming the login loop bug for 16.04.5 LTS with 4.4.0-142-generic

Also seen on AskUbuntu - https://askubuntu.com/q/1113704/66509 - with 16.04 LTS and 18.04 LTS.

Are we going to see a release on this soon?
Nearly a month for a userspace impacting regression :(

I tested 4.15.0-46 from bionic-proposed. I can confirm that it is working for Ubuntu 18.04.

I tested the kernel 4.15.0-46 from bionic/proposed. I confirm that it fix the problem for Ubuntu 18.04

I tested the kernel 4.4.0-143 from xenial-proposed on 16.04 and from trusty-proposed on 14.04 and it fix the problem for these 2 releases.

Thanks all for the tests.

@md_5: According to https://kernel.ubuntu.com, the promotion of the -proposed packages to -updates is scheduled to March, 4th.

Cheers,

Guilherme

Considering that this is a regression, with major impact (TTY login no longer possible, often no graphical login either), many affected users on (amongst other) the latest LTS release, and both this fact and how to fix it has been known since at least January 30, I am puzzled as to why this is being handled as an update with the default stable release cadence.

Could you kindly explain the following (since I may just be misunderstanding how things are handled):
Was rolling back the breaking (since incomplete) patchset (which introduced this issue) not an option?
Has testing on 18.04 and 16.04 brought up new problems, which would have had even more serious impact?
How is "importance" calculated (personally I would have set it to "critical" or "major", if those are available)?

Thanks.

Thanks for the link Guilherme, guess that answers that question.
Given March 4 is just round the corner, I am not sure there is much can be done at this point, but I will however just echo what Tom beat me to saying:

As a software developer I fully understand and recognise the need to delay things for various reasons even if it may seem everything is primed and ready to go.

That being said I think the overall response time here has not been great at all.

A kernel regression which breaks user space programs (including the default TTY login), and has a small, tested patch available should absolutely not take 3-4 weeks to be released.

I understand that this new March 4 release will have additional unrelated patches that need that testing time, but I think breaking from the standard SRU cycle to either revert the problematic patches, or apply the fix would absolutely have been worth it - just look at the 13 duplicates of this bug and the posts on various other web forums / support sites.

Am I wrong or the fix hasn't yet been released?

I understood the release date was the 4th March.

Hey Andrea,

I have the same problem as you. There are still some regions of the earth where it is still March 4th. I hope the fix will be released soon. It is sad that of all things, the both LTS versions which are advertised to be used for better reliability and stability, seem to get fixed at last...

I'm located in solar > earth > europe but still waiting for the -update version of xenial too (@_@)

Update?

This bug was fixed in the package linux - 4.15.0-46.49

---------------
linux (4.15.0-46.49) bionic; urgency=medium

  * linux: 4.15.0-46.49 -proposed tracker (LP: #1814726)

  * mprotect fails on ext4 with dax (LP: #1799237)
    - x86/speculation/l1tf: Exempt zeroed PTEs from inversion

  * kernel BUG at /build/linux-vxxS7y/linux-4.15.0/mm/slub.c:296! (LP: #1812086)
    - iscsi target: fix session creation failure handling
    - scsi: iscsi: target: Set conn->sess to NULL when iscsi_login_set_conn_values
      fails
    - scsi: iscsi: target: Fix conn_ops double free

  * user_copy in user from ubuntu_kernel_selftests failed on KVM kernel
    (LP: #1812198)
    - selftests: user: return Kselftest Skip code for skipped tests
    - selftests: kselftest: change KSFT_SKIP=4 instead of KSFT_PASS
    - selftests: kselftest: Remove outdated comment

  * RTL8822BE WiFi Disabled in Kernel 4.18.0-12 (LP: #1806472)
    - SAUCE: staging: rtlwifi: allow RTLWIFI_DEBUG_ST to be disabled
    - [Config] CONFIG_RTLWIFI_DEBUG_ST=n
    - SAUCE: Add r8822be to signature inclusion list

  * kernel oops in bcache module (LP: #1793901)
    - SAUCE: bcache: never writeback a discard operation

  * CVE-2018-18397
    - userfaultfd: use ENOENT instead of EFAULT if the atomic copy user fails
    - userfaultfd: shmem: allocate anonymous memory for MAP_PRIVATE shmem
    - userfaultfd: shmem/hugetlbfs: only allow to register VM_MAYWRITE vmas
    - userfaultfd: shmem: add i_size checks
    - userfaultfd: shmem: UFFDIO_COPY: set the page dirty if VM_WRITE is not set

  * Ignore "incomplete report" from Elan touchpanels (LP: #1813733)
    - HID: i2c-hid: Ignore input report if there's no data present on Elan
      touchpanels

  * Vsock connect fails with ENODEV for large CID (LP: #1813934)
    - vhost/vsock: fix vhost vsock cid hashing inconsistent

  * SRU: Fix thinkpad 11e 3rd boot hang (LP: #1804604)
    - ACPI / LPSS: Force LPSS quirks on boot

  * Bionic update: upstream stable patchset 2019-01-17 (LP: #1812229)
    - scsi: sd_zbc: Fix variable type and bogus comment
    - KVM/Eventfd: Avoid crash when assign and deassign specific eventfd in
      parallel.
    - x86/apm: Don't access __preempt_count with zeroed fs
    - x86/events/intel/ds: Fix bts_interrupt_threshold alignment
    - x86/MCE: Remove min interval polling limitation
    - fat: fix memory allocation failure handling of match_strdup()
    - ALSA: hda/realtek - Add Panasonic CF-SZ6 headset jack quirk
    - ARCv2: [plat-hsdk]: Save accl reg pair by default
    - ARC: Fix CONFIG_SWAP
    - ARC: configs: Remove CONFIG_INITRAMFS_SOURCE from defconfigs
    - ARC: mm: allow mprotect to make stack mappings executable
    - mm: memcg: fix use after free in mem_cgroup_iter()
    - mm/huge_memory.c: fix data loss when splitting a file pmd
    - cpufreq: intel_pstate: Register when ACPI PCCH is present
    - vfio/pci: Fix potential Spectre v1
    - stop_machine: Disable preemption when waking two stopper threads
    - drm/i915: Fix hotplug irq ack on i965/g4x
    - drm/nouveau: Use drm_connector_list_iter_* for iterating connectors
    - drm/nouveau: Avoid looping through fake MST connectors
    - gen_stats: Fix netl...

still available only in Proposed though

I just made right now the update on my ubuntu 18.02 and got the new 4.15.0-46-generic kernel. The bug is fixed to me. Thanks all

My system updated today, and I can confirm that the issue for login to
the shell seems to have gone away. tnx.

4.15.0-46-lowlatency #49-Ubuntu SMP PREEMPT Wed Feb 6 10:23:17 UTC 2019
x86_64 x86_64 x86_64 GNU/Linux

On Tue, 2019-03-05 at 22:51 +0000, SAMUEL TEIXEIRA SANTOS wrote:
> I just made right now the update on my ubuntu 18.02 and got the new
> 4.15.0-46-generic kernel. The bug is fixed to me. Thanks all
>

In xenial with hwe stack I got 4.15.0-46-generic now also and the bug is gone.

Regarding Ubuntu 16.04's "xenial" GA kernel (4.4), I've been told that the Kernel team ran into some issues with these and are now planning to release them next week. As Brandon discussed, installing the 4.15 hardware enablement (HWE) kernel - as documented at https://wiki.ubuntu.com/Kernel/LTSEnablementStack - works around this bug, too.

I was forced to install the LTSEnablementStack because with the regular 4.4.0 kernel, suddenly with an update, a certain combination of docking/undocking my laptop led to a system freeze. Newer kernels (4.8 back then) did not have the problem, also I could name the 4.4.0 kernel image that also ran fine. Even though I put an impressive amount of working hours (40h+) into pinpointing the problem by doing kernel bisecting, this still not enough. At some time, I just installed the LTSEnablementStack to get regular updates of the 4.8 and I could get back to work.

Confirming fix, login working again on bionic. Thanks for your work.

fix confirmed, thanks

Yes, the fix seems to have saved us. Confirmed, with kernel 4.15.0-46-generic, an Ubuntu 18.04 update on 4th of March.

Will the fix for 16.04 32 bit be available through normal channels in the near future?
Thank you.

This bug was fixed in the package linux - 4.4.0-143.169

---------------
linux (4.4.0-143.169) xenial; urgency=medium

  * linux: 4.4.0-143.169 -proposed tracker (LP: #1814647)

  * x86/kvm: Backport fixup and missing commits (LP: #1811646)
    - KVM: x86: avoid vmalloc(0) in the KVM_SET_CPUID
    - kvm: nVMX: VMCLEAR an active shadow VMCS after last use
    - X86/nVMX: Properly set spec_ctrl and pred_cmd before merging MSRs
    - KVM/VMX: Optimize vmx_vcpu_run() and svm_vcpu_run() by marking the RDMSR
      path as unlikely()
    - kvm: x86: IA32_ARCH_CAPABILITIES is always supported
    - KVM: SVM: Add MSR-based feature support for serializing LFENCE
    - KVM: X86: Allow userspace to define the microcode version
    - KVM: x86: SVM: Call x86_spec_ctrl_set_guest/host() with interrupts disabled
    - KVM: VMX: fixes for vmentry_l1d_flush module parameter
    - kvm: svm: Ensure an IBPB on all affected CPUs when freeing a vmcb
    - kvm: vmx: Scrub hardware GPRs at VM-exit
    - SAUCE: [Fix] x86/KVM/VMX: Add L1D flush logic
    - SAUCE: KVM: Move code fragments, cleanup and re-indent

  * linux-buildinfo: pull out ABI information into its own package
    (LP: #1806380)
    - [Packaging] limit preparation to linux-libc-dev in headers
    - [Packaging] commonise debhelper invocation
    - [Packaging] ABI -- accumulate abi information at the end of the build
    - [Packaging] buildinfo -- add basic build information
    - [Packaging] buildinfo -- add firmware information to the flavour ABI
    - [Packaging] buildinfo -- add compiler information to the flavour ABI
    - [Packaging] buildinfo -- add buildinfo support to getabis
    - [Config] buildinfo -- add retpoline version markers
    - [Packaging] getabis -- handle all known package combinations
    - [Packaging] getabis -- support parsing a simple version

  * signing: only install a signed kernel (LP: #1764794)
    - [Packaging] update to Debian like control scripts
    - [Packaging] switch to triggers for postinst.d postrm.d handling
    - [Packaging] signing -- switch to raw-signing tarballs
    - [Packaging] signing -- switch to linux-image as signed when available
    - [Packaging] printenv -- add signing options
    - [Packaging] fix invocation of header postinst hooks
    - [Packaging] signing -- add support for signing Opal kernel binaries
    - [Debian] Use src_pkg_name when constructing udeb control files
    - [Debian] Dynamically determine linux udebs package name
    - [Packaging] handle both linux-lts* and linux-hwe* as backports
    - [Config] linux-source-* is in the primary linux namespace
    - [Packaging] lookup the upstream tag
    - [Packaging] zfs/spl -- enhance provides information
    - [Packaging] switch up to debhelper 9
    - [Packaging] autopkgtest -- disable d-i when dropping flavours
    - [debian] support for ship_extras_package=false
    - [Debian] do_common_tools should always be on
    - [debian] do not force do_tools_common
    - [Packaging] Add linux-tools-host package for VM host tools
    - [Packaging] signing should be conditional
    - [Packaging] skip cloud tools packaging when not building package
    - [Packaging] add acpidbg
    - [debian] prep linu...

I'm on Kernal version
more /proc/version
Linux version 4.18.0-14-generic (buildd@lcy01-amd64-026) (gcc version 7.3.0 (Ubu
ntu 7.3.0-16ubuntu3)) #15~18.04.1-Ubuntu SMP Mon Jan 14 11:09:26 UTC 2019

And this bug is still occurring.
At ctrl-alt-f1 console, type my username, password prompt appears, new line appears, login incorrect appears 4 times, console resets.

Chris (~iplayfast): You're not running the latest HWE kernel image:
  linux-image-generic-hwe-18.04 | 4.18.0.18.68 | bionic-updates

"cat /proc/version" would report:
Linux version 4.18.0-18-generic (buildd@lcy01-amd64-006) (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) #19~18.04.1-Ubuntu SMP Fri Apr 5 10:22:13 UTC 2019

If you would like assistance with this, please consider the commercial https://www.ubuntu.com/support/community-support and community https://www.ubuntu.com/support/community-support support offerings rather than the bug tracker. Thanks.

Hi Chris, as per comment #21, the issue exists in kernel 4.18.0-14, and was fixed in 4.18.0-15.
So, you need to update your kernel.

Cheers,

Guilherme