ibrs/ibpb fixes result in excessive kernel logging

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1755627

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

We do not use apport and this is easily reproducible. If there is any pointed information I can provide beyond what I've already submitted, please let me know.

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.16 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16-rc4

We use LTS kernels, so no - unfortunately we cannot.

Returning to confirmed status - easily reproducible with LTS kernels.

It's worth noting, too, that this does not happen on Bionic @ 4.15 (using Ubuntu 4.15.0-1001.1-aws 4.15.3) but it does happen on Trusty (using Ubuntu 3.13.0-143.192-generic 3.13.11-ckt39)

For the sake of completeness on this one, I installed the following mainline packages on an AWS m5.large instance running 16.04.3 w/ 4.4.0-112-generic

- linux-headers-4.16.0-041600rc4_4.16.0-041600rc4.201803041930_all.deb
- linux-headers-4.16.0-041600rc4-generic_4.16.0-041600rc4.201803041930_amd64.deb
- linux-image-4.16.0-041600rc4-generic_4.16.0-041600rc4.201803041930_amd64.deb

This does NOT trigger the bug, either via `sysctl -a` or `sysctl kernel.ibrs_dump` as `/proc/sys/kernel/ibrs_dump` does not exist. This is the same behavior on Bionic w/ 4.15.

Also, to call it out again, this appears to affect Trusty's LTS kernel, too (it's not clear how one marks a bug as such from the Launchpad UI)

Thank you Leann!

https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/tree/kernel/sysctl.c#n2433
and
https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/tree/kernel/sysctl.c#n2446

are the source(s) of the noise

Looking at nearby functions like proc_dointvec_ibrs_ctrl and proc_dointvec_ipbp_ctrl, I suspect the intention was to use pr_debug instead of printk...

Introduced in this commit:
https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/commit/kernel/sysctl.c?id=5e7fa0233bdfe2906216606d0d4d156be8f86950

I haven't done any digging on 3.13 but expect it will be similar.

@bmoyles- FYI, we're just removing the kernel.ibrs_dump interface altogether -- its only purpose is to provide that debug info, and nothing like it made it to mainline.

Thank you @kamalmostafa - we'll keep an eye out for the updated packages in the repositories and follow up if anything is not as expected. Thanks again for fixing this!

@lscotte - Thanks for reporting the issue. Launchpad will post a note to this bug as soon as the kernel versions containing the fix reach the -proposed archive.

Hello,

The same here:

Apr 18 06:29:16 ovh-staging-rancher-docker-10 kernel: [37874.285196] sysctl_ibrs_enabled = 0, sysctl_ibpb_enabled = 0
Apr 18 06:29:16 host kernel: [37874.285197] use_ibrs = 4, use_ibpb = 4
Apr 18 06:29:16 host kernel: [37874.285202] read cpu 0 ibrs val 0
Apr 18 06:29:16 host kernel: [37874.285203] read cpu 1 ibrs val 0
Apr 18 06:29:16 host kernel: [37874.285203] read cpu 2 ibrs val 0
Apr 18 06:29:16 host kernel: [37874.285204] read cpu 3 ibrs val 0

Linux DESKTOP-KCAOO2V 4.4.0-17133-Microsoft #1-Microsoft Fri Mar 23 13:12:00 PST 2018 x86_64 x86_64 x86_64 GNU/Linux

Hello.

Today, I noticed the same entries in a log files such as '/var/log/syslog' and via dmesg(1) on 16.04 LTS Release (i386/x86_32 arch) with v4.4.0-120-generic (4.4.117) Linux kernel. The one thing, that is different is 'use_ibrs' value. In my case it's:

[~]$ sudo dmesg
[ 464.877859] use_ibrs = 0, use_ibpb = 4
[ 467.893757] sysctl_ibrs_enabled = 0, sysctl_ibpb_enabled = 0
[ 467.893762] use_ibrs = 4, use_ibpb = 4
(...)

I'm writing about this, because I don't see '0' in any of the above posts etc. Anyway, as Mr Kamal Mostafa wrote, I will wait for a "-proposed" kernel.

Thanks, best regards.

Just FYI folks: This fix does not yet appear in the kernels which just landed in -proposed (which are respins for an unrelated issue). The fix is still queued up for the next cycle (currently 4.4.0-123.147).

Is the fixed kernel available right now? Or when could it be available?
This "ubrs" messages are flooding our syslog log files on many machines.

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-trusty' to 'verification-done-trusty'. If the problem still exists, change the tag 'verification-needed-trusty' to 'verification-failed-trusty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Hello.

Yes, Mr. Kamal Mostafa is right. Linux kernel v4.4.0-123.147 will contain a fix for all of these issues with, for example, '/var/log/syslog' flooding with 'ibrs_dump', 'use_ibrs/ibpb' and 'sysctl_ibrs{,ibpb}_enabled' entries. Now, a "proposed" kernel is v4.4.0-122.146 with a fix for "Redpine: WiFi scan stopping issue observed with BLE (LP: #1757435)".

Anyway, next kernel version, mentioned above will be updated, at last, to the v4.4.128 stable release (current version, available e.g. in 16.04 LTS is v4.4.117) and will contain this patch, of course:

* ibrs/ibpb fixes result in excessive kernel logging (LP: #1755627)
  - SAUCE: remove ibrs_dump sysctl interface

So, we have to wait a little more, because so-called "master-next" branch (with all needed patches for already mentioned v4.4.128 Stable release) was updated only 24 hours ago. Additionally, there will be also a couple of 'x86/spectre' and 'x86/retpoline' patches (mainly in v4.4.118).

Thanks, best regards.

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-artful' to 'verification-done-artful'. If the problem still exists, change the tag 'verification-needed-artful' to 'verification-failed-artful'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Verified fixed in all affected -proposed kernels:
 trusty 3.13.0-146.195
 xenial 4.4.0-123.147
 artful 4.13.0-40.45

Hello.

I just wanto to confirm: everything seems to be okay after updating Linux kernel to v4.4.0-123-generic. However, I would like to ask a question about 'intel-microcode' and 'amd64-microcode' packages. During system updating process via apt(8), there was an information that "The following NEW packages will be installed" etc. and it was about two mentioned 'microcode' packages.

I did not have these packages installed, until then. It's an Intel processor, but it seems, that Intel Corporation will not publish any microcode updates for some processor. Intel reveals (on Apr. 3., 2018) list of processors that won't receive Meltdown and Spectre patches. It seems, that some of older processors won't receive microcode updates designed to mitigate the vulnerabilities: Bloomfield, Bloomfield Xeon, Clarksfield, Gulftown etc.

So, I would like to ask if it was normal, that apt(8) installed such a packages? And why both since it's an Intel processor? Can I remove both packages (since there is no changes related to the microcode and "Spectre & Meltdown" mitigation; just 'revision' change in '/proc/cpuinfo' virtual file or/and dmesg(1) etc.)?

In sum two questions:

✗ why apt(8) installed two 'microcode' packages during Linux kernel v4.4.0-123-generic updates?
✗ can 'intel/amd64-microcode' packages be removed (since there is no difference with "Spectre & Meltdown" mitigations)?

I apologize for asking such a questions here, but this bug is about 'ibrs/ibpb' (a method to "Spectre & Meltdown" mitigation etc.) and Linux kernel update (v4.4.0-123-generic) during which, two 'microcode' packages were installed.

Thanks, best regards.

I am still experiencing this issue after upgrading to xenial 16.04.4 LTS 4.4.0-124-generic.

It looks like it got deferred to 4.4.0-125 according to the changelog [1].

[1] https://launchpad.net/ubuntu/+source/linux/4.4.0-125.150

Indeed, the -proposed kernel versions which included this fix were all bumped out due to an unrelated security fix release. The next kernel versions (now available in -proposed) include the fix again:

 trusty 3.13.0-148.197
 xenial 4.4.0-125.150
 artful 4.13.0-42.47

This bug was fixed in the package linux - 3.13.0-149.199

---------------
linux (3.13.0-149.199) trusty; urgency=medium

  * CVE-2018-3639 (powerpc)
    - SAUCE: rfi-flush: update H_CPU_* macro names to upstream
    - SAUCE: rfi-flush: update plpar_get_cpu_characteristics() signature to
      upstream
    - powerpc/pseries: Support firmware disable of RFI flush
    - powerpc/powernv: Support firmware disable of RFI flush
    - powerpc/64s: Allow control of RFI flush via debugfs
    - powerpc/rfi-flush: Move the logic to avoid a redo into the debugfs code
    - powerpc/rfi-flush: Always enable fallback flush on pseries
    - powerpc/rfi-flush: Differentiate enabled and patched flush types
    - powerpc/pseries: Add new H_GET_CPU_CHARACTERISTICS flags
    - powerpc: Add security feature flags for Spectre/Meltdown
    - powerpc/pseries: Set or clear security feature flags
    - powerpc/powernv: Set or clear security feature flags
    - powerpc/powernv: Use the security flags in pnv_setup_rfi_flush()
    - powerpc/pseries: Use the security flags in pseries_setup_rfi_flush()
    - powerpc/pseries: Fix clearing of security feature flags
    - powerpc: Move default security feature flags
    - powerpc/pseries: Restore default security feature flags on setup
    - powerpc/64s: Add support for a store forwarding barrier at kernel entry/exit
    - SAUCE: powerpc/64s: Move the data access exception out-of-line

  * CVE-2018-3639 (x86)
    - arch: Introduce post-init read-only memory
    - SAUCE: Add X86_FEATURE_ARCH_CAPABILITIES
    - SAUCE: x86: Add alternative_msr_write
    - x86/nospec: Simplify alternative_msr_write()
    - x86/pti: Do not enable PTI on CPUs which are not vulnerable to Meltdown
    - x86/bugs: Concentrate bug detection into a separate function
    - x86/bugs: Concentrate bug reporting into a separate function
    - x86/msr: Add definitions for new speculation control MSRs
    - x86/bugs: Read SPEC_CTRL MSR during boot and re-use reserved bits
    - x86/bugs, KVM: Support the combination of guest and host IBRS
    - x86/bugs: Expose /sys/../spec_store_bypass
    - x86/cpufeatures: Add X86_FEATURE_RDS
    - x86/bugs: Provide boot parameters for the spec_store_bypass_disable
      mitigation
    - x86/bugs/intel: Set proper CPU features and setup RDS
    - x86/bugs: Whitelist allowed SPEC_CTRL MSR values
    - x86/bugs/AMD: Add support to disable RDS on Fam[15,16,17]h if requested
    - x86/KVM/VMX: Expose SPEC_CTRL Bit(2) to the guest
    - x86/speculation: Create spec-ctrl.h to avoid include hell
    - prctl: Add speculation control prctls
    - x86/process: Allow runtime control of Speculative Store Bypass
    - x86/speculation: Add prctl for Speculative Store Bypass mitigation
    - nospec: Allow getting/setting on non-current task
    - proc: Provide details on speculation flaw mitigations
    - seccomp: Enable speculation flaw mitigations
    - SAUCE: x86/bugs: Honour SPEC_CTRL default
    - x86/bugs: Make boot modes __ro_after_init
    - prctl: Add force disable speculation
    - seccomp: Use PR_SPEC_FORCE_DISABLE
    - seccomp: Add filter flag to opt-out of SSB mitigation
    - seccomp: Move speculation migitation control to ar...

This bug was fixed in the package linux - 4.4.0-127.153

---------------
linux (4.4.0-127.153) xenial; urgency=medium

  * CVE-2018-3639 (powerpc)
    - powerpc/pseries: Support firmware disable of RFI flush
    - powerpc/powernv: Support firmware disable of RFI flush
    - powerpc/rfi-flush: Move the logic to avoid a redo into the debugfs code
    - powerpc/rfi-flush: Make it possible to call setup_rfi_flush() again
    - powerpc/rfi-flush: Always enable fallback flush on pseries
    - powerpc/rfi-flush: Differentiate enabled and patched flush types
    - powerpc/rfi-flush: Call setup_rfi_flush() after LPM migration
    - powerpc/pseries: Add new H_GET_CPU_CHARACTERISTICS flags
    - powerpc: Add security feature flags for Spectre/Meltdown
    - powerpc/pseries: Set or clear security feature flags
    - powerpc/powernv: Set or clear security feature flags
    - powerpc/64s: Move cpu_show_meltdown()
    - powerpc/64s: Enhance the information in cpu_show_meltdown()
    - powerpc/powernv: Use the security flags in pnv_setup_rfi_flush()
    - powerpc/pseries: Use the security flags in pseries_setup_rfi_flush()
    - powerpc/64s: Wire up cpu_show_spectre_v1()
    - powerpc/64s: Wire up cpu_show_spectre_v2()
    - powerpc/pseries: Fix clearing of security feature flags
    - powerpc: Move default security feature flags
    - powerpc/pseries: Restore default security feature flags on setup
    - SAUCE: powerpc/64s: Add support for a store forwarding barrier at kernel
      entry/exit

  * CVE-2018-3639 (x86)
    - SAUCE: Clean up IBPB and IBRS control functions and macros
    - SAUCE: Fix up IBPB and IBRS kernel parameters documentation
    - SAUCE: Remove #define X86_FEATURE_PTI
    - x86/cpufeature: Move some of the scattered feature bits to x86_capability
    - x86/cpufeature: Cleanup get_cpu_cap()
    - x86/cpu: Probe CPUID leaf 6 even when cpuid_level == 6
    - x86/cpufeatures: Add CPUID_7_EDX CPUID leaf
    - x86/cpufeatures: Add Intel feature bits for Speculation Control
    - SAUCE: x86/kvm: Expose SPEC_CTRL from the leaf
    - x86/cpufeatures: Add AMD feature bits for Speculation Control
    - x86/msr: Add definitions for new speculation control MSRs
    - SAUCE: x86/msr: Rename MSR spec control feature bits
    - x86/pti: Do not enable PTI on CPUs which are not vulnerable to Meltdown
    - x86/cpufeature: Blacklist SPEC_CTRL/PRED_CMD on early Spectre v2 microcodes
    - x86/speculation: Add basic IBPB (Indirect Branch Prediction Barrier) support
    - x86/speculation: Add <asm/msr-index.h> dependency
    - x86/cpufeatures: Clean up Spectre v2 related CPUID flags
    - x86/cpuid: Fix up "virtual" IBRS/IBPB/STIBP feature bits on Intel
    - SAUCE: x86/speculation: Move vendor specific IBRS/IBPB control code
    - SAUCE: x86: Add alternative_msr_write
    - SAUCE: x86/nospec: Simplify alternative_msr_write()
    - SAUCE: x86/bugs: Concentrate bug detection into a separate function
    - SAUCE: x86/bugs: Concentrate bug reporting into a separate function
    - arch: Introduce post-init read-only memory
    - SAUCE: x86/bugs: Read SPEC_CTRL MSR during boot and re-use reserved bits
    - SAUCE: x86/bugs, KVM: Support the combination of guest a...

This bug was fixed in the package linux - 4.13.0-43.48

---------------
linux (4.13.0-43.48) artful; urgency=medium

  * CVE-2018-3639 (powerpc)
    - SAUCE: rfi-flush: update H_CPU_* macro names to upstream
    - SAUCE: rfi-flush: update plpar_get_cpu_characteristics() signature to
      upstream
    - SAUCE: update pseries_setup_rfi_flush() capitalization to upstream
    - powerpc/pseries: Support firmware disable of RFI flush
    - powerpc/powernv: Support firmware disable of RFI flush
    - powerpc/64s: Allow control of RFI flush via debugfs
    - powerpc/rfi-flush: Move the logic to avoid a redo into the debugfs code
    - powerpc/rfi-flush: Always enable fallback flush on pseries
    - powerpc/rfi-flush: Differentiate enabled and patched flush types
    - powerpc/pseries: Add new H_GET_CPU_CHARACTERISTICS flags
    - powerpc: Add security feature flags for Spectre/Meltdown
    - powerpc/powernv: Set or clear security feature flags
    - powerpc/pseries: Set or clear security feature flags
    - powerpc/powernv: Use the security flags in pnv_setup_rfi_flush()
    - powerpc/pseries: Use the security flags in pseries_setup_rfi_flush()
    - powerpc/pseries: Fix clearing of security feature flags
    - powerpc: Move default security feature flags
    - powerpc/pseries: Restore default security feature flags on setup
    - powerpc/64s: Add support for a store forwarding barrier at kernel entry/exit

  * CVE-2018-3639 (x86)
    - SAUCE: Add X86_FEATURE_ARCH_CAPABILITIES
    - SAUCE: x86: Add alternative_msr_write
    - x86/nospec: Simplify alternative_msr_write()
    - x86/pti: Do not enable PTI on CPUs which are not vulnerable to Meltdown
    - x86/bugs: Concentrate bug detection into a separate function
    - x86/bugs: Concentrate bug reporting into a separate function
    - x86/msr: Add definitions for new speculation control MSRs
    - x86/bugs: Read SPEC_CTRL MSR during boot and re-use reserved bits
    - x86/bugs, KVM: Support the combination of guest and host IBRS
    - x86/bugs: Expose /sys/../spec_store_bypass
    - x86/cpufeatures: Add X86_FEATURE_RDS
    - x86/bugs: Provide boot parameters for the spec_store_bypass_disable
      mitigation
    - x86/bugs/intel: Set proper CPU features and setup RDS
    - x86/bugs: Whitelist allowed SPEC_CTRL MSR values
    - x86/bugs/AMD: Add support to disable RDS on Fam[15,16,17]h if requested
    - x86/KVM/VMX: Expose SPEC_CTRL Bit(2) to the guest
    - x86/speculation: Create spec-ctrl.h to avoid include hell
    - prctl: Add speculation control prctls
    - x86/process: Allow runtime control of Speculative Store Bypass
    - x86/speculation: Add prctl for Speculative Store Bypass mitigation
    - nospec: Allow getting/setting on non-current task
    - proc: Provide details on speculation flaw mitigations
    - seccomp: Enable speculation flaw mitigations
    - SAUCE: x86/bugs: Honour SPEC_CTRL default
    - x86/bugs: Make boot modes __ro_after_init
    - prctl: Add force disable speculation
    - seccomp: Use PR_SPEC_FORCE_DISABLE
    - seccomp: Add filter flag to opt-out of SSB mitigation
    - seccomp: Move speculation migitation control to arch code
    - x86/speculation: Make "seccomp" the...