[UBUNTU 20.04] KVM nesting support leaks too much memory, might result in stalls during cleanup

Bug #1974017 reported by bugproxy
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Fix Released
Medium
Skipper Bug Screeners
linux (Ubuntu)
Invalid
Undecided
Skipper Bug Screeners
Focal
Fix Released
Medium
Canonical Kernel Team
Impish
Fix Released
Medium
Canonical Kernel Team
Jammy
Fix Released
Medium
Canonical Kernel Team

Bug Description

SRU Justification:
==================

[Impact]

 * If running KVM with nesting support (e.g. 'kvm.nested=1' on the kernel
   command line), the shadow page table code will produce too many entries
   in the shadow code.

 * The below mentioned upstream fix will prevent the entries from being
   piled up, by checking for existing entries at insert time.

 * This measurably reduces the list length and is faster than traversing
   the list at shutdown time only.

[Fix]

 * a06afe8383080c630a7a528b8382fc6bb4925b61 a06afe838308
   "KVM: s390: vsie/gmap: reduce gmap_rmap overhead"

[Test Plan]

 * A IBM zSystems or LinuxONE LPAR on a z13 or newer is needed.

 * Ubuntu focal, impish or jammy needs to be installed
   and the Ubuntu LPAR setup as (1st level) KVM host,
   allowing nested virtualization.

 * Now setup one (or more) KVM virtual machines,
   with similar Ubuntu releases,
   and define one or more of them again as (2nd level) KVM host.

 * Define several KVM virtual machines on this (2nd level) KVM host
   in a memory constraint fashion,
   so that a lot of memory mapping is caused.

 * Let such a system run for a while under load.

 * Now shutdown one (or more) 2nd level VMs and notice the
   time it takes.

 * With the patch in place this time should be considerably
   quicker than without.

 * The result is reduced mapping (gmap_rmap) overhead,
   less danger of leaking memory
   and a better responding system.

[Where problems could occur]

 * In case wrong entries are freed up this will harm the virtual
   memory management and may even lead to crashes.

 * In case the pointer handling is not done properly,
   again crashes may occur.

 * But with net just five new lines the patch is pretty short, readable
   and the modifications traceable in arch/s390/mm/gmap.c only.

 * The changes are limited to s390/mm only,
   hence don't affect other architectures.

[Other Info]

 * The commit was upstream accepted in v5.18-rc6.

 * Since the planned target kernel for kinetic is 5.19,
   the kinetic kernel does not need to be patched.

 * Hence the SRUs are for jammy, impish and focal.

__________

KVM nesting support consumes too much memory

When running KVM with nesting support (kvm.nested=1 on the kernel command line) the shadow page table code will produce too many entries in the shadow code.

There is an upstream fix that will prevent the majority of the problem:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a06afe8383080c630a7a528b8382fc6bb4925b61

The fix is needed for 20.04 and 22.04.

CVE References

bugproxy (bugproxy)
tags: added: architecture-s39064 bugnameltc-198271 severity-high targetmilestone-inin2004
Changed in ubuntu:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu → linux (Ubuntu)
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
importance: Undecided → Medium
Revision history for this message
Frank Heimes (fheimes) wrote (last edit ):

Since this seems to be upstream with 5.18-rc6, and the target kernel for kinetic/22.10 is 5.19,
this LP bug is an SRU for jammy, impish and focal.
I've marked the affected series accordingly.

Frank Heimes (fheimes)
Changed in linux (Ubuntu Jammy):
assignee: nobody → Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: New → In Progress
Changed in linux (Ubuntu Jammy):
status: New → In Progress
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

Since this seems to be upstream with 5.18-rc4, and the target kernel for kinetic/22.10 is 5.19,

Frank Heimes (fheimes)
description: updated
Revision history for this message
Frank Heimes (fheimes) wrote :

Kernel test builds for jammy, impish and focal are available here:
https://launchpad.net/~fheimes/+archive/ubuntu/lp1974017

Revision history for this message
Frank Heimes (fheimes) wrote :

SRU request submitted to the Ubuntu kernel team mailing list for jammy, impish and focal:
https://lists.ubuntu.com/archives/kernel-team/2022-May/thread.html#130450
Changing status to 'In Progress' for jammy, impish and focal.

Changed in linux (Ubuntu Impish):
status: New → In Progress
Changed in linux (Ubuntu Focal):
status: New → In Progress
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
Changed in linux (Ubuntu Impish):
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
Changed in linux (Ubuntu Jammy):
assignee: Frank Heimes (fheimes) → Canonical Kernel Team (canonical-kernel-team)
Stefan Bader (smb)
Changed in linux (Ubuntu Focal):
importance: Undecided → Medium
Changed in linux (Ubuntu Impish):
importance: Undecided → Medium
Changed in linux (Ubuntu Jammy):
importance: Undecided → Medium
Changed in linux (Ubuntu):
status: New → Invalid
Changed in linux (Ubuntu Focal):
status: In Progress → Fix Committed
Changed in linux (Ubuntu Impish):
status: In Progress → Fix Committed
Changed in linux (Ubuntu Jammy):
status: In Progress → Fix Committed
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: In Progress → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/5.15.0-36.37 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-jammy
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/5.4.0-121.137 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-focal
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/5.13.0-52.59 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-impish' to 'verification-done-impish'. If the problem still exists, change the tag 'verification-needed-impish' to 'verification-failed-impish'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-impish
bugproxy (bugproxy)
tags: added: verification-done-impish
removed: verification-needed-impish
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2022-06-20 07:08 EDT-------
verification done.

tags: added: verification-done-focal verification-done-jammy
removed: verification-needed-focal verification-needed-jammy
Revision history for this message
Frank Heimes (fheimes) wrote :

Thx for the verification!

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 5.4.0-121.137

---------------
linux (5.4.0-121.137) focal; urgency=medium

  * focal/linux: 5.4.0-121.137 -proposed tracker (LP: #1978666)

  * Packaging resync (LP: #1786013)
    - debian/dkms-versions -- update from kernel-versions (main/2022.05.30)

  * CVE-2022-28388
    - can: usb_8dev: usb_8dev_start_xmit(): fix double dev_kfree_skb() in error
      path

  * test_vxlan_under_vrf.sh in net from ubuntu_kernel_selftests failed (Check VM
    connectivity through VXLAN (underlay in the default VRF) [FAIL])
    (LP: #1871015)
    - selftests: net: test_vxlan_under_vrf: fix HV connectivity test

  * [UBUNTU 20.04] CPU-MF: add extended counter set definitions for new IBM z16
    (LP: #1974433)
    - s390/cpumf: add new extended counter set for IBM z16

  * [UBUNTU 20.04] KVM nesting support leaks too much memory, might result in
    stalls during cleanup (LP: #1974017)
    - KVM: s390: vsie/gmap: reduce gmap_rmap overhead

  * [UBUNTU 20.04] Null Pointer issue in nfs code running Ubuntu on IBM Z
    (LP: #1968096)
    - NFS: Fix up nfs_ctx_key_to_expire()

 -- Stefan Bader <email address hidden> Wed, 15 Jun 2022 15:13:27 +0200

Changed in linux (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 5.13.0-52.59

---------------
linux (5.13.0-52.59) impish; urgency=medium

  * impish/linux: 5.13.0-52.59 -proposed tracker (LP: #1978628)

  * CVE-2022-28388
    - can: usb_8dev: usb_8dev_start_xmit(): fix double dev_kfree_skb() in error
      path

  * test_vxlan_under_vrf.sh in net from ubuntu_kernel_selftests failed (Check VM
    connectivity through VXLAN (underlay in the default VRF) [FAIL])
    (LP: #1871015)
    - selftests: net: test_vxlan_under_vrf: fix HV connectivity test
    - selftests: test_vxlan_under_vrf: Fix broken test case

  * [UBUNTU 20.04] CPU-MF: add extended counter set definitions for new IBM z16
    (LP: #1974433)
    - s390/cpumf: add new extended counter set for IBM z16

  * [UBUNTU 20.04] KVM nesting support leaks too much memory, might result in
    stalls during cleanup (LP: #1974017)
    - KVM: s390: vsie/gmap: reduce gmap_rmap overhead

  * [UBUNTU 20.04] Null Pointer issue in nfs code running Ubuntu on IBM Z
    (LP: #1968096)
    - NFS: Fix up nfs_ctx_key_to_expire()

  * prevent kernel panic with overlayfs + shiftfs (LP: #1973620)
    - SAUCE: overlayfs: prevent dereferencing struct file in ovl_vm_prfile_set()

  * Packaging resync (LP: #1786013)
    - debian/dkms-versions -- update from kernel-versions (main/2022.05.30)

 -- Luke Nowakowski-Krijger <email address hidden> Wed, 15 Jun 2022 12:56:23 -0700

Changed in linux (Ubuntu Impish):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (5.0 KiB)

This bug was fixed in the package linux - 5.15.0-40.43

---------------
linux (5.15.0-40.43) jammy; urgency=medium

  * jammy/linux: 5.15.0-40.43 -proposed tracker (LP: #1978610)

  * Packaging resync (LP: #1786013)
    - debian/dkms-versions -- update from kernel-versions (main/2022.05.30)

  * [SRU][OEM-5.14/OEM-5.17/J][PATCH 0/2] Fix system hangs after s2idle on AMD
    A+A GPU (LP: #1975804)
    - Revert "drm/amd/pm: keep the BACO feature enabled for suspend"
    - drm/amd: Don't reset dGPUs if the system is going to s2idle

  * [SRU][OEM-5.14/OEM-5.17/J][PATCH 0/1] Read the discovery registers for
    AMD_SFH (LP: #1975798)
    - HID: amd_sfh: Add support for sensor discovery

  * [UBUNTU 20.04] CPU-MF: add extended counter set definitions for new IBM z16
    (LP: #1974433)
    - s390/cpumf: add new extended counter set for IBM z16

  * [UBUNTU 20.04] KVM nesting support leaks too much memory, might result in
    stalls during cleanup (LP: #1974017)
    - KVM: s390: vsie/gmap: reduce gmap_rmap overhead

  * [UBUNTU 20.04] Null Pointer issue in nfs code running Ubuntu on IBM Z
    (LP: #1968096)
    - NFS: Fix up nfs_ctx_key_to_expire()

  * Fix REG_WAIT timeout for Yellow Carp (LP: #1971417)
    - drm/amd/display: Clear encoder assignments when state cleared.
    - drm/amd/display: fix stale info in link encoder assignment
    - drm/amd/display: Query all entries in assignment table during updates.
    - drm/amd/display: Initialise encoder assignment when initialising dc_state

  * Enable hotspot feature for Realtek 8821CE (LP: #1969326)
    - rtw88: Add update beacon flow for AP mode
    - rtw88: 8821c: Enable TX report for management frames
    - rtw88: do PHY calibration while starting AP
    - rtw88: 8821c: fix debugfs rssi value
    - rtw88: add ieee80211:sta_rc_update ops

  * prevent kernel panic with overlayfs + shiftfs (LP: #1973620)
    - SAUCE: overlayfs: prevent dereferencing struct file in ovl_vm_prfile_set()

  * disable Intel DMA remapping by default (LP: #1971699)
    - Revert "UBUNTU: [Config] enable Intel DMA remapping options by default"

  * Mute/mic LEDs no function on Elitebook 630 (LP: #1974111)
    - ALSA: hda/realtek: fix right sounds and mute/micmute LEDs for HP machine

  * [Regression] Real-time Kernel Build Failure (LP: #1972899)
    - x86/mm: Include spinlock_t definition in pgtable.

  * build backport-iwlwifi-dkms as linux-modules-iwlwifi-ABI (LP: #1969434)
    - [Packaging] support standalone dkms module builds
    - [Packaging] drop do_<mod> arch specific configs

  * IPU6 camera has no function on Andrews MLK (LP: #1964983)
    - SAUCE: IPU6: 2022-03-11 alpha release for Andrews MLK
    - [Config] IPU6: enable OV02C10 sensor
    - SAUCE: IPU6: 2022-04-01 Andrews MLK PV release
    - SAUCE: spi: ljca: return when a sub-transaction first failed
    - SACUE: ljca: disable parallelly stub write
    - SAUCE: ljca: fix race condition issue in runtime PM
    - SAUCE: i2c-ljca: fix a null pointer access issue on tgl
    - SAUCE: ljca: fix a typo issue
    - SAUCE: ljca: assume stub enum failed as a warning
    - SAUCE: mei: cleanup header file including
    - SAUCE: intel_ulpss: Replaced by LJCA and remove
    ...

Read more...

Changed in linux (Ubuntu Jammy):
status: Fix Committed → Fix Released
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: Fix Committed → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2022-07-06 19:56 EDT-------
Fix verified and released, therefore we can close this bug.
Thanks everybody for your work.

Changing status to: CLOSED

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.