Ubuntu
linux package

Ubuntu 16.10 KVM: Running out of memory if trying to bringup guest with more than 64VFs.

Bug #1654067 reported by bugproxy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Undecided
Tim Gardner
Yakkety
Fix Released
Undecided
Tim Gardner
Zesty
Fix Released
Undecided
Tim Gardner

Bug Description

== Comment: #0 - Carol L. Soto - 2017-01-04 12:03:57 ==
---Problem Description---
If user try to create a guest with more than 64 VFs then the guest will have issues to come up but if user keeps trying to bringup this guest over and over the system will eventually start seeing out of memory messages.
Like this:
Jan 3 14:33:13 ltc-fire1 kernel: [ 1053.950753] systemd invoked oom-killer: gfp_mask=0x26000d0(GFP_TEMPORARY|__GFP_NOTRACK), order=0, oom_score_adj=0
Jan 3 14:33:16 ltc-fire1 kernel: [ 1053.950757] systemd cpuset=/ mems_allowed=0
Jan 3 14:33:16 ltc-fire1 kernel: [ 1053.950761] CPU: 104 PID: 1 Comm: systemd Tainted: G OE 4.8.0-34-generic #36-Ubuntu
Jan 3 14:33:16 ltc-fire1 kernel: [ 1053.950763] Call Trace:

---uname output---
Linux ltc-fire1 4.8.0-34-generic #36 SMP Tue Jan 3 15:27:22 CST 2017 ppc64le ppc64le ppc64le GNU/Linux

---Additional Hardware Info---
we are using Mellanox CX4 card with SRIOV enabled to be able to create a guest with more than 64 VFs.

Machine Type = P8

---Steps to Reproduce---
 create a guest with more than 64 VFs and then just try to bring up the guest by
virsh start --console ltc-fire1-vm3-ubuntu-16.10-69VFs.
Will attach example guest xml.

== Comment: #2 - Carol L. Soto - 2017-01-04 12:09:32 ==
If I build a kernel with these upstream patches, then I do not see the memory leaks messages in dmesg:

From 88f54a3581eb9deaa3bd1aade40aef266d782385 Mon Sep 17 00:00:00 2001
From: Alexey Kardashevskiy <email address hidden>
Date: Wed, 30 Nov 2016 17:51:59 +1100
Subject: powerpc/iommu: Pass mm_struct to init/cleanup helpers

From d7baee6901b34c4895eb78efdbf13a49079d7404 Mon Sep 17 00:00:00 2001
From: Alexey Kardashevskiy <email address hidden>
Date: Wed, 30 Nov 2016 17:52:00 +1100
Subject: powerpc/iommu: Stop using @current in mm_iommu_xxx

From 39701e56f5f16ea0cf8fc9e8472e645f8de91d23 Mon Sep 17 00:00:00 2001
From: Alexey Kardashevskiy <email address hidden>
Date: Wed, 30 Nov 2016 17:52:01 +1100
Subject: vfio/spapr: Postpone allocation of userspace version of TCE table

From 6f01cc692a16405235d5c34056455b182682123c Mon Sep 17 00:00:00 2001
From: Alexey Kardashevskiy <email address hidden>
Date: Wed, 30 Nov 2016 17:52:02 +1100
Subject: vfio/spapr: Add a helper to create default DMA window

From d9c728949ddc9de5734bf3b12ea906ca8a77f2a0 Mon Sep 17 00:00:00 2001
From: Alexey Kardashevskiy <email address hidden>
Date: Wed, 30 Nov 2016 17:52:03 +1100
Subject: vfio/spapr: Postpone default window creation

From bc82d122ae4a0e9f971f13403995898fcfa0c09e Mon Sep 17 00:00:00 2001
From: Alexey Kardashevskiy <email address hidden>
Date: Wed, 30 Nov 2016 17:52:04 +1100
Subject: vfio/spapr: Reference mm in tce_container

From 4b6fad7097f883335b6d9627c883cb7f276d94c9 Mon Sep 17 00:00:00 2001
From: Alexey Kardashevskiy <email address hidden>
Date: Wed, 30 Nov 2016 17:52:05 +1100
Subject: powerpc/mm/iommu, vfio/spapr: Put pages on VFIO container shutdown

Revision history for this message
bugproxy (bugproxy) wrote : guest xml

Default Comment by Bridge

tags: added: architecture-ppc64le bugnameltc-150216 severity-high targetmilestone-inin---
Revision history for this message
bugproxy (bugproxy) wrote : Kernel log where we can see the out ot mem messages.

Default Comment by Bridge

Changed in ubuntu:
assignee: nobody → Taco Screen team (taco-screen-team)
affects: ubuntu → linux (Ubuntu)
Revision history for this message
Tim Gardner (timg-tpi) wrote :

https://lists.ubuntu.com/archives/kernel-team/2017-January/081705.html

Changed in linux (Ubuntu Yakkety):
assignee: nobody → Tim Gardner (timg-tpi)
status: New → In Progress
Changed in linux (Ubuntu Zesty):
assignee: Taco Screen team (taco-screen-team) → Tim Gardner (timg-tpi)
status: New → In Progress
status: In Progress → Fix Committed
Luis Henriques (henrix)
Changed in linux (Ubuntu Yakkety):
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (14.3 KiB)

This bug was fixed in the package linux - 4.9.0-12.13

---------------
linux (4.9.0-12.13) zesty; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1655122

  * Zesty update to v4.9.2 stable release (LP: #1655114)
    - ssb: Fix error routine when fallback SPROM fails
    - ath10k: fix soft lockup during firmware crash/hw-restart
    - rtlwifi: Fix enter/exit power_save
    - perf/x86: Fix exclusion of BTS and LBR for Goldmont
    - perf annotate: Don't throw error for zero length symbols
    - perf/x86/intel/cstate: Prevent hotplug callback leak
    - rtl8xxxu: Work around issue with 8192eu and 8723bu devices not reconnecting
    - cfg80211/mac80211: fix BSS leaks when abandoning assoc attempts
    - ath9k: fix ath9k_hw_gpio_get() to return 0 or 1 on success
    - ath9k: Really fix LED polarity for some Mini PCI AR9220 MB92 cards.
    - ath9k: do not return early to fix rcu unlocking
    - Revert "mmc: sdhci: Reset cmd and data circuits after tuning failure"
    - mmc: sdhci: Fix recovery from tuning timeout
    - regulator: stw481x-vmmc: fix ages old enable error
    - mmc: sd: Meet alignment requirements for raw_ssr DMA
    - timekeeping_Force_unsigned_clocksource_to_nanoseconds_conversion
    - gpio: stmpe: fix interrupt handling bug
    - gpio: chardev: Return error for seek operations
    - arm64: tegra: Add VDD_GPU regulator to Jetson TX1
    - clk: bcm2835: Avoid overwriting the div info when disabling a pll_div clk
    - thermal: hwmon: Properly report critical temperature in sysfs
    - docs: sphinx-extensions: make rstFlatTable work with docutils 0.13
    - hv: acquire vmbus_connection.channel_mutex in vmbus_free_channels()
    - staging/lustre/osc: Revert erroneous list_for_each_entry_safe use
    - staging: lustre: ldlm: pl_recalc time handling is wrong
    - staging: comedi: ni_mio_common: fix M Series ni_ai_insn_read() data mask
    - staging: comedi: ni_mio_common: fix E series ni_ai_insn_read() data
    - ACPI / video: Add force_native quirk for Dell XPS 17 L702X
    - ACPI / video: Add force_native quirk for HP Pavilion dv6
    - drm/amdgpu/si: load the proper firmware on 0x87 oland boards
    - drm/amdgpu: add additional pci revision to dpm workaround
    - drm/amdgpu: Store CRTC relative amdgpu_crtc->cursor_x/y values
    - drm/amdgpu: Also call cursor_move_locked when the cursor size changes
    - drm/amd/amdgpu: enable GUI idle INT after enabling CGCG
    - drm/nouveau/gr: fallback to legacy paths during firmware lookup
    - drm/nouveau/kms: lvds panel strap moved again on maxwell
    - drm/nouveau/bios: require checksum to match for fast acpi shadow method
    - drm/nouveau/ltc: protect clearing of comptags with mutex
    - drm/nouveau/ttm: wait for bo fence to signal before unmapping vmas
    - drm/nouveau/i2c/gk110b,gm10x: use the correct implementation
    - drm/nouveau/fifo/gf100-: protect channel preempt with subdev mutex
    - drm/radeon: Also call cursor_move_locked when the cursor size changes
    - drm/radeon: Hide the HW cursor while it's out of bounds
    - drm/radeon: add additional pci revision to dpm workaround
    - drm/radeon/si: load the proper firmware on 0x87 oland boards
    - drm/...

Changed in linux (Ubuntu Zesty):
status: Fix Committed → Fix Released
bugproxy (bugproxy)
tags: added: targetmilestone-inin1704
removed: targetmilestone-inin---
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2017-01-23 15:26 EDT-------
I tried Ubuntu 17.04 kernel 4.9.0-12-generic and can not recreate the issue.

tags: added: verification-done
Revision history for this message
Thadeu Lima de Souza Cascardo (cascardo) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-yakkety' to 'verification-done-yakkety'. If the problem still exists, change the tag 'verification-needed-yakkety' to 'verification-failed-yakkety'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-yakkety
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2017-02-10 15:41 EDT-------
I tried kernel 4.8.0-38-generic #41-Ubuntu SMP Wed Feb 8 14:26:14 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux
and I can not recreate the issue.

tags: added: verification-done-yakkety
removed: verification-needed-yakkety
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (20.4 KiB)

This bug was fixed in the package linux - 4.8.0-38.41

---------------
linux (4.8.0-38.41) yakkety; urgency=low

  [ Thadeu Lima de Souza Cascardo ]

  * Release Tracking Bug
    - LP: #1661232

  * Backport Dirty COW patch to prevent wineserver freeze (LP: #1658270)
    - SAUCE: mm: Respect FOLL_FORCE/FOLL_COW for thp

  * Kdump through NMI SMP and single core not working on Ubuntu16.10
    (LP: #1630924)
    - x86/hyperv: Handle unknown NMIs on one CPU when unknown_nmi_panic
    - SAUCE: hv: don't reset hv_context.tsc_page on crash

  * Call trace when testing fstat stressor on ppc64el with virtual keyboard and
    mouse present (LP: #1652132)
    - HID: usbhid: Quirk a AMI virtual mouse and keyboard with ALWAYS_POLL

  * regression in linux-libc-dev in yakkety: C++ style comments are not allowed
    in ISO C90 (LP: #1659654)
    - generic syscalls: kill cruft from removed pkey syscalls

  * [16.04.2] POWER9 patches on top of 4.8 (LP: #1650263)
    - powerpc/book3s: Add a cpu table entry for different POWER9 revs
    - powerpc/mm/radix: Use different RTS encoding for different POWER9 revs
    - powerpc/mm/radix: Use different pte update sequence for different POWER9
      revs
    - powerpc/mm: Update the HID bit when switching from radix to hash
    - powerpc/64/kexec: NULL check "clear_all" in kexec_sequence
    - powerpc/64/kexec: Fix MMU cleanup on radix
    - powerpc/mm: Add radix flush all with IS=3
    - powerpc/64/kexec: Copy image with MMU off when possible
    - powerpc/64: Simplify adaptation to new ISA v3.00 HPTE format
    - powerpc/mm/radix: Invalidate ERAT on tlbiel for POWER9 DD1
    - powerpc/mm: Fix missing update of HID register on secondary CPUs
    - powerpc/64: Add some more SPRs and SPR bits for POWER9
    - powerpc/64: Provide functions for accessing POWER9 partition table
    - powerpc/powernv: Define real-mode versions of OPAL XICS accessors
    - powerpc/64: Define new ISA v3.00 logical PVR value and PCR register value
    - mm: update mmu_gather range correctly
    - mm/hugetlb: add tlb_remove_hugetlb_entry for handling hugetlb pages
    - mm: add tlb_remove_check_page_size_change to track page size change
    - powerpc: Revert Load Monitor Register Support
    - powerpc/mm: Correct process and partition table max size
    - powernv: Clear SPRN_PSSCR when a POWER9 CPU comes online
    - powerpc/mm/radix: Setup AMOR in HV mode to allow key 0
    - powerpc/mm: Detect instruction fetch denied and report
    - powerpc/mm/radix: Prevent kernel execution of user space
    - powerpc/mm: Rename hugetlb-radix.h to hugetlb.h
    - powerpc/mm/hugetlb: Handle hugepage size supported by hash config
    - powerpc/mm: Introduce _PAGE_LARGE software pte bits
    - powerpc/mm: Add radix__tlb_flush_pte_p9_dd1()
    - powerpc/mm: update radix__ptep_set_access_flag to not do full mm tlb flush
    - powerpc/mm: update radix__pte_update to not do full mm tlb flush
    - powerpc/mm: Batch tlb flush when invalidating pte entries
    - powerpc/sparse: Make a bunch of things static
    - powerpc/perf: factor out the event format field
    - powerpc/perf: update attribute_group data structure
    - powerpc/perf: power9 raw event format en...

Changed in linux (Ubuntu Yakkety):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.