Memory balloning in Hyper-V generation 2 does not work

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1294283

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

After some time the problem reappeared again today. Ballooning worked at first and the VM got more than the initial memory, but after some time this double kernel stack traces were back.

The time these end in the kernel log is just determined by the time I uploaded the kernel log, from my perspective it looks like these messages will continue to appear every 120 seconds.

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

The problem just reappeared after the latest kernel upgrade. Apport data is attached.

I believe this is another occurance of the issue in bug #1292400.

Bug 1292400 references an email from KY which describes a patch to mm/memory_hotplug.c. KY confirmed the patch in the referenced email was not submitted, but equivalent patches have been submitted.

As a way to validate if equivalent patches have been submitted, the Linux-Next kernel from April 15, 2015 was built and installed on a 14.10 Gen 2 VM. After rebooting into the Linux-Next kernel (version 4.0.0-next-20150415), ballooning worked.
Tested dynamic memory configuration:
    Minimum Memory: 1024MB
    Startup Memory : 2048MB
    Maximum Memory: 4096MB
    Memory buffer: 20%
    Memory Weight: 50

Two minutes after booting, memory was ballooned down from 2048MB to 1024MB
Ran stressapptest to generate 600MB of memory demand
Memory was ballooned up to 1762MB
After memory demand was removed, memory was ballooned down to 1024MB

Ballooning does work with the upstream kernel on a Ubuntu 14.10 Gen 2 VM.

This should be verified to work for 14.04.

Ballooning works. Hot Add has issues.
The original issue as reported has been resolved.

Initially, memory is successfully hot added to the VM. As demand continues to increase,

no additional memory is hot added. Eventually a hung task message is logged in syslog.

The call trace shows hot_add_req() on the call stack. At this point, attempting to

shutdown the system with "init 0" results in the system hung during the shutdown. The

console is showing messages of:
    INFO: task kworker/0:1:1250 blocked for more than 120 seconds.
          Not tainted 3.16.0-30-generic #40~14.04.1-Ubuntu
    INFO: task systemd-udev:1317 blocked for more than 120 seconds.
          Not tainted 3.16.0-30-generic #40~14.04.1-Ubuntu
See attached file for call traces in syslog.

Test environment
    Host: Windows Server 2012 R2
    VM: Ubuntu 14.04, upgraded to current packages (apt-get upgrade)
    VM Dynammic memory configuration
        Minimum Memory : 512MB
        Startup Memory : 512MB
        Maximum Memory : 4096
    Test is to run 8 instances of stressapptest where each instance consumes
    128MB for 90 seconds. There is a 5 second delay between starting each
    instance of stressapptest.

Consider adding patch https://github.com/alexngmsft/lis-next/commit/87e03a0943bb9e0760fcf2fe980908a68422a87c

Upstream original commit id: 4e4bd36f97b1492f19b3329ac74ed313da13de34

I built a Wily test kernel with a cherry-pick of commit 4e4bd36f97b1492f19b3329ac74ed313da13de34

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1294283/

Can you test this kernel and see if it resolves this bug?

Thanks in advance!

I was able to use this test kernel and run our memory stress test application on it. I was able to balloon it up from its startup memory to 3.5GB. Looks like the test kernel works okay. No errors on our stress tests, no unexpected messages in the syslog.

Thanks for testing. I sent an request for inclusion of this commit in Wily.

I built a Vivid test kernel with a cherry-pick of commit 4e4bd36f97b1492f19b3329ac74ed313da13de34

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1294283/vivid

Can you test this kernel and see if it resolves this bug?

Thanks in advance!

Hot Add issues remain.

Memory is Hot Added as demand increases.
Memory is not ballooned down after demand is reduced.
syslog has numerous call traces for hung task.
Call stack shows calls to mem_hotplug_begin()
Tail end of syslog attached as syslog.log.

Test procedure:
  - Create 15.04 VM.
  - Install test kernel.
  - Configure dynamic memory
      - Startup memory : 1536 MB
      - Minimum memory : 1024 MB
      - Maximum memory : 8192 MB
  - Reboot
  - Start 7 separate instances of stressapptest.
      - Each instance consumes 128 MB and runs for 90 seconds.
      - Allow 5 seconds before starting a new instance.
  - Wait for Hyper-V host to balloon down assigned memory.
  - Repeat test.

Test results:

VM State Assigned Mem Memory Demand
======== ============ =============
Idle 1024 MB 399 MB
stress x7 1778 MB 1778 MB
no stress 1778 MB 462 MB
10 min later 1778 MB 462 MB
20 min later 1778 MB 462

So the ballooning up of memory is now working, but it is not ballooning back down? Is this also the case with the Wily test kernel I posted in comment #36 or is that test kernel find and this is only happening with my Vivid test kernel?

Let's keep the terms clear here:

"hot add" raises the high water mark of available memory and will increase the total memory seen by the Linux VM. Because there's no "hot remove" (yet) free -m will increase with "hot add" but never decrease.

"balloon" will take memory and allocate it to buffers controlled by the hyper-v driver, which will then release that memory back to the host. This is "ballooning down". If the hyper-v driver cannot grab memory and put it aside, for whatever reason, it can not release it to the host.

The balloon code can also ask the host for memory and release it back to the OS if demand increases again.

From the host side, we are observing the numbers Nick is reporting. From the VM side, you won't be able to see it go down. The assigned memory should decrease, and what we usually see is that it gradually goes down to the minimum level.

Hi Joseph,

In Linux-next, we have seen an issue caused by memory not being onlined after hot adding. This bug caused our guest to falsely report memory to host as being assigned; even though it was actually inaccessible. We wonder if this is what you’re seeing as well.

There were two relevant patches to fix hot-add:
- https://lkml.org/lkml/diff/2015/7/17/374/1 <= this patch fixes BUG_ON crash when hot-adding memory
- https://lkml.org/lkml/diff/2015/8/3/104/1 <= this patch fixes the onlining issue described above

I have not reproduced this bug as I don't have and environment to do so. Maybe Jan Henke, the original bug reporter, can confirm your question in comment #42?

The Wily and Vivid test kernels I built only have the following commit cherry-picked per comment #34:

commit 4e4bd36f97b1492f19b3329ac74ed313da13de34
Author: Vitaly Kuznetsov <email address hidden>
Date: Fri May 29 11:18:02 2015 -0700

Looking at the patches you mentioned in comment #42, are the following two commits in mainline also needed to fix this bug?

commit 7ace99170789bc53cbb7e9e352d7a3851208fbcf
Author: Mel Gorman <email address hidden>
Date: Thu Aug 6 15:46:13 2015 -0700

    mm, meminit: allow early_pfn_to_nid to be used during runtime

commit e298ff75f133f2524bb6a9a305b17c5f6ff1a6b2
Author: Mel Gorman <email address hidden>
Date: Thu Aug 6 15:46:51 2015 -0700

    mm: initialize hotplugged pages as reserved

These last two commits were in mainline as of v4.2-rc6.

Sorry, I cannot comment on that matter. I do not have the setup to test it any more.

Do you need our assistance in setting up tests of this particular function on Hyper-V?

@Joshua do you mean me?

We're happy to help with the setup of a test case, and we can test things on our systems as well.

That still does not answer if you are asking me to test anything. I probably do not have anything to add here, especially since I upgraded the host system to Windows 10 in the mean time, any result would be therefore not exactly comparable.

Hi Joshua,

I found an evaluation copy of Hyper-V Server[0]. Do you happen to know if the evaluation copy will limit what type of tests can be performed?

I just wanted to check what patches we think are needed to resolve this bug? I build a test kernel with a cherry pick of only commit 4e4bd36f. However, it looks like there may be a total of three patches required:

commit 4e4bd36f
commit 7ace9917
commit e298ff75

Joshua or nmeier can you confirm this? If so, I'll build test kernels with all three of these commits for all Ubuntu releases.

Looking at a different request, may be missing

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=797f88c987b02a8de8d4fac94ec2877b92ec35a2
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=ba0c444153889a9b672974d85a4a57a8eeb49fe3

Wily has the following five commits now:

ba0c4441
797f88c9
4e4bd36f
7ace9917
e298ff75

Can Wily with the latest updates be tested to confirm it no longer exhibits this bug?

I built a Vivid test kernel with the five commits I also mentioned in comment #52. There were also some prerequisite commits required to get the patches to apply to a 3.19 kernel. The following commits were applied:

commit 530d15b
commit 0a1a86a
commit ba0c444
commit 797f88c
commit 4e4bd36
commit 8a942fd
commit 75a592a
commit 7ace991
commit e298ff7

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1294283/vivid/

Can this kernel be tested to see if it resolves this bug?

I tested kernel from #53 on Vivid and everything was ok. You can commit the changes on Vivid.

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-vivid' to 'verification-done-vivid'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

We'll test the kernel in -proposed right away. Thanks!

I built a trusty test kernel, which can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1294283/trusty/

Can you test this kernel and see if it resolves this bug? If it does, I'll submit the Trusty SRU request.

Thanks!

I tested kernel in -proposed on Vivid, memory ballooning works.

We tested the above test kernel on Trusty, works good, no errors found. You can submit the request.

Thanks!

This bug was fixed in the package linux - 3.19.0-41.46

---------------
linux (3.19.0-41.46) vivid; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1522918

  [ Upstream Kernel Changes ]

  * Revert "dm: fix AB-BA deadlock in __dm_destroy()"
    - LP: #1522766
  * dm: fix AB-BA deadlock in __dm_destroy()
    - LP: #1522766

linux (3.19.0-40.45) vivid; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1522786

  [ Andy Whitcroft ]

  * [Packaging] control -- prepare for new kernel-wedge semantics
    - LP: #1516686
  * [Debian] rebuild should only trigger for non-linux packages
    - LP: #1498862, #1516686
  * [Tests] gcc-multilib does not exist on ppc64el
    - LP: #1515541

  [ Joseph Salisbury ]

  * SAUCE: scsi_sysfs: protect against double execution of
    __scsi_remove_device()
    - LP: #1509029

  [ Luis Henriques ]

  * [Config] updateconfigs after 3.19.8-ckt10 stable update

  [ Upstream Kernel Changes ]

  * Revert "ARM64: unwind: Fix PC calculation"
    - LP: #1520309
  * Revert "md: allow a partially recovered device to be hot-added to an
    array."
    - LP: #1520309
  * tty: fix stall caused by missing memory barrier in drivers/tty/n_tty.c
    - LP: #1512815
  * HID: rmi: Print the firmware id of the touchpad
    - LP: #1515503
  * HID: rmi: Add functions for writing to registers
    - LP: #1515503
  * HID: rmi: Disable scanning if the device is not a wake source
    - LP: #1515503
  * HID: rmi: Set F01 interrupt enable register when not set
    - LP: #1515503
  * be2net: log link status
    - LP: #1513980
  * xhci: Workaround to get Intel xHCI reset working more reliably
  * Drivers: hv: hv_balloon: refuse to balloon below the floor
    - LP: #1294283
  * Drivers: hv: hv_balloon: survive ballooning request with num_pages=0
    - LP: #1294283
  * Drivers: hv: hv_balloon: correctly handle val.freeram<num_pages case
    - LP: #1294283
  * Drivers: hv: hv_balloon: correctly handle num_pages>INT_MAX case
    - LP: #1294283
  * Drivers: hv: balloon: check if ha_region_mutex was acquired in
    MEM_CANCEL_ONLINE case
    - LP: #1294283
  * mm: meminit: make __early_pfn_to_nid SMP-safe and introduce
    meminit_pfn_in_nid
    - LP: #1294283
  * mm: meminit: inline some helper functions
    - LP: #1294283
  * mm, meminit: allow early_pfn_to_nid to be used during runtime
    - LP: #1294283
  * mm: initialize hotplugged pages as reserved
    - LP: #1294283
  * gut proc_register() a bit
    - LP: #1519106
  * arm: factor out mmap ASLR into mmap_rnd
    - LP: #1518483
  * x86: standardize mmap_rnd() usage
    - LP: #1518483
  * arm64: standardize mmap_rnd() usage
    - LP: #1518483
  * mips: extract logic for mmap_rnd()
    - LP: #1518483
  * powerpc: standardize mmap_rnd() usage
    - LP: #1518483
  * s390: standardize mmap_rnd() usage
    - LP: #1518483
  * mm: expose arch_mmap_rnd when available
    - LP: #1518483
  * s390: redefine randomize_et_dyn for ELF_ET_DYN_BASE
    - LP: #1518483
  * mm: split ET_DYN ASLR from mmap ASLR
    - LP: #1518483
  * mm: fold arch_randomize_brk into ARCH_HAS_ELF_RANDOMIZE
    - LP: #1518483
  * isdn_ppp: Add checks for allocation failure in isdn_ppp_open()
   ...

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-trusty' to 'verification-done-trusty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Tested it with the Trusty proposed kernel 3.13.0-75-generic, ballooning works as expected, no issues detected.

Fix released in 3.13.0-77.121