Large memory guests, "error: monitor socket did not show up: No such file or directory"

Referred change is upstream in Libvirt 3.2, so assuming that is fixed.
But check on X-Z SRUs

Also needs backport of these to work properly
beaa447a2982bc78adb26c183560d0ee566c1268
67dcb797ed7f1fbb048aa47006576f424923933b

Guest has (aslmost) 16G as:
  <memory unit='KiB'>16777216</memory>
Matches
      <cell id='0' cpus='0-15' memory='8388608' unit='KiB' memAccess='shared'/>
      <cell id='1' cpus='16-31' memory='8388608' unit='KiB' memAccess='shared'/>
Alomst fully allocated:
  <currentMemory unit='KiB'>16776960</currentMemory>

The cmdline doesn't seem matchign:
 -m 124928

Anyway ignoring that and trying to reproduce on my side.

Prepared 20x1G hugepages
$ cat /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
20

#Default uvtool created guest but added memory to 19G huge pages:
  <memory unit='GiB'>19</memory>
  <currentMemory unit='GiB'>19</currentMemory>
  <memoryBacking>
          <hugepages>
                  <page size='1048576' unit='KiB' nodeset='0'/>
                  <page size='1048576' unit='KiB' nodeset='1'/>
          </hugepages>
  </memoryBacking>

That took about 1 second, not feeling anything more than on a smaller guest.
It really allocated all the memory from the hugepages:
virsh dominfo test-hp-bug-1705132 | grep memory
Max memory: 19922944 KiB
Used memory: 19922944 KiB
$ cat /sys/kernel/mm/hugepages/hugepages-1048576kB/free_hugepages
1
$ sudo cat /proc/$(pidof qemu-system-x86_64)/numa_maps | grep huge
7fc600000000 default file=/dev/hugepages-1048576/libvirt/qemu/qemu_back_mem.pc.ram.JUsDuG\040(deleted) huge anon=19 dirty=19 N0=19 kernelpagesize_kB=1048576

So far it is "not reproducible" for me, but I checked starting time.
1G 0,022s
10G 1,398s
19G 2,154s
I can see how that
a) scales if you go for lets say 250s
b) might get worse on slow cross node numa setups (I had 10G / s, if one has 2G/s that is *5 duration)

But I found my "-m <size>" arg always match my memory spec.
Remember that I pointed out in comment #3 that you had 124G in the -m but 16G in the XML you listed.

I prepared a ppa [2] for you to test backporting what was identified plus a few deps.
I would like to ask you for the following:

1. Please ensure that your -m <size> matches your XML and find why not
2. Please could you start guests with increasing sizes and report:
   2.1 the time a virsh start takes
   2.2 if it works or not.
   Sizes would be 1,10,20,40,80,...<as high as you can go>
3. Same test as #2 but with the ppa I linked
4. if there is any chance test #2 with Artful as well which has the fixes already included and confirming fixed there is a SRU prereq.

Status "incomplete" until that feedback is provided to think about SRU feasibility.

Note: Related Bileto ticket [1] and test builds to verify in [2]

[1]: https://bileto.ubuntu.com/#/ticket/2872
[2] https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2872

root@buneary:/home/ubuntu# time virsh start reproducer2
error: Failed to start domain reproducer2
error: monitor socket did not show up: No such file or directory

real 0m30.542s
user 0m0.016s
sys 0m0.004s

root@buneary:/home/ubuntu# virsh dominfo reproducer2
Id: -
Name: reproducer2
UUID: 2eaaa235-4bfc-4ed2-b551-915c0ccffd44
OS Type: hvm
State: shut off
CPU(s): 32
Max memory: 127926272 KiB
Used memory: 125829120 KiB
Persistent: yes
Autostart: disable
Managed save: no
Security model: apparmor
Security DOI: 0

Here is the guest definition: http://paste.ubuntu.com/25125500/

root@buneary:/home/ubuntu# virsh freepages 0 1G
1048576KiB: 60

root@buneary:/home/ubuntu# virsh freepages 1 1G
1048576KiB: 62

root@buneary:/home/ubuntu# numastat -m

Per-node system memory usage (in MBs):
                          Node 0 Node 1 Total
                 --------------- --------------- ---------------
MemTotal 64300.79 64507.32 128808.11
MemFree 1980.95 544.31 2525.26
MemUsed 62319.84 63963.01 126282.84
Active 325.93 78.04 403.98
Inactive 68.02 31.72 99.74
Active(anon) 46.50 19.14 65.64
Inactive(anon) 8.10 0.07 8.16
Active(file) 279.43 58.91 338.34
Inactive(file) 59.92 31.66 91.57
Unevictable 0.63 2.95 3.57
Mlocked 0.63 2.95 3.57
Dirty 0.00 0.00 0.00
Writeback 0.00 0.00 0.00
FilePages 348.50 93.38 441.87
Mapped 40.48 17.09 57.57
AnonPages 46.14 19.34 65.48
Shmem 8.61 0.95 9.56
KernelStack 5.28 4.41 9.69
PageTables 3.39 1.41 4.79
NFS_Unstable 0.00 0.00 0.00
Bounce 0.00 0.00 0.00
WritebackTmp 0.00 0.00 0.00
Slab 102.35 78.05 180.40
SReclaimable 44.11 29.07 73.19
SUnreclaim 58.24 48.97 107.21
AnonHugePages 20.00 4.00 24.00
HugePages_Total 61440.00 63488.00 124928.00
HugePages_Free 61440.00 63488.00 124928.00
HugePages_Surp 0.00 0.00 0.00

Testing your PPA now.

Christian,

The error isn't reproducible after installing the libvirt packages
from the PPA you provided. It took around ~42 seconds for the start to succeed.

root@buneary:/home/ubuntu# dpkg -l|grep libvirt
ii libvirt-bin 1.3.1-1ubuntu10.12~ppa1 amd64 programs for the libvirt library
ii libvirt0:amd64 1.3.1-1ubuntu10.12~ppa1 amd64 library for interfacing with different virtualization systems
ii libvirt0-dbg 1.3.1-1ubuntu10.12~ppa1 amd64 library for interfacing with different virtualization systems - debugging symbols

root@buneary:/home/ubuntu# time virsh start reproducer2
Domain reproducer2 started

real 0m42.134s
user 0m0.016s
sys 0m0.008s

It really allocated all the available huge pages on both nodes (0, 1).

root@buneary:/home/ubuntu# sudo cat /proc/$(pidof qemu-system-x86_64)/numa_maps | grep huge
7f5500000000 bind:1 file=/dev/hugepages/libvirt/qemu/qemu_back_mem._objects_ram-node1.DBggwd\040(deleted) huge dirty=62 N1=62 kernelpagesize_kB=1048576

7f64c0000000 bind:0 file=/dev/hugepages/libvirt/qemu/qemu_back_mem._objects_ram-node0.AmusgD\040(deleted) huge dirty=60 N0=60 kernelpagesize_kB=1048576

root@buneary:/home/ubuntu# cat /sys/kernel/mm/hugepages/hugepages-1048576kB/free_hugepages
0

Ok, verified fix.
Fix looks good in preliminary regression tests.
Changes are backports from upstream.

Setting to work in progress, yet I'm not sure how far we get before I'm away a bit.

Trying to clear the current ongoing SRUs.

Tested X and Z backports in ppa, working for me

Thanks for the initial SRU template btw, I extended it a bit but thanks to help out on not letting me write it alone niedbalski.

Uploading to unapproved now for SRU.

Note to the SRU Team: I know there are currently a SRUs in -proposed that are blocking the accept from -unapproved to -proposed.
Those are:
 xenial 1.3.1-1ubuntu10.11
 zesty 2.5.0-3ubuntu5.3

But they will very likely migrate next week
  - all things look good, (tests, bug, verification)
  - also look good on the pending SRU page
  - they just wait to have more days in -proposed.

So I prepared the uploads for this fix here to be in -proposed as soon as possible.
Which means after you accepted the former into -updates, please consider accepting the SRU for this bug into -proposed.

Raising UCA tasks for trusty/mitaka and xenial/ocata (xenial/newton does not include libvirt)

Hello Jorge, or anyone else affected,

Accepted libvirt into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/2.5.0-3ubuntu5.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Jorge, or anyone else affected,

Accepted libvirt into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/1.3.1-1ubuntu10.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello,

I can confirm that the current -proposed package fixes the issue for Xenial.

root@buneary:/home/ubuntu# dpkg -l | grep libvirt
ii libvirt-bin 1.3.1-1ubuntu10.12 amd64 programs for the libvirt library
ii libvirt0:amd64 1.3.1-1ubuntu10.12 amd64 library for interfacing with different virtualization systems
ii libvirt0-dbg 1.3.1-1ubuntu10.12 amd64 library for interfacing with different virtualization systems - debugging symbols
ii python-libvirt 1.3.1-1ubuntu1 amd64 libvirt Python
 bindings
ii uvtool-libvirt 0~bzr99-0ubuntu1 all Library and tools for using Ubuntu Cloud Images with libvirt

root@buneary:/home/ubuntu# time virsh start reproducer2
Domain reproducer2 started

real 0m35.450s
user 0m0.020s
sys 0m0.008s

It allocated all the available huge pages on both nodes (0, 1).

root@buneary:/home/ubuntu# sudo cat /proc/$(pidof qemu-system-x86_64)/numa_maps | grep huge
7f5500000000 bind:1 file=/dev/hugepages/libvirt/qemu/qemu_back_mem._objects_ram-node1.DBggwd\040(deleted) huge dirty=62 N1=62 kernelpagesize_kB=1048576

7f64c0000000 bind:0 file=/dev/hugepages/libvirt/qemu/qemu_back_mem._objects_ram-node0.AmusgD\040(deleted) huge dirty=60 N0=60 kernelpagesize_kB=1048576

root@buneary:/home/ubuntu# cat /sys/kernel/mm/hugepages/hugepages-1048576kB/free_hugepages
0

Marking the verification for xenial as done.

Hello Jorge, or anyone else affected,

Accepted libvirt into ocata-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ocata-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ocata-needed to verification-ocata-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ocata-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Jorge, or anyone else affected,

Accepted libvirt into mitaka-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:mitaka-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-mitaka-needed to verification-mitaka-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-mitaka-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This bug was fixed in the package libvirt - 1.3.1-1ubuntu10.12

---------------
libvirt (1.3.1-1ubuntu10.12) xenial; urgency=medium

  * d/p/ubuntu/bug-1705132-* qemu: Adaptive timeout for connecting to
    monitor (LP: #1705132)
    - includes backports that make backing off on timeouts exponentially
      but cap the exponential increase on 1s.

 -- Christian Ehrhardt <email address hidden> Wed, 19 Jul 2017 08:28:14 +0200

The verification of the Stable Release Update for libvirt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Hello,

I can confirm that the current -proposed package fixes the issue for Mitaka Cloud - Archive.
libvirt

root@buneary:/home/ubuntu# time virsh start reproducer2
Domain reproducer2 started

real 0m45.420s
user 0m0.015s
sys 0m0.012s

It allocated all the available huge pages on both nodes (0, 1).

root@buneary:/home/ubuntu# sudo cat /proc/$(pidof qemu-system-x86_64)/numa_maps | grep huge
78ff00000000 bind:1 file=/dev/hugepages/libvirt/qemu/qemu_back_mem._objects_ram-node1.DBggwd\040(deleted) huge dirty=62 N1=62 kernelpagesize_kB=1048576

78d4c0000000 bind:0 file=/dev/hugepages/libvirt/qemu/qemu_back_mem._objects_ram-node0.AmusgD\040(deleted) huge dirty=60 N0=60 kernelpagesize_kB=1048576

root@buneary:/home/ubuntu# cat /sys/kernel/mm/hugepages/hugepages-1048576kB/free_hugepages
0

Marking the verification for mitaka as done.

Hello,

I can confirm that the current -proposed package fixes the issue for Ocata Cloud - Archive.
libvirt and Zesty.

root@buneary-zesty:/home/ubuntu# time virsh start reproducer2
Domain reproducer2 started

real 0m38.310s
user 0m0.010s
sys 0m0.011s

It allocated all the available huge pages on both nodes (0, 1).

root@buneary:/home/ubuntu# sudo cat /proc/$(pidof qemu-system-x86_64)/numa_maps | grep huge
78ff00000000 bind:1 file=/dev/hugepages/libvirt/qemu/qemu_back_mem._objects_ram-node1.DBggwd\040(deleted) huge dirty=62 N1=62 kernelpagesize_kB=1048576

78d4c0000000 bind:0 file=/dev/hugepages/libvirt/qemu/qemu_back_mem._objects_ram-node0.AmusgD\040(deleted) huge dirty=60 N0=60 kernelpagesize_kB=1048576

root@buneary:/home/ubuntu# cat /sys/kernel/mm/hugepages/hugepages-1048576kB/free_hugepages
0

This bug was fixed in the package libvirt - 2.5.0-3ubuntu5.4

---------------
libvirt (2.5.0-3ubuntu5.4) zesty; urgency=medium

  * d/p/ubuntu/bug-1705132-* qemu: Adaptive timeout for connecting to
    monitor (LP: #1705132)
    - includes backports that make backing off on timeouts exponentially
      but cap the exponential increase on 1s.

 -- Christian Ehrhardt <email address hidden> Thu, 20 Jul 2017 13:06:02 +0200

There was a gating process kicking in due to [1].

I checked the code and didn't see it related to the SRU.

Also the issue occurred in the past on other versions which is another indicator to not be tied to the SRU.

To be sure SMB (Xen maintainer) tested the new version of libvirt and succeeded without running into issues.

Furthermore the stack shows _xend which should no more be around since >Xenial. Mabye that was run with a unsupported Xen version?

In any case @~ubuntu-sru (now subscribed) please proceed with the SRU phasing.
*will also ping on IRC*

[1]: https://errors.ubuntu.com/?release=Ubuntu%2017.04&package=libvirt&period=day&version=2.5.0-3ubuntu5.4

just for clarity i'm nominating for newton and it will be marked as invalid since there is no libvirt in newton uca

The verification of the Stable Release Update for libvirt has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package libvirt - 1.3.1-1ubuntu10.12~cloud0
---------------

 libvirt (1.3.1-1ubuntu10.12~cloud0) trusty-mitaka; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 libvirt (1.3.1-1ubuntu10.12) xenial; urgency=medium
 .
   * d/p/ubuntu/bug-1705132-* qemu: Adaptive timeout for connecting to
     monitor (LP: #1705132)
     - includes backports that make backing off on timeouts exponentially
       but cap the exponential increase on 1s.

The verification of the Stable Release Update for libvirt has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package libvirt - 2.5.0-3ubuntu5.4~cloud0
---------------

 libvirt (2.5.0-3ubuntu5.4~cloud0) xenial-ocata; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 libvirt (2.5.0-3ubuntu5.4) zesty; urgency=medium
 .
   * d/p/ubuntu/bug-1705132-* qemu: Adaptive timeout for connecting to
     monitor (LP: #1705132)
     - includes backports that make backing off on timeouts exponentially
       but cap the exponential increase on 1s.