No easy way to use hugetlbfs with QEMU

Bug #1315402 reported by Simon Déziel
22
This bug affects 4 people
Affects Status Importance Assigned to Milestone
qemu (Ubuntu)
Fix Released
High
Unassigned
Trusty
Fix Released
High
Unassigned

Bug Description

=====================================
Impact: ksm not enabled, users cannot easily use hugepages, and on older kernels the kvm_{amd,intel}.ko won't get mounted
Test case: cat /sys/kernel/mm/ksm/run
Regression potential: this upstart job was always installed up through near the end of trusty cycle, so there should be no regression potential
=====================================

In previous releases (tested with Saucy), if one wanted to use hugetlbfs for a VM, the needed procedure was:

1) Add "vm.nr_hugepages" to /etc/sysctl.d/60-hugepages.conf
2) start procps
3) Set KVM_HUGEPAGES=1 in /etc/default/qemu-kvm
4) restart qemu-kvm
5) Enable "<memoryBacking> <hugepages/></memoryBacking>" in the libvirt guest definition
6) Restart the guest

Now with Trusty, this doesn't work anymore as there is no (obvious?) way to have the hugetlbfs mounted where QEMU expects it.

On Saucy, the upstart job for qemu-kvm contains this:

 # mount hugepages if available and requested
 if [ "$KVM_HUGEPAGES" -eq "1" ]; then
  if ! grep -q hugetlbfs /proc/filesystems; then
   logger -t qemu-kvm "Error: hugepages not available in the kernel!"
  elif grep -q /run/hugepages/kvm /proc/mounts; then
   logger -t qemu-kvm "/run/hugepages/kvm already mounted"
  elif ! grep -q kvm /etc/group; then
   logger -t qemu-kvm "Error: group kvm does not exist!"
  else
   kvmgid=`grep kvm /etc/group | cut -d: -f 3`
   mkdir -p /run/hugepages/kvm
   mount -t hugetlbfs hugetlbfs-kvm -o mode=775,gid=$kvmgid /run/hugepages/kvm
  fi
 fi

But it seems there is nothing similar in Trusty and manually mounting the hugetlbfs doesn't help as qemu cannot find it there.

Host information:

$ lsb_release -rd
Description: Ubuntu 14.04 LTS
Release: 14.04
$ apt-cache policy qemu-system-x86 libvirt-bin
qemu-system-x86:
  Installed: 2.0.0+dfsg-2ubuntu1
  Candidate: 2.0.0+dfsg-2ubuntu1
  Version table:
 *** 2.0.0+dfsg-2ubuntu1 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2.0.0~rc1+dfsg-0ubuntu3.1 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
     2.0.0~rc1+dfsg-0ubuntu3 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
libvirt-bin:
  Installed: 1.2.2-0ubuntu13.1
  Candidate: 1.2.2-0ubuntu13.1
  Version table:
 *** 1.2.2-0ubuntu13.1 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.2.2-0ubuntu13 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: qemu-system-common 2.0.0+dfsg-2ubuntu1
ProcVersionSignature: Ubuntu 3.13.0-24.46-generic 3.13.9
Uname: Linux 3.13.0-24-generic x86_64
ApportVersion: 2.14.1-0ubuntu3
Architecture: amd64
CurrentDesktop: Unity
Date: Fri May 2 10:05:18 2014
InstallationDate: Installed on 2014-01-26 (95 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Alpha amd64 (20140124)
SourcePackage: qemu
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Simon Déziel (sdeziel) wrote :
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Hi,

could you show the contents of your /etc/init/qemu-kvm.conf? The package still provides the code you seem to be referring to, and on my laptop /run/hugepages/kvm is still mounted, so I'm curious what is going on on your system.

Changed in qemu (Ubuntu):
status: New → Incomplete
Revision history for this message
Simon Déziel (sdeziel) wrote : Re: [Bug 1315402] Re: No easy way to use hugetlbfs with QEMU

Hi Serge,

On 14-05-02 10:48 AM, Serge Hallyn wrote:
> could you show the contents of your /etc/init/qemu-kvm.conf? The
> package still provides the code you seem to be referring to, and on my
> laptop /run/hugepages/kvm is still mounted, so I'm curious what is going
> on on your system.

In fact, the qemu-kvm (transitional) package was missing on my system.
Installing it didn't help though as it doesn't provide
/etc/init/qemu-kvm.conf

$ find /etc/init -name '*qemu*.conf'
  # nothing

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Quoting Simon Déziel (<email address hidden>):
> Hi Serge,
>
> On 14-05-02 10:48 AM, Serge Hallyn wrote:
> > could you show the contents of your /etc/init/qemu-kvm.conf? The
> > package still provides the code you seem to be referring to, and on my
> > laptop /run/hugepages/kvm is still mounted, so I'm curious what is going
> > on on your system.
>
> In fact, the qemu-kvm (transitional) package was missing on my system.

The qemu-kvm.conf file is provided by qemu-system-x86 (the old name
was kept to ease transitions)

Revision history for this message
Simon Déziel (sdeziel) wrote :

On 14-05-02 12:04 PM, Serge Hallyn wrote:
> Quoting Simon Déziel (<email address hidden>):
>> Hi Serge,
>>
>> On 14-05-02 10:48 AM, Serge Hallyn wrote:
>>> could you show the contents of your /etc/init/qemu-kvm.conf? The
>>> package still provides the code you seem to be referring to, and on my
>>> laptop /run/hugepages/kvm is still mounted, so I'm curious what is going
>>> on on your system.
>>
>> In fact, the qemu-kvm (transitional) package was missing on my system.
>
> The qemu-kvm.conf file is provided by qemu-system-x86 (the old name
> was kept to ease transitions)

Understood but this package doesn't provide the upstart job you have.
May I ask to which package provided the file you have?

Maybe the -proposed package accidentally dropped the upstart job?

Some more info:

$ dpkg -L qemu-system-x86 | grep init
/etc/init.d
/etc/init.d/qemu-system-x86

$ apt-cache policy qemu-system-x86
qemu-system-x86:
  Installed: 2.0.0+dfsg-2ubuntu1
  Candidate: 2.0.0+dfsg-2ubuntu1
  Version table:
 *** 2.0.0+dfsg-2ubuntu1 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-proposed/main amd64
Packages
        100 /var/lib/dpkg/status
     2.0.0~rc1+dfsg-0ubuntu3.1 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64
Packages
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main
amd64 Packages
     2.0.0~rc1+dfsg-0ubuntu3 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

D'oh, right you are. debian/rules now has the wrong dh_installinit line. Thanks!

Changed in qemu (Ubuntu):
status: Incomplete → In Progress
importance: Undecided → High
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Pushed a fix to utopic. Since there is a package in trusty-proposed, we'll have to wait for that to clear before fixing this in trusty.

Changed in qemu (Ubuntu Trusty):
importance: Undecided → High
status: New → Confirmed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu - 2.0.0+dfsg-2ubuntu2

---------------
qemu (2.0.0+dfsg-2ubuntu2) utopic; urgency=medium

  * debian/rules: install the proper /etc/init/qemu-kvm.conf (LP: #1315402)
  * debian/control: drop the versioning requirement from libfdt-dev
    build-dependency, as it is longer needed (LP: #1295072)
 -- Serge Hallyn <email address hidden> Fri, 02 May 2014 11:43:44 -0500

Changed in qemu (Ubuntu):
status: In Progress → Fix Released
description: updated
Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello Simon, or anyone else affected,

Accepted qemu into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/qemu/2.0.0+dfsg-2ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in qemu (Ubuntu Trusty):
status: Confirmed → Fix Committed
tags: added: verification-needed
Revision history for this message
Simon Déziel (sdeziel) wrote :

Thanks Serge!

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu - 2.0.0+dfsg-2ubuntu1.1

---------------
qemu (2.0.0+dfsg-2ubuntu1.1) trusty-proposed; urgency=low

  * remove alternatives for qemu: different architectures
    aren't really alternatives and never had been (LP: #1316829)
  * debian/rules: install the proper /etc/init/qemu-kvm.conf (LP: #1315402)
  * debian/control: drop the versioning requirement from libfdt-dev
    build-dependency, as it is longer needed (LP: #1295072)
 -- Serge Hallyn <email address hidden> Wed, 07 May 2014 17:31:39 -0500

Changed in qemu (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Scott Kitterman (kitterman) wrote : Update Released

The verification of the Stable Release Update for qemu has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message
Dominique Ramaekers (dominique-ramaekers) wrote :

On 15.04 upstart isn't standard anymore...

Now the qemu-kvm.conf upstart script isn't run.

Shouldn't the upstart script be converted to a regular systemd script?

Right now, the hugepages functionality isn't working anymore on Ubuntu 15.04...

Revision history for this message
Serge Hallyn (serge-hallyn) wrote : Re: [Bug 1315402] Re: No easy way to use hugetlbfs with QEMU

Good point I will add a systemd unit while merging the new version into wily.
Then I'll SRU it into vivid. Would you mind opening a new bug against
qemu for the missing systemd job?

Revision history for this message
David Girault (dfgweb) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.