Ubuntu
netcf package

excessive memory use from libvirtd

Bug #1201938 reported by Robert Collins
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
netcf (Ubuntu)
Fix Released
High
Unassigned
Precise
Fix Released
High
Unassigned
Quantal
Fix Released
High
Unassigned
Raring
Fix Released
High
Unassigned

Bug Description

==================
SRU Justification
1. Impact: libvirt over time consumes excessive memory due to a call to libnetcf
2. Development fix: a patch to free a path which was being leaked
3. Stable fix: same as development fix
4. Test case: start libvirtd, and connect to it with virt-manager. Leave that running. Run top in a terminal, hit 'M' to sort by memory, and watch for RSS for libvirtd to grow. Give it an hour or two to grow by a few M.
5. Regression potential: there should be none as this only causes a path to be freed (after an earlier check for failure which should ensure the path is in fact allocated).
==================
libvirtd is using excessive amounts of memory after a week or so of uptime:
uptime
 07:47:15 up 9 days, 21:49, 16 users, load average: 1.06, 2.20, 1.70

from top (sorry for the truncated line, window got resized before I copied it out.
18978 root 20 0 7334m 6.6g 672 S 1.0 42.3 504

ProblemType: Bug
DistroRelease: Ubuntu 13.04
Package: libvirt-bin 1.0.2-0ubuntu11.13.04.2
ProcVersionSignature: Ubuntu 3.8.0-23.34-generic 3.8.11
Uname: Linux 3.8.0-23-generic x86_64
ApportVersion: 2.9.2-0ubuntu8
Architecture: amd64
Date: Wed Jul 17 07:44:41 2013
InstallationDate: Installed on 2013-01-13 (184 days ago)
InstallationMedia: Ubuntu-Server 12.10 "Quantal Quetzal" - Release amd64 (20121017.2)
MarkForUpload: True
SourcePackage: libvirt
UpgradeStatus: Upgraded to raring on 2013-04-22 (85 days ago)
---
ApportVersion: 2.9.2-0ubuntu8
Architecture: amd64
DistroRelease: Ubuntu 13.04
InstallationDate: Installed on 2013-01-13 (184 days ago)
InstallationMedia: Ubuntu-Server 12.10 "Quantal Quetzal" - Release amd64 (20121017.2)
KernLog:

MarkForUpload: True
Package: libvirt (not installed)
ProcCmdline: BOOT_IMAGE=/vmlinuz-3.8.0-23-generic root=/dev/mapper/hostname-root ro rootflags=subvol=@
ProcVersionSignature: Ubuntu 3.8.0-23.34-generic 3.8.11
Tags: raring
Uname: Linux 3.8.0-23-generic x86_64
UpgradeStatus: Upgraded to raring on 2013-04-22 (85 days ago)
UserGroups: adm cdrom dip libvirtd lpadmin plugdev sambashare sudo
modified.conffile..etc.apparmor.d.usr.lib.libvirt.virt.aa.helper: [modified]
modified.conffile..etc.apparmor.d.usr.sbin.libvirtd: [modified]
mtime.conffile..etc.apparmor.d.usr.lib.libvirt.virt.aa.helper: 2013-06-26T00:59:03.272193
mtime.conffile..etc.apparmor.d.usr.sbin.libvirtd: 2013-06-26T00:58:56.616193

See original description
Revision history for this message
Robert Collins (lifeless) wrote :
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in libvirt (Ubuntu):
status: New → Confirmed
Revision history for this message
Robert Collins (lifeless) wrote :

To reproduce - run something like virt-manager and a couple of vms for an extended period (days). Eventually libvirtd will use a lot of ram and then if you push it long enough get OOM killed.

Serge Hallyn (serge-hallyn)
Changed in libvirt (Ubuntu):
importance: Undecided → High
Revision history for this message
Robert Collins (lifeless) wrote : HookError_cloud_archive.txt

apport information

tags: added: apport-collected
description: updated
Revision history for this message
Robert Collins (lifeless) wrote : ProcEnviron.txt

apport information

Revision history for this message
Robert Collins (lifeless) wrote : RelatedPackageVersions.txt

apport information

Revision history for this message
Robert Collins (lifeless) wrote :

Not using rbd at all.

We're using libvirt to host the tripleo dev/test story (rather than requiring developers to have multiple physical machines).

Revision history for this message
Robert Collins (lifeless) wrote :

Running vms:
$ virsh list
 Id Name State
----------------------------------------------------
 1 seed running
 6 baremetal_2 running

I have virt-manager connected showing both cpu usage and host cpu usage.

With configuration:
$ virsh dumpxml seed | pastebinit
http://paste.ubuntu.com/5885447/
obertc@lifelesshp:~$ virsh dumpxml baremetal_2 | pastebinit
http://paste.ubuntu.com/5885449/

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

https://github.com/tripleo/incubator/blob/master/devtest.md

shows the relevant workflow

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

I can reproduce it - albeit slowly - even without any vms running.

I'll try with saucy next, then try to narrow it down to a few commits.

Changed in libvirt (Ubuntu):
status: Confirmed → Triaged
assignee: nobody → Serge Hallyn (serge-hallyn)
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Reproduced in saucy (1.0.6 libvirt) as well.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

I had previously reproduced this with a ppa build from an hourly upstream snapshot. I reported this upstream at https://www.redhat.com/archives/libvir-list/2013-July/msg01751.html

Today I can't reproduce this even in stock saucy.

libvirt hasn't been updated in the archive in the meantime. Could this have been a bug in one of the dependencies? Or am I missing a step? I have 2 vms running while virt-manager is running (locally) displaying guest and host cpu times.

Also was unable to reproduce this on a fedora 19 rackspace image.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Ok, very slow leak appears to still be in saucy. will re-try with a fresh fedora install (local so i'm not in a hurry :)

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

The leak was in libnetcf. Eric Blake has pushed a patch upstream to fix it. I will push packages today and tomorrow with the fix.

Changed in libvirt (Ubuntu):
status: Triaged → In Progress
Serge Hallyn (serge-hallyn)
Changed in netcf (Ubuntu):
importance: Undecided → High
status: New → In Progress
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package netcf - 1:0.2.3-3ubuntu1

---------------
netcf (1:0.2.3-3ubuntu1) saucy; urgency=low

  * netcf-debian-memleak.patch: prevent a memory leak when listing
    interfaces (LP: #1201938)
 -- Serge Hallyn <email address hidden> Wed, 14 Aug 2013 00:29:13 +0000

Changed in netcf (Ubuntu):
status: In Progress → Fix Released
Serge Hallyn (serge-hallyn)
no longer affects: libvirt (Ubuntu Precise)
no longer affects: libvirt (Ubuntu Quantal)
no longer affects: libvirt (Ubuntu)
no longer affects: libvirt (Ubuntu Raring)
description: updated
Serge Hallyn (serge-hallyn)
Changed in netcf (Ubuntu Precise):
status: New → Triaged
Changed in netcf (Ubuntu Quantal):
status: New → Triaged
Changed in netcf (Ubuntu Raring):
status: New → Triaged
Changed in netcf (Ubuntu Precise):
importance: Undecided → High
Changed in netcf (Ubuntu Quantal):
importance: Undecided → High
Changed in netcf (Ubuntu Raring):
importance: Undecided → High
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Note the raring debdiff (which I don't have perms to push) is at http://people.canonical.com/~serge/netcf-raring.debdiff.

Revision history for this message
Robert Collins (lifeless) wrote : Re: [Bug 1201938] Re: excessive memory use from libvirtd

This is much better for me, but I have seen an increase of 1M over 5
hours so far - so there may be another lower-frequency leak. I will
file a new bug if I find that to be the case.

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Robert, or anyone else affected,

Accepted netcf into raring-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/netcf/1:0.2.3-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in netcf (Ubuntu Raring):
status: Triaged → Fix Committed
tags: added: verification-needed
Changed in netcf (Ubuntu Quantal):
status: Triaged → Fix Committed
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Robert, or anyone else affected,

Accepted netcf into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/netcf/0.2.0-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in netcf (Ubuntu Precise):
status: Triaged → Fix Committed
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Robert, or anyone else affected,

Accepted netcf into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/netcf/0.1.9-2ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Robert Collins (lifeless) wrote :

Stable memory use after three days. Thanks!

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Confirmed for raring.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Verified in quantal.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Actually the precise libvirt isn't linked against netcf (as netcf isn't in main there).

confirmed that 'ncftool list' in precise still works.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package netcf - 0.1.9-2ubuntu3.1

---------------
netcf (0.1.9-2ubuntu3.1) precise-proposed; urgency=low

  * netcf-debian-memleak.patch: prevent a memory leak when listing
    interfaces (LP: #1201938)
 -- Serge Hallyn <email address hidden> Wed, 14 Aug 2013 15:32:11 +0000

Changed in netcf (Ubuntu Precise):
status: Fix Committed → Fix Released
Revision history for this message
Colin Watson (cjwatson) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package netcf - 0.2.0-1ubuntu1.1

---------------
netcf (0.2.0-1ubuntu1.1) quantal-proposed; urgency=low

  * netcf-debian-memleak.patch: prevent a memory leak when listing
    interfaces (LP: #1201938)
 -- Serge Hallyn <email address hidden> Wed, 14 Aug 2013 10:14:50 -0500

Changed in netcf (Ubuntu Quantal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package netcf - 1:0.2.3-1ubuntu0.1

---------------
netcf (1:0.2.3-1ubuntu0.1) raring-proposed; urgency=low

  * netcf-debian-memleak.patch: prevent a memory leak when listing
    interfaces (LP: #1201938)
 -- Serge Hallyn <email address hidden> Tue, 13 Aug 2013 22:19:14 -0500

Changed in netcf (Ubuntu Raring):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.