'unregister_netdevice: waiting for lo to become free. Usage count = 2' is reported and causing kernel hang when floodlight tests are run using utah

Bug #1179509 reported by Para Siva
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

When utah client is used to run floodlight smoke tests for saucy server images, the following error is reported.

May 13 03:51:09 utah-8671-saucy-server-i386 kernel: [ 154.327421] IPv6: ADDRCONF(NETDEV_UP): s1-eth1: link is not ready
May 13 03:51:19 utah-8671-saucy-server-i386 kernel: [ 164.100031] unregister_netdevice: waiting for lo to become free. Usage count = 2
...
...
May 13 03:54:33 utah-8671-saucy-server-i386 kernel: [ 358.452029] unregister_netdevice: waiting for lo to become free. Usage count = 2
May 13 03:54:35 utah-8671-saucy-server-i386 kernel: [ 360.636063] INFO: task mnexec:7033 blocked for more than 120 seconds.
May 13 03:54:35 utah-8671-saucy-server-i386 kernel: [ 360.636071] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 13 03:54:35 utah-8671-saucy-server-i386 kernel: [ 360.636076] mnexec D 000000f2 0 7033 6369 0x00000000
May 13 03:54:35 utah-8671-saucy-server-i386 kernel: [ 360.636081] de407f00 00000082 dffe7c20 000000f2 d1655300 0016eb12 00000000 c19cc3c0
May 13 03:54:35 utah-8671-saucy-server-i386 kernel: [ 360.636086] 00000000 00000023 c19cc3c0 dfbf23c0 dee42670 dfc56240 d1655300 00000000
May 13 03:54:35 utah-8671-saucy-server-i386 kernel: [ 360.636089] ded99540 00000000 b7666550 d1655300 02d12067 c2c7edd8 d1655300 de407f14
May 13 03:54:35 utah-8671-saucy-server-i386 kernel: [ 360.636092] Call Trace:
May 13 03:54:35 utah-8671-saucy-server-i386 kernel: [ 360.636113] [<c112e69a>] ? handle_mm_fault+0x22a/0x310
May 13 03:54:35 utah-8671-saucy-server-i386 kernel: [ 360.636126] [<c160efc3>] schedule_preempt_disabled+0x23/0x60
May 13 03:54:35 utah-8671-saucy-server-i386 kernel: [ 360.636129] [<c160d553>] __mutex_lock_slowpath+0xb3/0x110
May 13 03:54:35 utah-8671-saucy-server-i386 kernel: [ 360.636131] [<c160d10c>] mutex_lock+0x1c/0x30
May 13 03:54:35 utah-8671-saucy-server-i386 kernel: [ 360.636144] [<c1525944>] copy_net_ns+0x54/0xd0
May 13 03:54:35 utah-8671-saucy-server-i386 kernel: [ 360.636153] [<c106ec81>] create_new_namespaces+0xd1/0x150
May 13 03:54:35 utah-8671-saucy-server-i386 kernel: [ 360.636156] [<c106ee7e>] unshare_nsproxy_namespaces+0x4e/0x90
May 13 03:54:35 utah-8671-saucy-server-i386 kernel: [ 360.636164] [<c1049697>] sys_unshare+0x127/0x230
May 13 03:54:35 utah-8671-saucy-server-i386 kernel: [ 360.636170] [<c1612ba0>] ? arch_prepare_kprobe_ftrace+0x20/0x20
May 13 03:54:35 utah-8671-saucy-server-i386 kernel: [ 360.636174] [<c1616b4d>] sysenter_do_call+0x12/0x28
...
================================================
This is observed to occur only when utah is used to run the floodlight smoke tests. Running the floodlight tests without utah does not cause this issue and using utah to run some other tests does not cause this issue either.

Noticed a similar bug 1021471, but this one is occurring in KVM using libvirt.

The issue is only with the saucy kernel version, 3.9.0-1 and does not occur with 3.9.0-0. The tests were tried with saucy server VMs on precise as well as saucy hosts and the results is the same.

Steps to reproduce:
1. Install a default saucy server on a vm (either manually or preseeded default installation) with either i386 or amd64 image using KVM, libvirt
2. Do the following to install utah inside the VM
   sudo apt-add-repository -y ppa:utah/stable
   sudo apt-get update
   sudo apt-get install utah
3. Run the floodlight tests using
     sudo utah -r lp:ubuntu-test-cases/server/runlists/floodlight.run
4. Now "kernel: [ 358.452029] unregister_netdevice: waiting for lo to become free. Usage count = 2" will be continuously reported

Install and boot time syslogs are attached.

The impacted jobs are,
https://jenkins.qa.ubuntu.com/view/Saucy/view/Smoke%20Testing/job/saucy-server-amd64-smoke-floodlight/15/
and
https://jenkins.qa.ubuntu.com/view/Saucy/view/Smoke%20Testing/job/saucy-server-i386-smoke-floodlight/13/

Revision history for this message
Para Siva (psivaa) wrote :
description: updated
Revision history for this message
Para Siva (psivaa) wrote :
tags: added: iso-testing qa-daily-testing rls-s-incoming
Para Siva (psivaa)
Changed in utah:
importance: Undecided → High
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Could you please run apport-collect 1179509 from the system after this happens?

This sounds like a regression - there was a similar bug around 3.5 or so, but that was fixed upstream.

Revision history for this message
Brad Figg (brad-figg) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1179509

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Para Siva (psivaa) wrote :
Revision history for this message
Para Siva (psivaa) wrote :

Apologies that I am having to attach the logs as a single tar ball. apport-collect 1179509 doesn't seem to collect the relevant logs for me from the VM. (it simply asks for lp credentials and leaves me in my lp home page)

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Para Siva (psivaa)
description: updated
Para Siva (psivaa)
no longer affects: utah
Revision history for this message
Para Siva (psivaa) wrote :

Reported another bug ( bug 1181315) for the same issue with ubuntu-bug linux, and marking this as a duplicate of the new one.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.