nova-compute exception : Timeout migrating for instance-00000009. nwfilter not found

Bug #1039398 reported by Naveen Joy
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
Undecided
Unassigned

Bug Description

I am using the latest version of nova cloned from git hub. Seeing the below exception shortly after starting nova-compute

sudo nova-compute
2012-08-21 00:03:15 INFO nova.compute.manager [-] Loading compute driver 'libvirt.LibvirtDriver'
Loading compute driver 'libvirt.LibvirtDriver'
2012-08-21 00:03:15 AUDIT nova.service [-] Starting compute node (version 2012.2-LOCALBRANCH:LOCALREVISION)
Starting compute node (version 2012.2-LOCALBRANCH:LOCALREVISION)
2012-08-21 00:03:16 WARNING nova.common.deprecated [req-f287d489-96cc-4e0a-aa58-ef0197f24072 None None] Deprecated Config: The root_helper option (which lets you specify a root wrapper different from nova-rootwrap, and defaults to using sudo) is now deprecated. You should use the rootwrap_config option instead.
Deprecated Config: The root_helper option (which lets you specify a root wrapper different from nova-rootwrap, and defaults to using sudo) is now deprecated. You should use the rootwrap_config option instead.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 336, in fire_timers
    timer()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 56, in __call__
    cb(*args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
    result = function(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/nova-2012.2-py2.7.egg/nova/service.py", line 132, in run_server
    server.start()
  File "/usr/local/lib/python2.7/dist-packages/nova-2012.2-py2.7.egg/nova/service.py", line 398, in start
    self.manager.init_host()
  File "/usr/local/lib/python2.7/dist-packages/nova-2012.2-py2.7.egg/nova/compute/manager.py", line 352, in init_host
    self._legacy_nw_info(net_info))
  File "/usr/local/lib/python2.7/dist-packages/nova-2012.2-py2.7.egg/nova/virt/libvirt/driver.py", line 2432, in ensure_filtering_rules_for_instance
    raise exception.NovaException(msg % instance_ref["name"])
NovaException: Timeout migrating for instance-00000009. nwfilter not found.
2012-08-21 00:03:46 CRITICAL nova [-] Timeout migrating for instance-00000009. nwfilter not found.
Timeout migrating for instance-00000009. nwfilter not found.

 nova list
+--------------------------------------+--------+--------+---------------+
| ID | Name | Status | Networks |
+--------------------------------------+--------+--------+---------------+
| fd7bdde0-ee53-4c8c-ad99-9c2945f021d0 | cirros | ACTIVE | net1=10.1.1.2 |
+--------------------------------------+--------+--------+---------------+

Revision history for this message
Mark McLoughlin (markmc) wrote :

Thanks for the report. Unfortunately, I can't reproduce here

Could you provide some more details:

 1) How is Nova configured? Perhaps just include your nova.conf files

 2) Could you also reproduce this with 'debug=true' in your nova.conf and attach the logs

 3) Please run 'sudo virsh nwfilter-list' after the error occurs and attach the output

 4) Could you also detail the commands you used to create the instance

Mark McLoughlin (markmc)
Changed in nova:
status: New → Incomplete
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/11799

Changed in nova:
assignee: nobody → Mark McLoughlin (markmc)
status: Incomplete → In Progress
Revision history for this message
Mark McLoughlin (markmc) wrote :

https://review.openstack.org/11799 isn't a fix, it just clears up the error message

Changed in nova:
status: In Progress → Confirmed
status: Confirmed → Incomplete
assignee: Mark McLoughlin (markmc) → nobody
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/11799
Committed: http://github.com/openstack/nova/commit/a29442d81ead759e1b33d01ef1603fa677fa86bf
Submitter: Jenkins
Branch: master

commit a29442d81ead759e1b33d01ef1603fa677fa86bf
Author: Mark McLoughlin <email address hidden>
Date: Wed Aug 22 14:56:30 2012 +0100

    Clarify nwfilter not found error message

    In bug #1039398 the user got an error message about an instance's
    nwfilter not being found, but it gave the impression that live
    migration was involved.

    The ensure_filtering_rules_for_instance() method was originally
    just used for live migration, but since:

      https://code.launchpad.net/~chemikadze/nova/driver-agnostic-restart-instances/+merge/69069

    it has also been used when re-starting the compute service when
    there are running VMs.

    Clarify the error message and the comments in the code to reflect
    this.

    Waiting up to 30 seconds before reporting an error here seems
    pretty bad. It appears this is to handle the case where the
    nwfilters get defined in a separate thread. For reference, the
    code was added by this merge:

      https://code.launchpad.net/~nttdata/nova/live-migration/+merge/44940

    For now, just add a comment explaining why we're polling for
    the existence of the nwfilter.

    Change-Id: Ieb085a6753ac72116d5f5a706fd1a908703286ad

Changed in nova:
status: Incomplete → Fix Committed
Revision history for this message
Naveen Joy (najoy) wrote :

Please find the requested data in the attachment. Thanks for looking into this issue. Let me know if you need anything else.

Revision history for this message
Naveen Joy (najoy) wrote :

Attachment

Revision history for this message
Mark McLoughlin (markmc) wrote :

Ah, I see the issue:

  firewall_driver=nova.virt.firewall.IptablesFirewallDriver

that should be:

  firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver

or you can just leave it unset in nova.conf

I've filed bug #1040430 about the value being wrong in nova.conf.sample

Changed in nova:
status: Fix Committed → Confirmed
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.