OpenStack Compute (nova)

Libvirt: Raise wrong exception message when binding vif failed

Bug #1390336 reported by Eli Qiao
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Eli Qiao
OpenStack Compute (nova) 12.0.0 "liberty"

Bug Description

nova get an NovaException with wrong message due to wrong try to build an instance on compute node.

2014-11-07 14:40:54.446 ERROR nova.compute.manager [-] [instance: 9b56e64b-71b5-4598-b4df-45bb85b43ed6] Instance failed to spawn
2014-11-07 14:40:54.446 TRACE nova.compute.manager [instance: 9b56e64b-71b5-4598-b4df-45bb85b43ed6] Traceback (most recent call last):
2014-11-07 14:40:54.446 TRACE nova.compute.manager [instance: 9b56e64b-71b5-4598-b4df-45bb85b43ed6] File "/opt/stack/nova/nova/compute/manager.py", line 2244, in _build_resources
2014-11-07 14:40:54.446 TRACE nova.compute.manager [instance: 9b56e64b-71b5-4598-b4df-45bb85b43ed6] yield resources
2014-11-07 14:40:54.446 TRACE nova.compute.manager [instance: 9b56e64b-71b5-4598-b4df-45bb85b43ed6] File "/opt/stack/nova/nova/compute/manager.py", line 2114, in _build_and_run_instance
2014-11-07 14:40:54.446 TRACE nova.compute.manager [instance: 9b56e64b-71b5-4598-b4df-45bb85b43ed6] block_device_info=block_device_info)
2014-11-07 14:40:54.446 TRACE nova.compute.manager [instance: 9b56e64b-71b5-4598-b4df-45bb85b43ed6] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2597, in spawn
2014-11-07 14:40:54.446 TRACE nova.compute.manager [instance: 9b56e64b-71b5-4598-b4df-45bb85b43ed6] write_to_disk=True)
2014-11-07 14:40:54.446 TRACE nova.compute.manager [instance: 9b56e64b-71b5-4598-b4df-45bb85b43ed6] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4157, in _get_guest_xml
2014-11-07 14:40:54.446 TRACE nova.compute.manager [instance: 9b56e64b-71b5-4598-b4df-45bb85b43ed6] context)
2014-11-07 14:40:54.446 TRACE nova.compute.manager [instance: 9b56e64b-71b5-4598-b4df-45bb85b43ed6] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4018, in _get_guest_config
2014-11-07 14:40:54.446 TRACE nova.compute.manager [instance: 9b56e64b-71b5-4598-b4df-45bb85b43ed6] flavor, virt_type)
2014-11-07 14:40:54.446 TRACE nova.compute.manager [instance: 9b56e64b-71b5-4598-b4df-45bb85b43ed6] File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 352, in get_config
2014-11-07 14:40:54.446 TRACE nova.compute.manager [instance: 9b56e64b-71b5-4598-b4df-45bb85b43ed6] _("Unexpected vif_type=%s") % vif_type)
2014-11-07 14:40:54.446 TRACE nova.compute.manager [instance: 9b56e64b-71b5-4598-b4df-45bb85b43ed6] NovaException: Unexpected vif_type=binding_failed

Eli Qiao (taget-9)
Changed in nova:
assignee: nobody → Eli Qiao (taget-9)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/133154

Changed in nova:
status: New → In Progress
Revision history for this message
Jay Pipes (jaypipes) wrote :

Wouldn't the source of the problem here be that vif['type'] has "binding_failed" in it?

Revision history for this message
Eli Qiao (taget-9) wrote :

hmm, network_info if allocated from neutron, the source of problem is from neutron. and in my environment,
I know this may be a configure problem.

but, even this, don't we should handle it in nova?

Revision history for this message
Rito (ribs-1988) wrote :

Facing the Same Exception " NovaException: Unexpected vif_type=binding_failed" Hence Error while spawn ,
My setup is I am having NODE1(Controller+Network) and Compute , In compute other than neutron-openvswitch-agent , neutron-sriov-agent is also running ,
followed https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking for PCI passthrough . After much debugging found like in compute not able to bind the port.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Eli Qiao (<email address hidden>) on branch: master
Review: https://review.openstack.org/133154
Reason: will resubmit in another ps

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/156112

Revision history for this message
Brent Eagles (beagles) wrote :

Why doesn't neutron indicate an error has occurred through an exception? Is the port still valid even when this error has occurred?

Davanum Srinivas (DIMS) (dims-v)
Changed in nova:
importance: Undecided → Low
Revision history for this message
Brent Eagles (beagles) wrote :

I removed neutron from this bz after confirming that the binding_failed is a valid state and indicates a potentially transient condition.

no longer affects: neutron
Revision history for this message
Matt Riedemann (mriedem) wrote :

I've seen this issue too when the neutron ovs agent is down, in the case I saw that was due to bad rpc configuration. So this is a legit bug for nova since the serviceability side of nova isn't great on explaining what is failing here.

tags: added: libvirt network neutron
Matt Riedemann (mriedem)
Changed in nova:
importance: Low → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/156112
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7fc38d0c8498f73f6502520fee99158d130bbfc3
Submitter: Jenkins
Branch: master

commit 7fc38d0c8498f73f6502520fee99158d130bbfc3
Author: Eli Qiao <email address hidden>
Date: Thu Aug 13 15:02:40 2015 +0800

    Neutron: Check port binding status

    Neutron api creates/uses a port which the port vif_type is
    'binding_failed' without any verification, this leads to libvirt
    driver failing to create instance, and cause vif driver to raise
    a wrong exception message.

    This patch adds a new exception PortBindingFailed to indicate port binding
    failed. If port is newly created, delete it if PortBindingFailed exception
    catched.

    Closes-Bug: #1390336
    Change-Id: I66763ff5293473d1283d55f76dfe33d153abaaec

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → liberty-rc1
status: Fix Committed → Fix Released
Revision history for this message
Bjoern (bjoern-t) wrote :

What are the chances to get this back ported, if possible into Juno and Kilo ?

Thierry Carrez (ttx)
Changed in nova:
milestone: liberty-rc1 → 12.0.0
Revision history for this message
John L. Villalovos (happycamp) wrote :

So this fix seems to have caused this bug:

https://bugs.launchpad.net/nova/+bug/1527061

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.