instance can't obtain IP address by DHCP when using LibvirtHybridOVSBridgeDriver

When an instance fails to obtain an IP address from DHCP, OVS has a port "qvo****" but the port cannot be seen as a bridge port. More precisely, "ovs-vsctl show" contains "qvo****" in br-int, but "brctl show br-int" or "ovs-ofctl show br-int" does not contain "qvo****". It is very strange behavior.

When nova-compute launches an instance on KVM, vif driver plug() is called twice
accorrding to the nova-compute log: http://paste.openstack.org/show/21031/.

LibvirtHybridOVSBridgeDriver plug() calles nova.network.linux_net._create_veth_pair()
to create veth pair. _create_veth_pair() checks whether the specified device exists
and in case the device exists it first deletes the device and (re)creates it.
It may be a cause of the problem.

After I changed plug() in Hybrid vif driver not to call _create_veth_pair() when the veth pair already exists, this issue is addressed. I will post a patch to apply this change soon.

Note that I am not sure why plug() method of VIF driver is called twice, and the above solution may be a workaround for this bug.

My workaround patch is here: http://paste.openstack.org/show/21033/
But it is a good chance to identify the original reason (plug is called twice during launching an instance).

Similar discussion is also here: bug 1053457.

plug should be idempotent, so your fix makes sense to me. That said, we should investigate why it is getting called twice.

I found plug operation to be actually potentially called twice - at least for the libvirt virt driver

spawn --> to_xml --> get_guest_config --> plug (used for adding device info to VM configuration)
spawn --> _create_domain_and_network --> plug_vifs --> plug (the "real" plug action)

Therefore we need the call to be idempotent, as stated by Vish, and we should take this patch.

However, this still do not quite explain why the behaviour is apparently not deterministic (or at least we have users confirming it works and other users say it doesn't work, and finally also users saying it works at times!)

See bug 1053457 as potentially related

amotoki: please propose the patch to gerrit.

Fix proposed to branch: master
Review: https://review.openstack.org/13434

Reviewed: https://review.openstack.org/13434
Committed: http://github.com/openstack/nova/commit/577ba6267994baecd5f9e05105624536d320cc9b
Submitter: Jenkins
Branch: master

commit 577ba6267994baecd5f9e05105624536d320cc9b
Author: Akihiro MOTOKI <email address hidden>
Date: Fri Sep 21 10:08:43 2012 +0900

    Ensure hybrid driver creates veth pair only once.

    Fixes bug 1053312

    This commit fixes a bug that an instance can't obtain IP address by DHCP
    when using LibvirtHybridOVSBridgeDriver. When nova-compute launches an
    instance on KVM, vif driver plug() may be called twice and this causes
    a behavior that a port of OVS cannot seen as a bridge port. So this patch
    make sure create_veth_pair() is not called if a port already exists.

    Change-Id: I76792df9f0dd28224d7d57989bf1e8ecadb92606

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/13495

Reviewed: https://review.openstack.org/13495
Committed: http://github.com/openstack/nova/commit/3254f58359479e31935928f61dd46628ae60df6a
Submitter: Jenkins
Branch: milestone-proposed

commit 3254f58359479e31935928f61dd46628ae60df6a
Author: Akihiro MOTOKI <email address hidden>
Date: Fri Sep 21 10:08:43 2012 +0900

    Ensure hybrid driver creates veth pair only once.

    Fixes bug 1053312

    This commit fixes a bug that an instance can't obtain IP address by DHCP
    when using LibvirtHybridOVSBridgeDriver. When nova-compute launches an
    instance on KVM, vif driver plug() may be called twice and this causes
    a behavior that a port of OVS cannot seen as a bridge port. So this patch
    make sure create_veth_pair() is not called if a port already exists.

    Change-Id: I76792df9f0dd28224d7d57989bf1e8ecadb92606
    (cherry picked from commit 577ba6267994baecd5f9e05105624536d320cc9b)

/nova/virt/libvirt/vif.py has the patch. But still see the issue on the compute node. Compute node dhcp fails and cannot ping the instance eth0 ip. when change libvirt_vif_driver from nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver to nova.virt.libvirt.vif.LibvirtOpenVswitchDriver, ping works.

setup is 2 node grizzly configuration.

psl let me know if there is anything that i may be missing.

after adding qvo* interface manually to ovs-vsctl, dhcp and ping works fine. can you pls let me know why it was not added in the first place?