OpenStack Compute (nova)

nova is not setting the MTU provided by Neutron

Bug #1623876 reported by Kevin Benton
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Kevin Benton
os-vif
Fix Released
Undecided
Kevin Benton

Bug Description

Spotted in gate grenade job. We can see neutron MTU is 1450 but the mtu set calls in privsep use 1500.
http://logs.openstack.org/56/369956/3/gate/gate-grenade-dsvm-neutron-ubuntu-trusty/83daad8/logs/new/screen-n-cpu.txt.gz#_2016-09-15_01_16_57_512

Relevant log snippet:

2016-09-15 01:16:57.512 25573 DEBUG nova.network.os_vif_util [req-53929d93-d999-4035-8aae-f8d9fd1b2efb tempest-AttachInterfacesTestJSON-355908889 tempest-AttachInterfacesTestJSON-355908889] Converting VIF {"profile": {}, "ovs_interfaceid": "8dfdfd9b-da9d-4215-abbd-4dffdc48494b", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "10.1.0.9"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "10.1.0.0/28", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "10.1.0.1"}}], "meta": {"injected": false, "tenant_id": "563ca55619b1402ebf0c792ec604a774", "mtu": 1450}, "id": "6e1f0d14-a238-4da9-a2d5-659a0f28479c", "label": "tempest-AttachInterfacesTestJSON-32449395-network"}, "devname": "tap8dfdfd9b-da", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true, "ovs_hybrid_plug": true}, "address": "fa:16:3e:38:52:12", "active": false, "type": "ovs", "id": "8dfdfd9b-da9d-4215-abbd-4dffdc48494b", "qbg_params": null} nova_to_osvif_vif /opt/stack/new/nova/nova/network/os_vif_util.py:362
2016-09-15 01:16:57.513 25573 DEBUG nova.network.os_vif_util [req-53929d93-d999-4035-8aae-f8d9fd1b2efb tempest-AttachInterfacesTestJSON-355908889 tempest-AttachInterfacesTestJSON-355908889] Converted object VIFBridge(active=False,address=fa:16:3e:38:52:12,bridge_name='qbr8dfdfd9b-da',has_traffic_filtering=True,id=8dfdfd9b-da9d-4215-abbd-4dffdc48494b,network=Network(6e1f0d14-a238-4da9-a2d5-659a0f28479c),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap8dfdfd9b-da') nova_to_osvif_vif /opt/stack/new/nova/nova/network/os_vif_util.py:374
2016-09-15 01:16:57.514 25573 DEBUG os_vif [req-53929d93-d999-4035-8aae-f8d9fd1b2efb tempest-AttachInterfacesTestJSON-355908889 tempest-AttachInterfacesTestJSON-355908889] Plugging vif VIFBridge(active=False,address=fa:16:3e:38:52:12,bridge_name='qbr8dfdfd9b-da',has_traffic_filtering=True,id=8dfdfd9b-da9d-4215-abbd-4dffdc48494b,network=Network(6e1f0d14-a238-4da9-a2d5-659a0f28479c),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap8dfdfd9b-da') plug /usr/local/lib/python2.7/dist-packages/os_vif/__init__.py:76
2016-09-15 01:16:57.515 25573 DEBUG oslo.privsep.daemon [-] privsep: request[140021949493072]: (3, 'vif_plug_ovs.linux_net.ensure_bridge', (u'qbr8dfdfd9b-da',), {}) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.515 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): brctl addbr qbr8dfdfd9b-da out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.517 25573 DEBUG neutronclient.v2_0.client [req-6754757c-066c-488a-bc16-6bd451c28cdc tempest-ServerActionsTestJSON-1197364963 tempest-ServerActionsTestJSON-1197364963] GET call to neutron for http://127.0.0.1:9696/v2.0/subnets.json?id=2b899e3c-17dc-478a-bd39-91132bb057ab used request id req-e82dd927-a0f8-48ba-bbbc-56724a10a29d _append_request_id /usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:127
2016-09-15 01:16:57.521 25573 DEBUG oslo.privsep.daemon [-] CMD "brctl addbr qbr8dfdfd9b-da" returned: 0 in 0.005s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.521 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): brctl setfd qbr8dfdfd9b-da 0 out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.524 25573 DEBUG oslo.privsep.daemon [-] CMD "brctl setfd qbr8dfdfd9b-da 0" returned: 0 in 0.004s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.524 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): brctl stp qbr8dfdfd9b-da off out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.528 25573 DEBUG oslo.privsep.daemon [-] CMD "brctl stp qbr8dfdfd9b-da off" returned: 0 in 0.003s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.528 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): tee /sys/class/net/qbr8dfdfd9b-da/bridge/multicast_snooping out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.532 25573 DEBUG oslo.privsep.daemon [-] CMD "tee /sys/class/net/qbr8dfdfd9b-da/bridge/multicast_snooping" returned: 0 in 0.004s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.532 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): tee /proc/sys/net/ipv6/conf/qbr8dfdfd9b-da/disable_ipv6 out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.537 25573 DEBUG oslo.privsep.daemon [-] CMD "tee /proc/sys/net/ipv6/conf/qbr8dfdfd9b-da/disable_ipv6" returned: 0 in 0.004s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.537 25573 DEBUG oslo.privsep.daemon [-] privsep: reply[140021949493072]: (4, None) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.537 25573 DEBUG oslo.privsep.daemon [-] privsep: request[140021949493072]: (3, 'vif_plug_ovs.linux_net.create_veth_pair', ('qvb8dfdfd9b-da', 'qvo8dfdfd9b-da', 1500), {}) out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.538 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): ip link add qvb8dfdfd9b-da type veth peer name qvo8dfdfd9b-da out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.546 25573 DEBUG oslo.privsep.daemon [-] CMD "ip link add qvb8dfdfd9b-da type veth peer name qvo8dfdfd9b-da" returned: 0 in 0.007s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.547 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): ip link set qvb8dfdfd9b-da up out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.550 25573 DEBUG oslo.privsep.daemon [-] CMD "ip link set qvb8dfdfd9b-da up" returned: 0 in 0.003s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.550 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): ip link set qvb8dfdfd9b-da promisc on out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.556 25573 DEBUG oslo.privsep.daemon [-] CMD "ip link set qvb8dfdfd9b-da promisc on" returned: 0 in 0.006s out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.556 25573 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): ip link set qvb8dfdfd9b-da mtu 1500 out_of_band /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
2016-09-15 01:16:57.560 25573 DEBUG oslo.privsep.daemon [-] CMD "ip link set qvb8dfdfd9b-da mtu 1500" returned: 0 in 0.004s out_of_band /usr/local/lib/python2.7/dist-

Kevin Benton (kevinbenton)
Changed in nova:
assignee: nobody → Kevin Benton (kevinbenton)
Changed in os-vif:
assignee: nobody → Kevin Benton (kevinbenton)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to os-vif (master)

Fix proposed to branch: master
Review: https://review.openstack.org/370667

Changed in os-vif:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/370679

Changed in nova:
status: New → In Progress
Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

Given the discussion we had on IRC with tonyb and other folks, it seems we need to have new os-vif version plus some minimum requirement bump for os-vif in order to fix that for Newton.

Putting it as Critical and blocker for Newton GA, but we can discuss on that to see whether we should wait for that.

Changed in nova:
importance: Undecided → Critical
tags: added: network newton-rc-potential
Revision history for this message
Daniel Berrange (berrange) wrote :

We don't need any os-vif version bump. We're doing a targetted workaround in nova/virt/libvirt/vif.py for Newton. We'll do the proper fix with os-vif version bump in Ocata.

Revision history for this message
Matt Riedemann (mriedem) wrote :

Nova workaround that danpb mentioned: https://review.openstack.org/#/c/370681/

Revision history for this message
Matt Riedemann (mriedem) wrote :

Dropped from critical to high as this only impacts libvirt deployments.

Changed in nova:
importance: Critical → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/370681
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=77f546623bb6c0b2a4441940f1740abd45bd3352
Submitter: Jenkins
Branch: master

commit 77f546623bb6c0b2a4441940f1740abd45bd3352
Author: John Garbutt <email address hidden>
Date: Thu Sep 15 11:29:45 2016 +0100

    Override MTU for os_vif attachments

    os-vif does not current respect the neutron provided mtu, it just uses
    the configuration inside os-vif. This is a big regression from mitaka
    and liberty.

    Long term, this is something os-vif will be able to do by correctly
    parsing and acting on the network info. For now we just set the mtu for
    a second time once os-vif has done what it wants to do.

    Closes-Bug: #1623876

    Change-Id: Id4ca38fa1bb84f8cdb5edcd9ccb7acd8c8e9b60c

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to os-vif (master)

Reviewed: https://review.openstack.org/370667
Committed: https://git.openstack.org/cgit/openstack/os-vif/commit/?id=f3130fe8e23fb07532bb3d740fffaf07481c1e70
Submitter: Jenkins
Branch: master

commit f3130fe8e23fb07532bb3d740fffaf07481c1e70
Author: Kevin Benton <email address hidden>
Date: Wed Sep 14 17:41:31 2016 -0700

    Add MTU to Network model and use it in plugging

    This adds an MTU field to the network model and
    has the vif_plug_ovs and vif_plug_linux_bridge drivers
    check for it before referencing the global config variable.

    Closes-Bug: #1623876
    Change-Id: I327c901a285bca23560f49a921a5d030f7f71cad

Changed in os-vif:
status: In Progress → Fix Released
Revision history for this message
Richard Theis (rtheis) wrote :

The nova change appears to break networking-ovn. Please see http://logs.openstack.org/74/353074/5/check/gate-tempest-dsvm-networking-ovn/4c8149e/ and http://paste.openstack.org/show/579969/

Revision history for this message
Richard Theis (rtheis) wrote :

I've filed https://bugs.launchpad.net/nova/+bug/1624392.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 14.0.0.0rc1

This issue was fixed in the openstack/nova 14.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/os-vif 1.3.0

This issue was fixed in the openstack/os-vif 1.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/394669

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/os-vif 1.3.0

This issue was fixed in the openstack/os-vif 1.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.openstack.org/370679
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=43565ffb9785f6e4848a001cc4be5cec91bfeab1
Submitter: Jenkins
Branch: master

commit 43565ffb9785f6e4848a001cc4be5cec91bfeab1
Author: Kevin Benton <email address hidden>
Date: Wed Sep 14 17:46:29 2016 -0700

    Pass MTU into os-vif Network object

    This passes the MTU from the network_info into the
    constructed os-vif object so it can be used by the
    os-vif plugging libraries to set the correct MTU.

    Now with os-vif >=1.3.0 in the global requirements,
    os-vif has correct support for MTUs on the os-vif
    objects so we can use them correctly.

    The follow-up patch removes the MTU hacks that were
    added last minute for the related bug.

    Related-Bug: #1623876
    Change-Id: Iae0d83d22511f35f00ed6d8fde03f8c32dbedb26

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/394669
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f67d110eac69d5d176034cc4b1af6fee98e3fd7c
Submitter: Jenkins
Branch: master

commit f67d110eac69d5d176034cc4b1af6fee98e3fd7c
Author: Kevin Benton <email address hidden>
Date: Mon Nov 7 14:59:42 2016 -0800

    Revert MTU hacks for bug 1623876

    Now with the native support for MTUs in the os-vif data model,
    we no longer require the MTU hacks added at the end of Newton
    to honor the MTUs set by Neutron.

    Revert "Stop ovn networking failing on mtu"
    This reverts commit fc0e281743917dbda25fb1911500439abed192ca.
    Revert "Override MTU for os_vif attachments"
    This reverts commit 77f546623bb6c0b2a4441940f1740abd45bd3352.

    Related-Bug: #1623876
    Change-Id: I82d7a1630e81254dfd6196d51add19ac2cbc5bdb

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.