OpenStack Compute (nova)

nova-network does not start if interfaces are already configured (QuantumLinuxBridgeInterfaceDriver)

Bug #1042249 reported by Lothar Gesslein on 2012-08-27

This bug report is a duplicate of: Bug #1000580: Quantum Linux Bridge interface driver plug operation fails with RTNETLINK error. Edit Remove

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Incomplete	Undecided	Unassigned

Bug Description

When i restart nova-network i get the following error:

2012-08-27 15:59:22 DEBUG nova.utils [req-1d408c19-7f61-4ca2-8767-f112bbbc921d None None] Attempting to grab semaphore "get_dhcp" for method "_get_dhcp_ip"... from (pid=15115) i
nner /usr/lib64/python2.6/site-packages/nova/utils.py:927
2012-08-27 15:59:22 DEBUG nova.utils [req-1d408c19-7f61-4ca2-8767-f112bbbc921d None None] Got semaphore "get_dhcp" for method "_get_dhcp_ip"... from (pid=15115) inner /usr/lib64
/python2.6/site-packages/nova/utils.py:931
2012-08-27 15:59:22 DEBUG nova.utils [req-1d408c19-7f61-4ca2-8767-f112bbbc921d None None] Running cmd (subprocess): ip link show dev gw-8240b34f-71 from (pid=15115) execute /usr
/lib64/python2.6/site-packages/nova/utils.py:219
2012-08-27 15:59:22 DEBUG nova.utils [req-1d408c19-7f61-4ca2-8767-f112bbbc921d None None] Running cmd (subprocess): ip link show dev brq8240b34f-71 from (pid=15115) execute /usr
/lib64/python2.6/site-packages/nova/utils.py:219
2012-08-27 15:59:22 DEBUG nova.utils [req-1d408c19-7f61-4ca2-8767-f112bbbc921d None None] Running cmd (subprocess): sudo ip address add 10.68.152.11/21 dev brq8240b34f-71 from (
pid=15115) execute /usr/lib64/python2.6/site-packages/nova/utils.py:219
2012-08-27 15:59:22 DEBUG nova.utils [req-1d408c19-7f61-4ca2-8767-f112bbbc921d None None] Result was 254 from (pid=15115) execute /usr/lib64/python2.6/site-packages/nova/utils.p
y:235
2012-08-27 15:59:22 CRITICAL nova [-] Unexpected error while running command.
Command: sudo ip address add 10.68.152.11/21 dev brq8240b34f-71
Exit code: 254
Stdout: ''
Stderr: 'RTNETLINK answers: File exists\n'
2012-08-27 15:59:22 TRACE nova Traceback (most recent call last):
2012-08-27 15:59:22 TRACE nova File "/usr/bin/nova-network", line 49, in <module>
2012-08-27 15:59:22 TRACE nova service.wait()
2012-08-27 15:59:22 TRACE nova File "/usr/lib64/python2.6/site-packages/nova/service.py", line 413, in wait
2012-08-27 15:59:22 TRACE nova _launcher.wait()
2012-08-27 15:59:22 TRACE nova File "/usr/lib64/python2.6/site-packages/nova/service.py", line 131, in wait
2012-08-27 15:59:22 TRACE nova service.wait()
2012-08-27 15:59:22 TRACE nova File "/usr/lib64/python2.6/site-packages/eventlet/greenthread.py", line 166, in wait
2012-08-27 15:59:22 TRACE nova return self._exit_event.wait()
2012-08-27 15:59:22 TRACE nova File "/usr/lib64/python2.6/site-packages/eventlet/event.py", line 116, in wait
2012-08-27 15:59:22 TRACE nova return hubs.get_hub().switch()
2012-08-27 15:59:22 TRACE nova File "/usr/lib64/python2.6/site-packages/eventlet/hubs/hub.py", line 177, in switch
2012-08-27 15:59:22 TRACE nova return self.greenlet.switch()
2012-08-27 15:59:22 TRACE nova File "/usr/lib64/python2.6/site-packages/eventlet/greenthread.py", line 192, in main
2012-08-27 15:59:22 TRACE nova result = function(*args, **kwargs)
2012-08-27 15:59:22 TRACE nova File "/usr/lib64/python2.6/site-packages/nova/service.py", line 101, in run_server
2012-08-27 15:59:22 TRACE nova server.start()
2012-08-27 15:59:22 TRACE nova File "/usr/lib64/python2.6/site-packages/nova/service.py", line 162, in start
2012-08-27 15:59:22 TRACE nova self.manager.init_host()
2012-08-27 15:59:22 TRACE nova File "/usr/lib64/python2.6/site-packages/nova/network/quantum/manager.py", line 97, in init_host
2012-08-27 15:59:22 TRACE nova super(QuantumManager, self).init_host()
2012-08-27 15:59:22 TRACE nova File "/usr/lib64/python2.6/site-packages/nova/network/manager.py", line 768, in init_host
2012-08-27 15:59:22 TRACE nova self._setup_network_on_host(ctxt, network)
2012-08-27 15:59:22 TRACE nova File "/usr/lib64/python2.6/site-packages/nova/network/quantum/manager.py", line 122, in _setup_network_on_host
2012-08-27 15:59:22 TRACE nova self.l3driver.initialize_gateway(network)
2012-08-27 15:59:22 TRACE nova File "/usr/lib64/python2.6/site-packages/nova/network/l3.py", line 98, in initialize_gateway
2012-08-27 15:59:22 TRACE nova gateway=(network_ref['gateway'] is not None))
2012-08-27 15:59:22 TRACE nova File "/usr/lib64/python2.6/site-packages/nova/network/linux_net.py", line 905, in plug
2012-08-27 15:59:22 TRACE nova return _get_interface_driver().plug(network, mac_address, gateway)
2012-08-27 15:59:22 TRACE nova File "/usr/lib64/python2.6/site-packages/nova/network/linux_net.py", line 1164, in plug
2012-08-27 15:59:22 TRACE nova run_as_root=True)
2012-08-27 15:59:22 TRACE nova File "/usr/lib64/python2.6/site-packages/nova/utils.py", line 242, in execute
2012-08-27 15:59:22 TRACE nova cmd=' '.join(cmd))
2012-08-27 15:59:22 TRACE nova ProcessExecutionError: Unexpected error while running command.
2012-08-27 15:59:22 TRACE nova Command: sudo ip address add 10.68.152.11/21 dev brq8240b34f-71
2012-08-27 15:59:22 TRACE nova Exit code: 254
2012-08-27 15:59:22 TRACE nova Stdout: ''
2012-08-27 15:59:22 TRACE nova Stderr: 'RTNETLINK answers: File exists\n'
2012-08-27 15:59:22 TRACE nova

Stopping nova-network does not unconfigure the bridge/gw-interfaces, so they retain their ip address, but nova-network insists that it needs to add the ip address to the device in order to start.

This is the patch we run now in order to work around the issue:

index 1869857..70f388d 100755
--- a/nova/network/linux_net.py
+++ b/nova/network/linux_net.py
@@ -1151,8 +1151,15 @@ class QuantumLinuxBridgeInterfaceDriver(LinuxNetInterfaceDriver):

full_ip = '%s/%s' % (network['dhcp_server'],
network['cidr'].rpartition('/')[2])
- utils.execute('ip', 'address', 'add', full_ip, 'dev', bridge,
- run_as_root=True)
+ try:
+ utils.execute('ip', 'address', 'add', full_ip, 'dev', bridge,
+ run_as_root=True)
+ except exception.ProcessExecutionError as e:
+ if "RTNETLINK answers: File exists" in str(e):
+ LOG.warning(_("IP %s already configured on dev %s"), full_ip, bridge)
+ else:
+ LOG.warning(_("Failed adding ip address %s to dev %s"), full_ip, bridge)
+ raise

return dev

Tags:

Revision history for this message

Mark McLoughlin (markmc) wrote on 2012-08-30:

Thanks for the report!

So, this is with linuxnet_interface_driver=nova.network.linux_net.QuantumLinuxBridgeInterfaceDriver

I'm not sure I understand how your patch fixes the problem. The code is:

        if not _device_exists(bridge):
            utils.execute('brctl', 'addbr', bridge, run_as_root=True)
            ...
            full_ip = '%s/%s' % (network['dhcp_server'],
                                 network['cidr'].rpartition('/')[2])
            utils.execute('ip', 'address', 'add', full_ip, 'dev', bridge,
                          run_as_root=True, check_exit_code=[0, 2, 254])

i.e. we only assign an IP address if the bridge did not already exist. How can the IP be already assigned to a bridge we've just created?

Is the root cause actually that we _device_exists() is returning False for a bridge that does exist?

Could you post some more details of your setup too? e.g.

* nova and quantum versions
* nova and quantum config (e.g. grep -v '^#' nova.conf quantum.conf)
* what commands you used to create networks

Changed in nova:
status:	New → Incomplete
summary:	nova-network does not start if interfaces are already configured + (QuantumLinuxBridgeInterfaceDriver)
tags:	added: quantum

Revision history for this message

Lothar Gesslein (lgesslein) wrote on 2012-08-30:

This is on stable essex 2012.1.2. class QuantumLinuxBridgeInterfaceDriver function plug.

There the conditional "if not _device_exists(bridge):" does _not_ enclose the utils.execute call to ip addr add, and thus it will fail in the way i described if it cannot assign the ip. Sorry for not mentioning the version.

Revision history for this message

Mark McLoughlin (markmc) wrote on 2012-08-30:

Ah, thanks - that makes sense

Report a bug

This report contains Public information

Everyone can see this information.

Duplicate of bug #1000580 Remove

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.