vlan100 and br100 still available after stopping nova-network: restart of nova-network not possible, br100 in state DOWN after starting nova-network

Bug #715180 reported by Christian Berendt
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Vish Ishaya

Bug Description

After starting nova-network I have two new interfaces:

---snip---
6: vlan100@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 00:14:5e:6d:66:20 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::214:5eff:fe6d:6620/64 scope link
       valid_lft forever preferred_lft forever
7: br100: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
    link/ether 00:14:5e:6d:66:20 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.1/25 brd 192.168.0.127 scope global br100
---snap---

First problem: br100 is in state DOWN. I think this should be up..?

After stopping nova-network the two interfaces are still available:

---snip---
6: vlan100@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 00:14:5e:6d:66:20 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::214:5eff:fe6d:6620/64 scope link
       valid_lft forever preferred_lft forever
7: br100: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
    link/ether 00:14:5e:6d:66:20 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.1/25 brd 192.168.0.127 scope global br100
---snap---

If I now try to start nova-network again I become an error, because now we're using "ip address add" to assign the ip adress and not ifconfig (ifconfig overwrites the old settings) --> failed because the ip address is already there from the last time.

I think first of all br100 has to be set into state UP after starting nova-network. Second all vlan-interfaces and all bridges has to be stopped and to be removed after stopping nova-network. Then there is no problem while adding the new ip address after starting nova-network.

Is that right in this way..? I think that's an urgent problem and we should commit a fix asap.

---snip---
Feb 8 15:26:41 mandarin 2011-02-08 15:26:41,592 CRITICAL nova.root [-] Unexpected error while running command. Command: sudo ip addr add 192.168.0.1/255.255.255.128 dev br100 broadcast 192.168.0.127 Exit code: 2 Stdout: '' Stderr: 'RTNETLINK answers: File exists\n' (nova.root): TRACE: Traceback (most recent call last): (nova.root): TRACE: File "/usr/bin/nova-network", line 44, in <module> (nova.root): TRACE: service.serve() (nova.root): TRACE: File "/usr/lib64/python2.6/site-packages/nova/service.py", line 231, in serve (nova.root): TRACE: x.start() (nova.root): TRACE: File "/usr/lib64/python2.6/site-packages/nova/service.py", line 81, in start (nova.root): TRACE: self.manager.init_host() (nova.root): TRACE: File "/usr/lib64/python2.6/site-packages/nova/network/manager.py", line 467, in init_host (nova.root): TRACE: super(VlanManager, self).init_host() (nova.root): TRACE: File "/usr/lib64/python2.6/site-packages/nova/network/manager.py", line 125, in init_host (nova.root): TRACE: self._on_set_network_host(ctxt, network['id']) (nova.root): TRACE: File "/usr/lib64/python2.6/site-packages/nova/network/manager.py", line 568, in _on_set_network_host (nova.root): TRACE: network_ref) (nova.root): TRACE: File "/usr/lib64/python2.6/site-packages/nova/network/linux_net.py", line 173, in ensure_vlan_bridge (nova.root): TRACE: ensure_bridge(bridge, interface, net_attrs) (nova.root): TRACE: File "/usr/lib64/python2.6/site-packages/nova/network/linux_net.py", line 202, in ensure_bridge (nova.root): TRACE: net_attrs['broadcast'])) (nova.root): TRACE: File "/usr/lib64/python2.6/site-packages/nova/network/linux_net.py", line 331, in _execute (nova.root): TRACE: return utils.execute(cmd, *args, **kwargs) (nova.root): TRACE: File "/usr/lib64/python2.6/site-packages/nova/utils.py", line 147, in execute (nova.root): TRACE: cmd=cmd) (nova.root): TRACE: ProcessExecutionError: Unexpected error while running command. (nova.root): TRACE: Command: sudo ip addr add 192.168.0.1/255.255.255.128 dev br100 broadcast 192.168.0.127 (nova.root): TRACE: Exit code: 2 (nova.root): TRACE: Stdout: '' (nova.root): TRACE: Stderr: 'RTNETLINK answers: File exists\n' (nova.root): TRACE:
---snap---

/etc/nova.nova.conf:
[...]
--network_manager=nova.network.manager.VlanManager
--fixed_range=192.168.0.0/24
--network_size=5000

Related branches

summary: - vlan100 and br100 still available after stopping nova-network
+ vlan100 and br100 still available after stopping nova-network: restart
+ of nova-network not possible, br100 in state DOWN after starting nova-
+ network
Revision history for this message
Christian Berendt (berendt) wrote :

br100 is not set up because of a wrong intent in method ensure_bridge(). I'll link my branch later...

--- nova/network/linux_net.py.orig 2011-02-08 15:51:07.690001548 +0100
+++ nova/network/linux_net.py 2011-02-08 15:50:35.413334881 +0100
@@ -203,9 +203,9 @@
         if(FLAGS.use_ipv6):
             _execute("sudo ip -f inet6 addr change %s dev %s" %
                      (net_attrs['cidr_v6'], bridge))
- _execute("sudo ip link set %s up" % bridge)
- else:
- _execute("sudo ip link set %s up" % bridge)
+
+ _execute("sudo ip link set %s up" % bridge)
+
     if FLAGS.use_nova_chains:
         (out, err) = _execute("sudo iptables -N nova_forward",
                               check_exit_code=False)

Revision history for this message
Christian Berendt (berendt) wrote :

lp:~berendt/nova/lp715180_fixing_state_of_bridge should fix the problem with the bridge to come up after the creation

i'm not sure where to handle the destruction of the create vlan-interfaces and bridges

Revision history for this message
Vish Ishaya (vishvananda) wrote :

I think that this is fixed by my (under review) branch here: https://code.launchpad.net/~vishvananda/nova/lp710959/+merge/48341

Changed in nova:
importance: Undecided → High
status: New → In Progress
assignee: nobody → Vish Ishaya (vishvananda)
Revision history for this message
Vish Ishaya (vishvananda) wrote :

Removing the vlans and bridges is a little tough, because we wan't networking to stay up if compute crashes or is rebooted/upgraded. Some kind of clean command should probably be provided for uninstall.

Thierry Carrez (ttx)
Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → 2011.2
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.