nova-network crashes on restart with floating ips assigned

Bug #711948 reported by Vish Ishaya
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Vish Ishaya

Bug Description

Sometimes when starting nova network, it crashes trying to create iptables rules for floating forward with the following traceback:

AUDIT [N/A] Starting network node
Traceback (most recent call last):
  File "/usr/bin/nova-network", line 44, in <module>
    service.serve()
  File "/usr/lib/pymodules/python2.6/nova/service.py", line 239, in serve
    x.start()
  File "/usr/lib/pymodules/python2.6/nova/service.py", line 80, in start
    self.manager.init_host()
  File "/usr/lib/pymodules/python2.6/nova/network/manager.py", line 448, in init_host
    super(VlanManager, self).init_host()
  File "/usr/lib/pymodules/python2.6/nova/network/manager.py", line 131, in init_host
    fixed_address)
  File "/usr/lib/pymodules/python2.6/nova/network/linux_net.py", line 155, in ensure_floating_forward
    % (fixed_ip, floating_ip))
  File "/usr/lib/pymodules/python2.6/nova/network/linux_net.py", line 296, in _confirm_rule
    _execute("sudo iptables %s %s %s" % (loc, chain, cmd))
  File "/usr/lib/pymodules/python2.6/nova/network/linux_net.py", line 277, in _execute
    return utils.execute(cmd, *args, **kwargs)
  File "/usr/lib/pymodules/python2.6/nova/utils.py", line 144, in execute
    cmd=cmd)
nova.exception.ProcessExecutionError: Unexpected error while running command.
Command: sudo iptables -I SNATTING -t nat -s 10.0.0.3 -j SNAT --to 140.221.100.12
Exit code: 1
Stdout: ''
Stderr: 'iptables: No chain/target/match by that name.\n'

This happens if the iptables rules have been flushed due to manual changes or node restart. It is due to two commands being run out of order. Manual workaround is as follows:

on the network host:
sudo iptables -t nat -N SNATTING
sudo iptables -t nat -A POSTROUTING -j SNATTING

then nova-network should start normally

if you have --use_nova_chains set to true, the error message will reference nova_snatting instead of SNATTING and the workouround is the following commands:
sudo iptables -t nat -N nova_snatting
sudo iptables -t nat -A POSTROUTING -j nova_snatting

This can be fixed by moving where init_host is run.

Related branches

Changed in nova:
importance: Undecided → Medium
status: New → In Progress
assignee: nobody → Vish Ishaya (vishvananda)
Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → 2011.2
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.