OpenStack Compute (nova)

euca-authorize messes up nova-compute-local chains on nodes with more than 1 running instance

Bug #798346 reported by Kevin Bringard on 2011-06-16

This bug affects 5 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Fix Released	High	Unassigned	OpenStack Compute (nova) 2011.3 "diablo"

Bug Description

When modifying a security group with running instances, the nova-compute-local chain is incorrectly updated.

IPTables output before adding/revoking a rule:

Chain nova-compute-inst-89 (1 references)
pkts bytes target prot opt in out source destination
    0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
    0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
    0 0 ACCEPT udp -- * * 10.16.1.1 0.0.0.0/0 udp spt:67 dpt:68
    0 0 ACCEPT tcp -- * * 192.168.100.0/24 0.0.0.0/0 tcp dpt:22
    0 0 ACCEPT icmp -- * * 192.168.100.0/24 0.0.0.0/0
    0 0 ACCEPT tcp -- * * 10.16.1.0/24 0.0.0.0/0 tcp dpt:22
    0 0 ACCEPT icmp -- * * 10.16.1.0/24 0.0.0.0/0
    0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
    0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3000
    0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3001
    0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3002
    0 0 nova-compute-sg-fallback all -- * * 0.0.0.0/0 0.0.0.0/0

Chain nova-compute-inst-93 (1 references)
pkts bytes target prot opt in out source destination
    0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
    0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
    0 0 ACCEPT udp -- * * 10.16.1.1 0.0.0.0/0 udp spt:67 dpt:68
    0 0 ACCEPT tcp -- * * 192.168.100.0/24 0.0.0.0/0 tcp dpt:22
    0 0 ACCEPT icmp -- * * 192.168.100.0/24 0.0.0.0/0
    0 0 ACCEPT tcp -- * * 10.16.1.0/24 0.0.0.0/0 tcp dpt:22
    0 0 ACCEPT icmp -- * * 10.16.1.0/24 0.0.0.0/0
    0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
    0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3000
    0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3001
    0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3002
    0 0 nova-compute-sg-fallback all -- * * 0.0.0.0/0 0.0.0.0/0

Chain nova-compute-local (1 references)
pkts bytes target prot opt in out source destination
0 0 nova-compute-inst-89 all -- * * 0.0.0.0/0 10.16.1.3
0 0 nova-compute-inst-93 all -- * * 0.0.0.0/0 10.16.1.7

iptables output after revoking a rule (specifically tcp dpt 3002):

Chain nova-compute-inst-89 (1 references)
pkts bytes target prot opt in out source destination
    0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
    0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
    0 0 ACCEPT udp -- * * 10.16.1.1 0.0.0.0/0 udp spt:67 dpt:68
    0 0 ACCEPT tcp -- * * 192.168.100.0/24 0.0.0.0/0 tcp dpt:22
    0 0 ACCEPT icmp -- * * 192.168.100.0/24 0.0.0.0/0
    0 0 ACCEPT tcp -- * * 10.16.1.0/24 0.0.0.0/0 tcp dpt:22
    0 0 ACCEPT icmp -- * * 10.16.1.0/24 0.0.0.0/0
    0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
    0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3000
    0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3001
    0 0 nova-compute-sg-fallback all -- * * 0.0.0.0/0 0.0.0.0/0

Chain nova-compute-inst-93 (1 references)
pkts bytes target prot opt in out source destination
    0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
    0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
    0 0 ACCEPT udp -- * * 10.16.1.1 0.0.0.0/0 udp spt:67 dpt:68
    0 0 ACCEPT tcp -- * * 192.168.100.0/24 0.0.0.0/0 tcp dpt:22
    0 0 ACCEPT icmp -- * * 192.168.100.0/24 0.0.0.0/0
    0 0 ACCEPT tcp -- * * 10.16.1.0/24 0.0.0.0/0 tcp dpt:22
    0 0 ACCEPT icmp -- * * 10.16.1.0/24 0.0.0.0/0
    0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
    0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3000
    0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3001
    0 0 nova-compute-sg-fallback all -- * * 0.0.0.0/0 0.0.0.0/0

Notice that the nova-compute-local chain has been updated to set destination to the first destination IP in the chain for all of them. I have tested that this happens with any number of instances > 1.

This incorrect state can be resolved by flushing the iptables rules and restarting compute, causing it to rebuild the chains:

Chain nova-compute-inst-89 (0 references)
pkts bytes target prot opt in out source destination

Chain nova-compute-inst-93 (0 references)
pkts bytes target prot opt in out source destination

Chain nova-compute-local (0 references)
pkts bytes target prot opt in out source destination

service nova-compute restart
nova-compute start/running, process 2797

Chain nova-compute-inst-89 (1 references)
pkts bytes target prot opt in out source destination
    0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
    0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
    0 0 ACCEPT udp -- * * 10.16.1.1 0.0.0.0/0 udp spt:67 dpt:68
    0 0 ACCEPT tcp -- * * 192.168.100.0/24 0.0.0.0/0 tcp dpt:22
    0 0 ACCEPT icmp -- * * 192.168.100.0/24 0.0.0.0/0
    0 0 ACCEPT tcp -- * * 10.16.1.0/24 0.0.0.0/0 tcp dpt:22
    0 0 ACCEPT icmp -- * * 10.16.1.0/24 0.0.0.0/0
    0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
    0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3000
    0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3001
    0 0 nova-compute-sg-fallback all -- * * 0.0.0.0/0 0.0.0.0/0

Chain nova-compute-inst-93 (1 references)
pkts bytes target prot opt in out source destination
    0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
    0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
    0 0 ACCEPT udp -- * * 10.16.1.1 0.0.0.0/0 udp spt:67 dpt:68
    0 0 ACCEPT tcp -- * * 192.168.100.0/24 0.0.0.0/0 tcp dpt:22
    0 0 ACCEPT icmp -- * * 192.168.100.0/24 0.0.0.0/0
    0 0 ACCEPT tcp -- * * 10.16.1.0/24 0.0.0.0/0 tcp dpt:22
    0 0 ACCEPT icmp -- * * 10.16.1.0/24 0.0.0.0/0
    0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
    0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3000
    0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3001
    0 0 nova-compute-sg-fallback all -- * * 0.0.0.0/0 0.0.0.0/0

This causes all other local chains to be moot and allows all traffic to every other instance. It only appears to effect instances in the same project (which makes sense).

Tags:

Thierry Carrez (ttx) on 2011-06-17

Changed in nova:
importance:	Undecided → High
status:	New → Confirmed

Revision history for this message

Soren Hansen (soren) wrote on 2011-06-17:

This is due to do_refresh_security_group sets network_info only on the first iteration of its loop and then recycles that for subsequent iterations.

network_info is a per-instance thing. It makes no sense to pass it to methods that act on more than one instance, so lets stop doing that. Ever again. :)

Changed in nova:
status:	Confirmed → Triaged

Revision history for this message

Kevin Bringard (kbringard) wrote on 2011-08-25:

This appears to have been fixed. Can anyone else confirm?

Thierry Carrez (ttx) on 2011-09-02

tags:

added: security-group

Revision history for this message

Soren Hansen (soren) wrote on 2011-09-02:

Yup, Vish (accidentally?) fixed this in r1435. Yay.

Changed in nova:
status:	Triaged → Fix Committed

Thierry Carrez (ttx) on 2011-09-09

Changed in nova:
milestone:	none → diablo-4

Thierry Carrez (ttx) on 2011-09-22

Changed in nova:
milestone:	diablo-4 → 2011.3
status:	Fix Committed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.