juju tries to delete in-use security groups
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
pyjuju |
Fix Released
|
Medium
|
Jim Baker | ||
0.5 |
Fix Released
|
Medium
|
Unassigned |
Bug Description
I see this from time to time on my openstack logs:
2012-06-26 16:14:17 ERROR nova.api.ec2 [req-7c2edc93-
2012-06-26 16:14:17 TRACE nova.api.ec2 Traceback (most recent call last):
2012-06-26 16:14:17 TRACE nova.api.ec2 File "/opt/stack/
2012-06-26 16:14:17 TRACE nova.api.ec2 result = api_request.
2012-06-26 16:14:17 TRACE nova.api.ec2 File "/opt/stack/
2012-06-26 16:14:17 TRACE nova.api.ec2 result = method(context, **args)
2012-06-26 16:14:17 TRACE nova.api.ec2 File "/opt/stack/
2012-06-26 16:14:17 TRACE nova.api.ec2 self.security_
2012-06-26 16:14:17 TRACE nova.api.ec2 File "/opt/stack/
2012-06-26 16:14:17 TRACE nova.api.ec2 self.raise_
2012-06-26 16:14:17 TRACE nova.api.ec2 File "/opt/stack/
2012-06-26 16:14:17 TRACE nova.api.ec2 raise exception.
2012-06-26 16:14:17 TRACE nova.api.ec2 InvalidGroup: Group not valid. Reason: Security group is still in use
2012-06-26 16:14:17 TRACE nova.api.ec2
2012-06-26 16:14:17 ERROR nova.api.ec2 [req-7c2edc93-
2012-06-26 16:14:17 ERROR nova.api.ec2 [req-7c2edc93-
Note that it happens sometime after a deploy to a unit has failed with a hung or wedged machine. At a guess, the wedged machine holds a reference and juju is trying to clean up before getting things going. Problem is, that hung instance needs to be cleared first.
2012-06-26 04:18:18,491: juju.agents.
Juju sees the error (from provisioning.log):
2012-06-26 04:18:18,851: juju.ec2@DEBUG: Cannot delete security group juju-devstack-7: Error Message: An unknown error has occurred. Please try your request again.
2012-06-26 04:18:18,854: juju.agents.
Traceback (most recent call last):
File "/usr/lib/
machine_
File "/usr/lib/
result = result.
File "/usr/lib/
return g.throw(self.type, self.value, self.tb)
File "/usr/lib/
{"machine-id": machine_state.id, "constraints": constraints})
File "/usr/lib/
result = result.
File "/usr/lib/
return g.throw(self.type, self.value, self.tb)
File "/usr/lib/
machines = yield self.start_
File "/usr/lib/
result = result.
File "/usr/lib/
return g.throw(self.type, self.value, self.tb)
File "/usr/lib/
security_groups = yield self._ensure_
File "/usr/lib/
result = result.
File "/usr/lib/
return g.throw(self.type, self.value, self.tb)
File "/usr/lib/
% (juju_machine_
ProviderInterac
Related branches
- Kapil Thangavelu (community): Approve
-
Diff: 714 lines (+87/-348)11 files modifiedjuju/providers/ec2/__init__.py (+16/-22)
juju/providers/ec2/launch.py (+6/-24)
juju/providers/ec2/securitygroup.py (+0/-43)
juju/providers/ec2/tests/common.py (+0/-12)
juju/providers/ec2/tests/test_launch.py (+8/-14)
juju/providers/ec2/tests/test_provider.py (+2/-2)
juju/providers/ec2/tests/test_securitygroup.py (+3/-174)
juju/providers/ec2/tests/test_shutdown.py (+8/-53)
juju/providers/ec2/tests/test_utils.py (+4/-2)
juju/providers/ec2/utils.py (+4/-2)
juju/state/tests/test_firewall.py (+36/-0)
Changed in juju: | |
assignee: | nobody → Jim Baker (jimbaker) |
milestone: | none → honolulu |
status: | New → In Progress |
Changed in juju: | |
importance: | Undecided → Medium |
This is the entire lead-up: instance_ type='m1. small', image_id= 'ami-00000004' ) expose@ DEBUG: Assigned units for machine 7: old=None, new=set( ['elasticsearch /0']) expose@ INFO: No provisioned machine for machine 7 provision@ INFO: Starting machine id:7 ... verification is disabled for this environment provision@ ERROR: Cannot process machine 7 python2. 7/dist- packages/ juju/agents/ provision. py", line 187, in process_machines state_id, provider_machines) python2. 7/dist- packages/ twisted/ internet/ defer.py" , line 1037, in _inlineCallbacks throwExceptionI ntoGenerator( g) python2. 7/dist- packages/ twisted/ python/ failure. py", line 382, in throwExceptionI ntoGenerator python2. 7/dist- packages/ juju/agents/ provision. py", line 228, in process_machine python2. 7/dist- packages/ twisted/ internet/ defer.py" , line 1037, in _inlineCallbacks throwExceptionI ntoGenerator( g) python2. 7/dist- packages/ twisted/ python/ failure. py", line 382, in throwExceptionI ntoGenerator python2. 7/dist- packages/ juju/providers/ common/ launch. py", line 64, in run machine( machine_ id, zookeepers) python2. 7/dist- packages/ twisted/ internet/ defer.py" , line 1037, in _inlineCallbacks throwExceptionI ntoGenerator( g) python2. 7/dist- packages/ twisted/ python/ failure. py", line 382, in throwExceptionI ntoGenerator python2. 7/dist- packages/ juju/providers/ ec2/launch. py", line 42, in start_machine groups( machine_ id) python2. 7/dist- packages/ twisted/ internet/ defer.py" , line 1037, in _inlineCallbacks throwExceptionI ntoGenerator( g) python2. 7/dist- packages/ twisted/ python/ failure. py", line 382, in throwExceptionI ntoGenerator python2. 7/dist- packages/ juju/providers/ ec2/launch. py", line 114, in _ensure_groups group, e.get_error_ messages( ))...
2012-06-26 04:08:42,305: juju.ec2@DEBUG: Creating juju machine security group juju-devstack-7
2012-06-26 04:08:42,531: juju.ec2@DEBUG: Launching with machine spec MachineSpec(
2012-06-26 04:08:43,289: juju.state.
2012-06-26 04:08:43,421: juju.state.
2012-06-26 04:09:13,687: juju.agents.
2012-06-26 04:09:13,707: juju.ec2@WARNING: ssl-hostname-
2012-06-26 04:09:13,707: juju.ec2@WARNING: EC2 API calls not using secure transport
2012-06-26 04:09:13,708: juju.ec2@WARNING: S3 API calls not using secure transport
2012-06-26 04:09:13,708: juju.ec2@WARNING: Ubuntu Cloud Image lookups encrypted but not authenticated
2012-06-26 04:09:14,063: juju.ec2@DEBUG: Cannot delete security group juju-devstack-7: Error Message: An unknown error has occurred. Please try your request again.
2012-06-26 04:09:14,065: juju.agents.
Traceback (most recent call last):
File "/usr/lib/
machine_
File "/usr/lib/
result = result.
File "/usr/lib/
return g.throw(self.type, self.value, self.tb)
File "/usr/lib/
{"machine-id": machine_state.id, "constraints": constraints})
File "/usr/lib/
result = result.
File "/usr/lib/
return g.throw(self.type, self.value, self.tb)
File "/usr/lib/
machines = yield self.start_
File "/usr/lib/
result = result.
File "/usr/lib/
return g.throw(self.type, self.value, self.tb)
File "/usr/lib/
security_groups = yield self._ensure_
File "/usr/lib/
result = result.
File "/usr/lib/
return g.throw(self.type, self.value, self.tb)
File "/usr/lib/
% (juju_machine_