OpenStack Compute (nova)

port 3333 (nova-objectstore) not available if running on hosts with nova-network service

Bug #727426 reported by Christian Berendt
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
Undecided
Unassigned

Bug Description

We're using on one testing installation the nova-objectstore on the same nodes like our nova-network and nova-compute services and also have one more node with nova-compute. This additional node can't access the nova-objectstore.

Adding a new iptables rule (iptables -I INPUT -p tcp --dport 3333 -j ACCEPT) solves the problem...

---snip---
2011-03-01 21:18:44,197 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib64/python2.6/site-packages/nova/exception.py", line 116, in _wrap
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt_conn.py", line 428, in spawn
(nova.exception): TRACE: self._create_image(instance, xml)
(nova.exception): TRACE: File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt_conn.py", line 708, in _create_image
(nova.exception): TRACE: project=project)
(nova.exception): TRACE: File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt_conn.py", line 553, in _cache_image
(nova.exception): TRACE: fn(target=base, *args, **kwargs)
(nova.exception): TRACE: File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt_conn.py", line 563, in _fetch_image
(nova.exception): TRACE: images.fetch(image_id, target, user, project)
(nova.exception): TRACE: File "/usr/lib64/python2.6/site-packages/nova/virt/images.py", line 51, in fetch
(nova.exception): TRACE: return f(image, path, user, project)
(nova.exception): TRACE: File "/usr/lib64/python2.6/site-packages/nova/virt/images.py", line 98, in _fetch_s3_image
(nova.exception): TRACE: return utils.execute(cmd_out)
(nova.exception): TRACE: File "/usr/lib64/python2.6/site-packages/nova/utils.py", line 151, in execute
(nova.exception): TRACE: cmd=cmd)
(nova.exception): TRACE: ProcessExecutionError: Unexpected error while running command.
(nova.exception): TRACE: Command: /usr/bin/curl --fail --silent http://10.68.32.148:3333/_images/ami-ezpl7q46/image -H 'Date: Tue, 01 Mar 2011 21:18:44 GMT' -H 'Authorization: AWS 3df6cde4-eac8-441b-90d9-e2a2260af940:sap:1avf/kcV6yIqnW71/jSsLFj4hPM=' -o /var/lib/nova/instances/_base/ami-ezpl7q46
(nova.exception): TRACE: Exit code: 22
(nova.exception): TRACE: Stdout: ''
(nova.exception): TRACE: Stderr: ''
(nova.exception): TRACE:
2011-03-01 21:18:44,200 ERROR nova.compute.manager [6OXVQW837J1LBWBAT18V sapadmin sap] instance 16: Failed to spawn
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 213, in run_instance
(nova.compute.manager): TRACE: self.driver.spawn(instance_ref)
(nova.compute.manager): TRACE: File "/usr/lib64/python2.6/site-packages/nova/exception.py", line 122, in _wrap
(nova.compute.manager): TRACE: raise Error(str(e))
(nova.compute.manager): TRACE: Error: Unexpected error while running command.
(nova.compute.manager): TRACE: Command: /usr/bin/curl --fail --silent http://10.68.32.148:3333/_images/ami-ezpl7q46/image -H 'Date: Tue, 01 Mar 2011 21:18:44 GMT' -H 'Authorization: AWS 3df6cde4-eac8-441b-90d9-e2a2260af940:sap:1avf/kcV6yIqnW71/jSsLFj4hPM=' -o /var/lib/nova/instances/_base/ami-ezpl7q46
(nova.compute.manager): TRACE: Exit code: 22
(nova.compute.manager): TRACE: Stdout: ''
(nova.compute.manager): TRACE: Stderr: ''
(nova.compute.manager): TRACE:
---snap---

Revision history for this message
Soren Hansen (soren) wrote : Re: [Bug 727426] [NEW] port 3333 (nova-objectstore) not available if running on hosts with nova-network service

2011/3/1 Christian Berendt <email address hidden>:
> Public bug reported:
>
> We're using on one testing installation the nova-objectstore on the same
> nodes like our nova-network and nova-compute services and also have one
> more node with nova-compute. This additional node can't access the nova-
> objectstore.
>
> Adding a new iptables rule (iptables -I INPUT -p tcp --dport 3333 -j
> ACCEPT) solves the problem...

Can you tell which rule is blocking it right now? AFAIR, we don't
touch INPUT at all, so my best guess is that you have a default policy
of DROP for INPUT. We don't touch the default policy, so maybe
something else is messing this up for you?

--
Soren Hansen        | http://linux2go.dk/
Ubuntu Developer    | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/

Thierry Carrez (ttx)
Changed in nova:
status: New → Incomplete
Revision history for this message
Christian Berendt (berendt) wrote :

I think this is only a problem on development systems. I'm running all nova services (including the objectstore) on one system and I have a second system running nova-compute and nova-network. Probably nova-network on the system with nova-objectstore doesn't care of the nova-objectstore. I'll test this when I fixed my development systems ;)

But I think in a productive environment you will a) not use nova-objectstore at all and b) if you use nova-objectstore you will not use it on the same systems as nova-compute/nova-network.

So I think we could close this ticket. Adding a check into nova-network to handle a running nova-objectstore is IMO not required.

Thierry Carrez (ttx)
Changed in nova:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.