After a cluster network restart get EC2ResponseError: 403 Forbidden from nodes

Bug #467521 reported by migomez
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
eucalyptus (Ubuntu)
Fix Released
Low
Unassigned

Bug Description

OS: Ubuntu 9.10 (Final Release)

After restarting network in the cluster using /etc/init.d/network restart, nodes can not connect to cluster anymore.

When trying to connect from node to cluster the following error is reported:
root@ubuntu2:~# euca-describe-availability-zones verbose
Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
EC2ResponseError: 403 Forbidden
Failure: 403 Forbidden

Both cluster and node were rebooted but the error keeps happening, there is not way to communicate nodes to cluster anymore.

Revision history for this message
Mathias Gug (mathiaz) wrote :

How long did you wait for the Node Controllers to reconnect to the Cluster Controllers? Did you try to run the euca-describe-availability-zones multiples times?

You may see bug 444352.

Changed in eucalyptus (Ubuntu):
importance: Undecided → Low
status: New → Incomplete
Revision history for this message
migomez (migomez) wrote :

Both cluster and nodes have been rebooted several times, as well as trying euca-describe-availability-zones multiple times since the moment it happened.

Even after rebooting and restarting services we keep getting:
root@ubuntu2:~# euca-describe-availability-zones verbose
Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
EC2ResponseError: 403 Forbidden
Failure: 403 Forbidden

To make sure it was not an error we made in a configuration file, we installed both the cluster and the node in another server, made sure everything worked, and performed the network restart again, and again, we get the same problem when running commands from node to cluster (if we run the command inside the cluster there is not error).

Thanks

Revision history for this message
Mathias Gug (mathiaz) wrote : Re: [Bug 467521] Re: After a cluster network restart get EC2ResponseError: 403 Forbidden from nodes

On Mon, Nov 02, 2009 at 07:23:31PM -0000, migomez wrote:
>
> To make sure it was not an error we made in a configuration file, we
> installed both the cluster and the node in another server,

Are you running the cluster and the node on the same physical server?

> made sure
> everything worked,

How? Does an "euca-describe-availability-zones verbose" from the node
work? Were you able to register images and start instances?

> and performed the network restart again, and again,
> we get the same problem when running commands from node to cluster (if
> we run the command inside the cluster there is not error).

Is the network connectivity working correctly after the restart? Are the
nodes able to ping the CC?

Revision history for this message
migomez (migomez) wrote :

>Are you running the cluster and the node on the same physical server?
Cluster and nodes are two different servers.

>How? Does an "euca-describe-availability-zones verbose" from the node work? Were you able to register images and start instances?
Same error, any euca* command from the Node gets the same forbidden message.

>Is the network connectivity working correctly after the restart? Are the nodes able to ping the CC?
All the other services work ok, both servers are able to ping each other.

Before the /etc/init.d/network restart on the cluster everything work without any problem, please note, we installed everything in two other servers (as cluster and node) everything worked perfect between cluster and node, until the moment we run /etc/init.d/network restart on the cluster, at this moment we started getting the same forbidden problem again. This problem was replicated on our test environment.

Thanks,

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package eucalyptus - 1.6.2~bzr1136-0ubuntu3

---------------
eucalyptus (1.6.2~bzr1136-0ubuntu3) lucid; urgency=low

  * eucalyptus-cc.eucalyptus-cc-publication.upstart,
    eucalyptus-common.eucalyptus.upstart, eucalyptus-network.upstart,
    eucalyptus-sc.eucalyptus-sc-publication.upstart,
    eucalyptus-walrus.eucalyptus-walrus-publication.upstart, rules: rework
    our eucalyptus starting condition to depend on a new upstart emitted
    signal, eucalyptus-network-is-ready, which is only fired once the
    network interface providing the default route is in fact up, and
    listening on a real ip address, LP: #503180
  * debian/eucalyptus-common.eucalyptus.upstart:
    - don't respawn eucalyptus-cloud
    - ensure that the iptables module gets loaded soon enough, otherwise
      much bad behavior happens in various nasty ways, most notably, a
      wedged database which renders the CLC non responsive on restart/reboot,
      LP: #503180 and dupes, LP: #444352, #444946, #467521, #478573, #480048
 -- Dustin Kirkland <email address hidden> Tue, 02 Feb 2010 17:13:52 -0800

Changed in eucalyptus (Ubuntu):
status: Incomplete → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.