DescribeAvailabilityZones: services alive status is not consistent (if clocks are not synced)

This can happen if you have clock skew. Try installing ntp on all of your boxes so the clocks are in sync.

On Jan 27, 2011, at 11:24 AM, Tushar Patil wrote:

> Public bug reported:
>
> Tested on Revision No : 624
>
> If I run describe-availability-zones repeatedly for 4-5 times, each time
> it shows service status as different.
>
> I am using default "service_down_time" flag which is set to 60.
>
> root@ubuntu-network-api-server:/home/openstack/nova# euca-describe-availability-zones verbose
> AVAILABILITYZONE nova available
> AVAILABILITYZONE |- ubuntu-compute-01
> AVAILABILITYZONE | |- nova-compute enabled XXX 2011-01-27 19:19:59
> AVAILABILITYZONE |- ubuntu-compute-03
> AVAILABILITYZONE | |- nova-compute enabled XXX 2011-01-27 19:19:58
> AVAILABILITYZONE |- ubuntu-compute-02
> AVAILABILITYZONE | |- nova-compute enabled :-) 2011-01-27 19:19:54
> AVAILABILITYZONE |- ubuntu-network-api-server
> AVAILABILITYZONE | |- nova-scheduler enabled :-) 2011-01-27 19:19:50
> AVAILABILITYZONE | |- nova-network enabled :-) 2011-01-27 19:19:52
>
> root@ubuntu-network-api-server:/home/openstack/nova# euca-describe-availability-zones verbose
> AVAILABILITYZONE nova available
> AVAILABILITYZONE |- ubuntu-compute-01
> AVAILABILITYZONE | |- nova-compute enabled :-) 2011-01-27 19:24:00
> AVAILABILITYZONE |- ubuntu-compute-03
> AVAILABILITYZONE | |- nova-compute enabled XXX 2011-01-27 19:24:09
> AVAILABILITYZONE |- ubuntu-compute-02
> AVAILABILITYZONE | |- nova-compute enabled XXX 2011-01-27 19:24:05
> AVAILABILITYZONE |- ubuntu-network-api-server
> AVAILABILITYZONE | |- nova-scheduler enabled :-) 2011-01-27 19:24:01
> AVAILABILITYZONE | |- nova-network enabled :-) 2011-01-27 19:23:54
>
> ** Affects: nova
> Importance: Undecided
> Status: New
>
> --
> You received this bug notification because you are a member of Nova Bug
> Team, which is subscribed to OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/708822
>
> Title:
> DescribeAvailabilityZones: services alive status is not consistent
>
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> Tested on Revision No : 624
>
> If I run describe-availability-zones repeatedly for 4-5 times, each
> time it shows service status as different.
>
> I am using default "service_down_time" flag which is set to 60.
>
> root@ubuntu-network-api-server:/home/openstack/nova# euca-describe-availability-zones verbose
> AVAILABILITYZONE nova available
> AVAILABILITYZONE |- ubuntu-compute-01
> AVAILABILITYZONE | |- nova-compute enabled XXX 2011-01-27 19:19:59
> AVAILABILITYZONE |- ubuntu-compute-03
> AVAILABILITYZONE | |- nova-compute enabled XXX 2011-01-27 19:19:58
> AVAILABILITYZONE |- ubuntu-compute-02
> AVAILABILITYZONE | |- nova-compute enabled :-) 2011-01-27 19:19:54
> AVAILABILITYZONE |- ubuntu-network-api-server
> AVAILABILITYZONE | |- nova-scheduler enabled :-) 2011-01-27 19:19...

If the time are sync on all the nodes including api server, I don't think we need to find this issue.

In my previous comment, I meant to say we don't need to fix the issue

@tushar: any chance you could merge with trunk and resubmit fix for Cactus ?

Unassigning since Tushar seems to have gone AWOL, to give someone else a chance to work on this.
@Tushar: feel free to reassign yourself and propose again.

the fundamental problem here is that comparison is performed between time on machine where API server running vs time where SQL is running. We just experienced exactly the same issue with scheduler, where node was off a bit...

We either need to have a strict requirement to have clocks on all nodes synchronized or to implement a fix on sqlalchemy.api level - updated/created times wil be set and verified always on the same machine.

Is this bug still valid?

@Tom -- I don't think so. As long as the docs say you should have NTP deployed, which everyone should anyways, this bug is invalid.