nova returns wrong date format in AWS API Calls

I believe this is a matter of implementation. In nova/api/ec2/apirequest.py:166, we _render_data according to the data class:

if isinstance(data, list):
            for item in data:
                data_el.appendChild(self._render_data(xml, 'item', item))
        elif isinstance(data, dict):
            self._render_dict(xml, data_el, data)
        elif hasattr(data, '__dict__'):
            self._render_dict(xml, data_el, data.__dict__)
        elif isinstance(data, bool):
            data_el.appendChild(xml.createTextNode(str(data).lower()))
        elif data != None:
            data_el.appendChild(xml.createTextNode(str(data)))

However, we never actually check to see if the data is a datetime instance (which is what comes out of our database). By adding these two lines (before the final elif date != None):

        elif isinstance(data, datetime.datetime):
            data_el.appendChild(xml.createTextNode(data.isoformat()))

I believe we will catch _all_ the dates (not sure if there are any others in the responses) and convert them to ISO format.

We need to add this type of parsing to our integration tests.

bobya -- can you pull this branch (https://code.launchpad.net/~ken-pepple/nova/lp721297) and test against your tool ?

sorry for the verbosity (this was hard to test) ... but I now see iso compliant response in my testing:

2011-02-19 02:17:20,951 DEBUG nova.api.request [-] <?xml version="1.0" ?><DescribeInstancesResponse xmlns="http://ec2.amazonaws.com/doc/2009-11-30/"><requestId>N-L40NGYDKWV2G3NG-KD</requestId><reservationSet><item><ownerId>mtbcal</ownerId><groupSet/><reservationId>r-gle2kkul</reservationId><instancesSet><item><displayDescription/><displayName/><keyName>mykey (mtbcal, None)</keyName><instanceId>i-00000004</instanceId><instanceState><code/><name>scheduling</name></instanceState><publicDnsName/><imageId>ami-5b5mum3v</imageId><productCodesSet/><privateDnsName/><dnsName/><launchTime>2011-02-19T10:13:42</launchTime><placement><availabilityZone>unknown zone</availabilityZone></placement><amiLaunchIndex>0</amiLaunchIndex><instanceType>m1.small</instanceType></item></instancesSet></item><item><ownerId>mtbcal</ownerId><groupSet><item><groupId>default</groupId></item></groupSet><reservationId>r-2ofpbpqz</reservationId><instancesSet><item><displayDescription/><displayName>Server 3</displayName><keyName>mykey (mtbcal, openstack)</keyName><instanceId>i-00000003</instanceId><instanceState><code>1</code><name>running</name></instanceState><publicDnsName/><imageId>ami-5b5mum3v</imageId><productCodesSet/><privateDnsName>172.16.0.3</privateDnsName><dnsName>172.16.0.3</dnsName><launchTime>2011-02-19T10:02:45</launchTime><placement><availabilityZone>nova</availabilityZone></placement><amiLaunchIndex>0</amiLaunchIndex><instanceType>m1.small</instanceType></item></instancesSet></item></reservationSet></DescribeInstancesResponse> from MainProcess (pid=24085) _render_response /home/kpepple/src/nova/lp721297/nova/api/ec2/apirequest.py:150

Sorry for the delay I will do the testing today and will reply as soon as we are done. Thanks for the fast reply

Hi Ken,
we have tested the fix today but found that python uses the old ISO8601 format that doesn't contain the 'Z' symbol at the end. Unfortunately the parsers in the Amazon java SDK use the new ISO8601 UTC format more information is available here:
http://en.wikipedia.org/wiki/ISO_8601
in the UTC section.
We couldn't find a way to make python write the 'Z' symbol at the end of a date format. I don't know if you can help with this, but this is the way the Amazon API is written and it will break compatibility with it if it is not fixed.
The current fix does really convert the date to ISO format , but in "Local Time" not in UTC.

Ken, this should work: datetime.datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ")

>>> print datetime.datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ")
2011-02-21T16:12:55Z

bobya, jaypipes:

So I left the 'z'ed off the ISO date on purpose ... although it appears I *may* have been misguided. My rationale:

     when we create the instance record, we create a record with created_at field
     filled with a date like "2011-02-21 19:01:45.653573" (sqlite) which should be
     localtime not utctime (which would have the "z" suffix).

However, now that I look at it, we are logging utctime just not calling it that.

Let me investigate whether this is DB dependent or not and then I'll update the bug and re-propose the merge. If we are logging utctime everywhere, it's a trivial change -- i should have something this afternoon PST.

bobya --

For the record, i maintain this is a bug in Amazon's implementation, as they define the launchTime as xs:dateTime (http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/index.html?ApiReference-ItemType-RunningInstancesItemType.html) and returning a localtime value (sans 'z'ed) is valid by my reading of http://www.w3.org/TR/xmlschema-2/#dateTime

:)

Ok, with the fix I just pushed (#705), I now get:

2011-02-21 14:20:25,790 DEBUG nova.api.request [-] <?xml version="1.0" ?><RunInstancesResponse xmlns="http://ec2.amazonaws.com/doc/2009-11-30/"><requestId>IDGIWP-7CJ9292AFG6RP</requestId><ownerId>mtbcal</ownerId><groupSet><item><groupId>default</groupId></item></groupSet><reservationId>r-xgnisy5c</reservationId><instancesSet><item><displayDescription/><displayName>Server 6</displayName><keyName>pk.pem (mtbcal, None)</keyName><instanceId>i-00000006</instanceId><instanceState><code/><name>scheduling</name></instanceState><publicDnsName/><imageId>ami-5b5mum3v</imageId><productCodesSet/><privateDnsName/><dnsName/><launchTime>2011-02-21T22:20:25Z</launchTime><placement><availabilityZone>unknown zone</availabilityZone></placement><amiLaunchIndex>0</amiLaunchIndex><instanceType>m1.small</instanceType></item></instancesSet></RunInstancesResponse> from MainProcess (pid=30279) _render_response /home/kpepple/src/nova/lp721297/nova/api/ec2/apirequest.py:155

when i do a euca-run-instance. I believe "<launchTime>2011-02-21T22:20:25Z</launchTime>" is the format you need.

I've also added a small test to prevent regressions on this.

Can you test this (need to $lp merge lp:~ken-pepple/nova/lp721297) and report back ?
If it doesn't work, can you write a simple test using that SDK that I can test against ?

Thanks Ken,
It is working fine now. I agree to you that this is also a problem with the amazon implementation, which doesn't fully support xs:DateTime.
I will see if I can report some bug to their implementation.

Do not set to FixCommitted until the fix is merged into trunk.

ttx, it's been committed twice (#713, #703) :)

Got lost by branch link, sorry about that