EC2 metadata retrieval fails with spaces in a resource name

Bug #962046 reported by Juerg Haefliger
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init
Invalid
Low
Unassigned
python-boto (Ubuntu)
Fix Released
Low
Unassigned
Precise
Fix Released
Medium
Edward Hope-Morley
Quantal
Fix Released
Medium
Edward Hope-Morley
Raring
Fix Released
Medium
Edward Hope-Morley

Bug Description

[Impact]

Example when using a key name 'my key':

Caught exception reading instance data: http://169.254.169.254/2009-04-04/meta-data/mpi/my key

Is a boto issue, see https://github.com/boto/boto/issues/659

Patches provided for Precise, Quantal and Raring. Fix already exists in Saucy.

[Test Case]

There are two ways to test this. The first provides that the change will affect
the url in the required manner but does not test the package. The second will
test the package but needs to be run within an Openstack instance against an
API that has calls with spaces on their name.

====
#!/usr/bin/env python
import urllib

resource = "http://169.254.169.254/2009-04-04/meta-data/i love spaces"
print "unsafe: %s" % resource

resource = urllib.quote(resource, safe="/:")
print "safe: %s" % resource

===
#!/usr/bin/env python
from boto import utils
utils.get_instance_metadata()

[Regression Potential]

This is a very low risk change which makes any non-alphnumeric characters in a meta api call name safe for urlib. It is unlikely that an api call name would have anything other than alphanumeric chars, whitespace and perhaps '_' but even if they did they would be safe here so there is no known regression potential.

Dave Walker (davewalker)
Changed in python-boto (Ubuntu):
importance: Undecided → Low
Revision history for this message
Scott Moser (smoser) wrote :

Could you give an example of how you start an instance such that '/meta-data/mpi/my key' exists?
I just tried this with a keypair (euca-import-keypair --public-key-file my.pub "my key", and then run instance with '--key "my key"'), and that worked fine (thi sis actually because of the way that public-keys are displayed and handled specially in the MD).

Anyway, can you explain how you did this so I can reproduce?

Changed in cloud-init:
status: New → Triaged
Changed in python-boto (Ubuntu):
status: New → Triaged
Changed in cloud-init:
importance: Undecided → Low
Revision history for this message
Juerg Haefliger (juergh) wrote :

nova boot --flavor <FLAVOR_ID> --image <IMAGE_ID> --key_name 'my key' <INSTANCE_NAME>

Revision history for this message
Scott Moser (smoser) wrote :

Ah.
I didn't realize that this was when the metadata service was coming from nova.
So, then arguably this is a bug in the metadata service on nova, as I could not reproduce against the MD on EC2.

Revision history for this message
Juerg Haefliger (juergh) wrote :

What does the MD look like on EC2? Curious, can you send a dump? Yes it seems to be a problem with the nova MD service in that it's different from EC2 but I still think boto should be able to handle special characters in MD URLs or is there an EC2 spec that this will never happen with EC2?

Revision history for this message
Scott Moser (smoser) wrote :

Hm.. Its interesting. digging further, I don't understand how this caused a problem on nova. It seems (either by design or happenstance) that the general behavior of the metadata service handles this issue, and boto is designed to read the metadata service as it is implemented.

I actually verified functional 'boto.utils.get_instance_metadata' on both nova and ec2 with a key named "my brickies" (nova folsom as packaged in Ubuntu). The output of the commands below was identical.

Example:
$ wget http://169.254.169.254/2009-04-04/meta-data/public-keys -O - -q; echo
0=my brickies
$ wget http://169.254.169.254/2009-04-04/meta-data/public-keys/0/ -O - -q; echo
openssh-key
$ wget http://169.254.169.254/2009-04-04/meta-data/public-keys/0/openssh-key -O - -q;
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA3I7VUf2l5gSn5uavROsc5HRDpZdQueUq5ozemNSj8T7enqKHOEaFoU2VoPgGEWC9RyzSQVeyD6s7APMcE82EtmW4skVEgEGSbDc1pvxzxtchBj78hJP6Cf5TCMFSXw+Fz5rF1dR23QDbN1mkHs7adr8GW4kSWqU7Q7NDwfIrJJtO7Hi42GyXtvEONHbiRPOe8stqUly7MvUoN+5kfjBM8Qqpfl2+FNhTYWpMfYdPUnE7u536WqzFmsaqJctz3gBxH9Ex7dFtrxR4qiqEr9Qtlu3xGn7Bw07/+i1D+ey3ONkZLN+LQ714cgj8fRS4Hj29SCmXp5Kt5/82cD/VN3NtHw== my brickies

So, the public-keys (also in nova) is special cased, and you do not ever refer to the key name when crawling. So the space shouldn't actually be used.

I agree the general suggestion, though, that boto should not be doing requests with unencoded urls.

Revision history for this message
Scott Moser (smoser) wrote :

I'm also still confused at the initial report, you have "mpi". On the nova I'm looking at:
$ wget http://169.254.169.254/2009-04-04/ -O - -q ; echo
meta-data/
user-data
$ wget http://169.254.169.254/2009-04-04/meta-data/ -O - -q; echo
reservation-id
public-keys/
security-groups
public-ipv4
ami-manifest-path
instance-type
instance-id
local-ipv4
local-hostname
placement/
ami-launch-index
public-hostname
hostname
ami-id
instance-action
block-device-mapping/

$ wget http://169.254.169.254/2009-04-04/meta-data/mpi -O -
--2012-06-04 13:41:23-- http://169.254.169.254/2009-04-04/meta-data/mpi
Connecting to 169.254.169.254:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2012-06-04 13:41:23 ERROR 404: Not Found.

Attemping to import a 'my brickies' keypair name via either web ui or command line client on HP cloud gives:
$ nova keypair-add "my brickies" --pub_key ~/.<email address hidden>
ERROR: Invalid keypair name (HTTP 400)

Revision history for this message
Juerg Haefliger (juergh) wrote :

Hmm... On the hp cloud I get:

ubuntu@juergh-8419-5531:~$ curl 169.254.169.254/2009-04-04/meta-data
local-ipv4
reservation-id
local-hostname
placement/
ami-launch-index
public-hostname
mpi/
hostname
ami-id
public-keys/
instance-action
public-ipv4
block-device-mapping/
ami-manifest-path
security-groups
instance-type
instance-id

We 'fixed' the issue by not allowing spaces in key names anymore.

Revision history for this message
Juerg Haefliger (juergh) wrote :

mpi/ was present in diablo but removed subsequently:

commit 8ecdc44690ced882205112e017f79dc98cd6aaca
Author: Jesse Andrews <email address hidden>
Date: Tue Mar 6 20:49:16 2012 -0800

    remove undocumented, unused mpi 'extension' to ec2 metadata

Revision history for this message
Scott Moser (smoser) wrote :

This is fixed in upstream boto at https://github.com/boto/boto/commit/fa10336d5db5f6c0f4e953fce21bd3c53234c8f5
So any release of boto after 2012-06-15 should have it.

Revision history for this message
Scott Moser (smoser) wrote :

Marking 'invalid' on cloud-init, because it will be appropriately fixed in boto.

Changed in cloud-init:
status: Triaged → Invalid
Chris J Arges (arges)
Changed in python-boto (Ubuntu Precise):
assignee: nobody → Chris J Arges (arges)
Changed in python-boto (Ubuntu Quantal):
assignee: nobody → Chris J Arges (arges)
Changed in python-boto (Ubuntu Precise):
importance: Undecided → Medium
Changed in python-boto (Ubuntu Quantal):
importance: Undecided → Medium
Changed in python-boto (Ubuntu Precise):
status: New → In Progress
Changed in python-boto (Ubuntu Quantal):
status: New → In Progress
Revision history for this message
Chris J Arges (arges) wrote :

Here are some packages with the backported fix:
http://people.canonical.com/~arges/lp962046/

Please test and verify if this works.

Revision history for this message
Juerg Haefliger (juergh) wrote :

Sorry this took so long. Finally found the time to test the 12.04 package and it works. Thanks.

Revision history for this message
Edward Hope-Morley (hopem) wrote :
Revision history for this message
Edward Hope-Morley (hopem) wrote :
Revision history for this message
Edward Hope-Morley (hopem) wrote :

[Impact]

 This is a very low risk change. It has been tested on Precise. A quantal patch is also provided.

[Test Case]

#!/usr/bin/env python
from boto import utils
utils.get_instance_metadata()

[Regression Potential]

 * not likely.

description: updated
Revision history for this message
Edward Hope-Morley (hopem) wrote :
Revision history for this message
Edward Hope-Morley (hopem) wrote :

updated SRU testcase and added debdiff for Raring

description: updated
Iain Lane (laney)
Changed in python-boto (Ubuntu Raring):
status: New → In Progress
assignee: nobody → Chris J Arges (arges)
Iain Lane (laney)
Changed in python-boto (Ubuntu):
status: Triaged → Fix Released
description: updated
description: updated
Changed in python-boto (Ubuntu Raring):
importance: Undecided → Medium
Changed in python-boto (Ubuntu Precise):
assignee: Chris J Arges (arges) → Edward Hope-Morley (hopem)
Changed in python-boto (Ubuntu Quantal):
assignee: Chris J Arges (arges) → Edward Hope-Morley (hopem)
Changed in python-boto (Ubuntu Raring):
assignee: Chris J Arges (arges) → Edward Hope-Morley (hopem)
Revision history for this message
Martin Pitt (pitti) wrote :

This was uploaded and is sitting in the SRU review queues, unsubscribing sponsors.

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Juerg, or anyone else affected,

Accepted python-boto into raring-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/python-boto/2.3.0-1ubuntu0.13.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in python-boto (Ubuntu Raring):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Juerg, or anyone else affected,

Accepted python-boto into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/python-boto/2.3.0-1ubuntu0.12.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in python-boto (Ubuntu Quantal):
status: In Progress → Fix Committed
Changed in python-boto (Ubuntu Precise):
status: In Progress → Fix Committed
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Juerg, or anyone else affected,

Accepted python-boto into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/python-boto/2.2.2-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote : [python-boto/precise] verification still needed

The fix for this bug has been awaiting testing feedback in the -proposed repository for precise for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

tags: added: removal-candidate
Revision history for this message
Scott Moser (smoser) wrote :

I've marked this verification done for Juerg, per an email on the subject.
Juerg originally reported, and works on the team at HP where this bug was found.

As the problem was/is well understood and the fix both accepted upstream boto and minimally invasive, hopefully this report can be accepted as 'verification-done'.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Chris Halse Rogers (raof) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-boto - 2.3.0-1ubuntu0.13.04.1

---------------
python-boto (2.3.0-1ubuntu0.13.04.1) raring; urgency=low

  * Fix failures when there are spaces in a resource name. (LP: #962046)
 -- Chris J Arges <email address hidden> Fri, 08 Feb 2013 09:50:51 -0600

Changed in python-boto (Ubuntu Raring):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-boto - 2.3.0-1ubuntu0.12.10.1

---------------
python-boto (2.3.0-1ubuntu0.12.10.1) quantal; urgency=low

  * Fix failures when there are spaces in a resource name. (LP: #962046)
 -- Chris J Arges <email address hidden> Fri, 08 Feb 2013 09:50:51 -0600

Changed in python-boto (Ubuntu Quantal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-boto - 2.2.2-0ubuntu3

---------------
python-boto (2.2.2-0ubuntu3) precise; urgency=low

  * Fix failures when there are spaces in a resource name. (LP: #962046)
 -- Chris J Arges <email address hidden> Fri, 08 Feb 2013 09:23:57 -0600

Changed in python-boto (Ubuntu Precise):
status: Fix Committed → Fix Released
Revision history for this message
James Falcon (falcojr) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.