[SRU] not able to upload binary files when booting a vm

For me the description in the bug is confusing as you can encode every binary file into UTF-8 as it is 8 bit clean.

The error message I get is

ERROR: 'ascii' codec can't decode byte 0xdc in position 24: ordinal not in range(128)

Therefore for me it sounds as if the code tries to transcode the file from ASCII to UTF-8. As ASCII only allows 7 bits every byte with the high bit set cannot be interpreted. You get the same effect when trying this:

$ recode ascii..utf8 < /bin/ping > /dev/null
recode: Invalid input in step `ANSI_X3.4-1968..UTF-8'

Also you cannot interpret binary files as UTF-8 as not all byte combinations are valid:

$ recode utf8..iso8859-1 < /bin/ping > /dev/null
recode: Invalid input in step `UTF-8..ISO-8859-1'

IMHO the root problem is the interpretation of a file as some binary string encoding. If there is a need to interpret is as a string, I suggest to use an 8 bit clean charset like ISO-8859-1:

$ recode iso8859-1..utf8 < /bin/ping > /dev/null

works fine. Certainly this would break multi-byte UTF-8 characters in text files iff they are not to be copied somewhere but interpreted as text. But this is the fundamental problem: binary files should be treated as such and not interpreted as text while text files should be treated with the encoding - e.g. UTF-8 - configured for the platform.

You cannot treat every binary file as utf-8 stream and convert it to plain string using ascii encoding.

You can simply try,
>>> f = open('/bin/ping')
>>> data = f.read()
>>> data.encode('utf-8')

and it will give you below error message,

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xa9 in position 24: ordinal not in range(128)

In fact, this is what the current code is doing - https://github.com/openstack/python-novaclient/blob/master/novaclient/v1_1/servers.py#L509 . We cannot use other char set encoding like ISO-8859-1, because that's not what the server is expecting. The reason we need a text representation of a binary file is also due to the signature of the nova server api - POST /v2/​{tenant_id}​/servers. Again that's what the server expects. I believe there were justification when they do the design desicsion. But now we are just dealing with a problem on the client side. Let's make the change mininal and best fit into the existing infrastructure.

I am thinking that just changing that line to "cont = base64.b64encode(data).decode('utf-8')" will do the work - directly using base64 encoding which was also the original way of doing the encoding before commit 8b264fc61d21fe4d0c7405914fb084f898956888 and that worked for you as well. I will submit a patch for the change after having some tests.

Fix proposed to branch: master
Review: https://review.openstack.org/145554

I can confirm, that this change works. For Icehouse I've used

--- servers.py.orig 2015-01-08 10:17:20.970389398 +0000
+++ servers.py 2015-01-08 10:17:37.053987841 +0000
@@ -492,7 +492,7 @@
                     data = file_or_string
                 personality.append({
                     'path': filepath,
- 'contents': base64.b64encode(data.encode('utf-8')),
+ 'contents': base64.b64encode(data).encode('utf-8'),
                 })

         if availability_zone:

sorry, correction: the VM boots, but the file does not appear with my change

Thanks for the testing. Can you please check that with the change in place whether injecting a text file works or not? Thanks.

There might be other issues with your deployment. We can discuss about it in a separate thread. I have tested the patch and it works for me.

Sorry for the confusion - you were right, I was missing in my compute's nova.conf, so injection did not work at all:

[libvirt]
inject_partition=1
inject_key=True

I've tested it with a binary and a text file (passwd and it's gzipped version) and it works without problems. So thanks for the quick fix!

np. Good to know that the problem is solved:)

Reviewed: https://review.openstack.org/145554
Committed: https://git.openstack.org/cgit/openstack/python-novaclient/commit/?id=f75ea86a2a4ee0bed076ee9e4ea5021eb0643e16
Submitter: Jenkins
Branch: master

commit f75ea86a2a4ee0bed076ee9e4ea5021eb0643e16
Author: Liang Chen <email address hidden>
Date: Wed Jan 7 13:23:10 2015 -0500

    Directly using base64 encoding for injected files

    Binary files cannot be treated as utf-8 byte streams and converted
    to plain. Change this back to just using base64 encoding as it was
    before commit 8b264fc61d21fe4d0c7405914fb084f898956888.

    Change-Id: I4ef6142676022b2e2f3178e7bfa24ed985fcae2c
    Closes-Bug: #1408088

The attachment "python-novaclient_2.17.0-lp1408088.patch" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

patch for ubuntu archive

Hello Liang, or anyone else affected,

Accepted python-novaclient into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-novaclient/1:2.17.0-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hi Chris,

The version 1:2.17.0-0ubuntu1.1 is tested, and it fixes the bug.

Thanks,
Liang

This bug was fixed in the package python-novaclient - 1:2.17.0-0ubuntu1.1

---------------
python-novaclient (1:2.17.0-0ubuntu1.1) trusty-proposed; urgency=medium

  [ Liang Chen ]
  * debian/patches/0001-Directly-using-base64-encoding-for-injected-files.patch:
    Make sure binary files can be uploaded when booting a VM. (LP: #1408088)
 -- Chuck Short <email address hidden> Thu, 09 Apr 2015 11:21:06 -0400

The verification of the Stable Release Update for python-novaclient has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.