curtin

'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) (curtin can't handle non-ascii characters in dpkg --list output)

Bug #1370249 reported by Daniel Manrique on 2014-09-16

This bug affects 1 person

	Status	Importance	Assigned to
curtin	Fix Released	High	Blake Rouse
curtin (Ubuntu)	Fix Released	Undecided	Unassigned
Trusty	Fix Released	Undecided	Unassigned
Utopic	Fix Released	Undecided	Unassigned

Bug Description

I was trying to deploy an image based on an Ubuntu Desktop 14.04 install using MAAS 1.7.0~beta2+bzr2967-0ubuntu1~ppa1 on Ubuntu 14.04 server.

I made a "curtin-compatible" tar.gz which includes cloud-init, put it in place of root-tgz in my maas server, and tried to deploy a node (using juju bootstrap).

Juju ended up complaining and timing out, and the node never fully completed installation. On closer look, I found curtin crashes when running "curthooks", specifically at some point it does dpkg --list, and the code that handles this (utils.subp) assumes output from its commands is pure ascii and so crashes when the output contains non-ascii. For Ubuntu Desktop, in particular, the output contains "Déjȧ Dup", which is what causes things to barf.

I SSHd into the half-installed node and ran curtin manually like so:

sudo CURTIN_STACKTRACE=1 /curtin/bin/curtin --verbose --verbose --log-file /media/root-rw/curtin-install.log install --config=configs/config-000.cfg http://10.10.10.1/MAAS/static/images/ubuntu/amd64/generic/trusty/release/root-tgz

It ended in this:

Running command ['chroot', '/tmp/tmpQ_dk0_/target', 'dpkg-query', '--list'] with allowed return codes [0] (shell=False, capture=True)
Traceback (most recent call last):
  File "/curtin/curtin/commands/main.py", line 96, in main
    sys.exit(args.func(args))
  File "/curtin/curtin/commands/curthooks.py", line 443, in curthooks
    apply_debconf_selections(cfg, target)
  File "/curtin/curtin/commands/curthooks.py", line 219, in apply_debconf_selections
    pkgs_installed = get_installed_packages(target)
  File "/curtin/curtin/commands/curthooks.py", line 259, in get_installed_packages
    (out, _err) = util.subp(cmd, capture=True)
  File "/curtin/curtin/util.py", line 58, in subp
    out = out.decode()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128)
'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128)
Traceback (most recent call last):
  File "/curtin/curtin/commands/main.py", line 96, in main
    sys.exit(args.func(args))
  File "/curtin/curtin/commands/install.py", line 267, in cmd_install
    stage.run()
  File "/curtin/curtin/commands/install.py", line 104, in run
    util.subp(cmd, shell=shell, env=self.env)
  File "/curtin/curtin/util.py", line 68, in subp
    cmd=args)
ProcessExecutionError: Unexpected error while running command.
Command: ['curtin', 'curthooks']
Exit code: 3
Reason: -
Stdout: ''
Stderr: ''
Unexpected error while running command.
Command: ['curtin', 'curthooks']
Exit code: 3
Reason: -
Stdout: ''
Stderr: ''

I'm attaching the full log as well.

SRU curtin
==========

[Impact]

curtin can't handle non-ascii characters in dpkg --list output as mentioned in the original bug report descripition. This bug doesn't allow curtin to continue and terminates the installation.

This fix is needed so that we can make subprocess calls with output that contains non-ascii characters.

The patch supplied by the bug reporter enables non-ascii character subprocess output decoding for curtin.

[Test Case]

To reproduce the bug, follow the same steps as the bug report description mentions. Namely, making sure that dpkg --list will contain output with non-ascii characters.

[Regression Potential]

utf-8 is backwards compatible with ascii, so the likely hood of regressions is small.

See original description

Tags:

Related branches

lp:ubuntu/utopic-proposed/curtin

lp:~smoser/ubuntu/trusty/curtin/sru-utopic-level

lp:ubuntu/trusty-proposed/curtin

Revision history for this message

Daniel Manrique (roadmr) wrote on 2014-09-16:

curtin-console.log Edit (38.4 KiB, text/plain)

description:

updated

Revision history for this message

Daniel Manrique (roadmr) wrote on 2014-09-16:

curtin-subprocess-decode-utf8 Edit (539 bytes, text/plain)

The attached patch makes things work, but it may be too naive. I don't think explicitly expecting utf-8 is riskier than the old behavior (implicitly assuming ascii) but I'm not an expert with this codebase and may be missing something. Review appreciated :)

Revision history for this message

Blake Rouse (blake-rouse) wrote on 2014-09-16:

I think this patch is acceptable. If you could make a MP for this, I will get it merged.

Changed in curtin:
status:	New → Triaged
importance:	Undecided → High
assignee:	nobody → Blake Rouse (blake-rouse)

Revision history for this message

Daniel Manrique (roadmr) wrote on 2014-09-17:

A merge has been proposed, I also added a small test for this.

Blake Rouse (blake-rouse) on 2014-09-18

Changed in curtin:
status:	Triaged → Fix Committed

Scott Moser (smoser) on 2014-10-21

Changed in curtin (Ubuntu):
status:	New → Fix Released

Newell Jensen (newell-jensen) on 2014-10-24

description:

updated

Newell Jensen (newell-jensen) on 2014-10-24

description:

updated

Revision history for this message

Brian Murray (brian-murray) wrote on 2014-10-30: Please test proposed package

Hello Daniel, or anyone else affected,

Accepted curtin into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/curtin/0.1.0~bzr195-0ubuntu1~14.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in curtin (Ubuntu Trusty):
status:	New → Fix Committed
tags:	added: verification-needed

Larry Michel (lmic) on 2014-12-17

tags:

added: verification-done
removed: verification-needed

Revision history for this message

Larry Michel (lmic) wrote on 2014-12-17:

I verified this for trusty deployment with maas 1.7.0+bzr3299-0ubuntu3~trusty1 and curtin version was 0.1.0~bzr195-0ubuntu1~14.04.1~ppa0.

The steps to test this was to edit the root-tgz after uncompressing and extracting it, doing a chroot. I then did a sudo apt-get install deja-dup-backend-gvfs which also installed deja-dup and fonts-dejavu-core. The non-ascii characters were in description for deja-dup-backend-gvfs: Remote server support for Déjà Dup. After exiting, I then recreated the compressed tgz image from the modified files and copied the file to cached filename in /var/lib/maas/boot-resources/cache, and to root-tgz file in /var/lib/maas/boot-resources/current/ubuntu/amd64/generic/trusty/daily. Finally, I restarted tgt.

I then deployed trusty through maas which worked. The 2nd test was to do a juju bootstrap which also passed. To make sure that I was dealing with the right image, I verified that the deja-up packages were installed on the deployed node.

Revision history for this message

Brian Murray (brian-murray) wrote on 2015-01-05: Update Released

The verification of the Stable Release Update for curtin has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2015-01-05:

This bug was fixed in the package curtin - 0.1.0~bzr195-0ubuntu1~14.04.1

---------------
curtin (0.1.0~bzr195-0ubuntu1~14.04.1) trusty-proposed; urgency=medium

  * New upstream snapshot.
    - hardware enablement: ppc64 support (LP: #1386394)
    - hardware enablement: know kernel mapping for utopic (hwe-u = 3.16)
      (LP: #1386394)
    - feature: support installing disk images including windows. (LP: #1386394)
    - feature: support creating swap file (LP: #1386394)
    - feature: support reporting logs back to MAAS (LP: #1386394)
    - feature: enable logging of installation to /var/log/curtin/install.log
      (LP: #1378910)
    - bug fix: extract tar files with xattr support when available (LP: #1313550)
    - bug fix: fix broken use of os.path.join for curtin hooks (LP: #1328521)
    - bug fix: util.subp to decode command output as utf-8 (LP: #1370249).
    - bug fix: call update-grub to ensure that /boot/grub/grub.cfg is created
      (LP: #1373137)
    - bug fix: do not use '--acl' when extracting tar files (LP: #1382632)
      as it inadvertently writes default directory acls.
    - bug fix: invoke lsblk with '--output' rather than '--out' to avoid
      ambiguity in newer versions of lsblk (LP: #1386275)
    - internal: part2bd helper added in helpers/common
    - internal: helpers: inherit curtin_verbosity (make the helper tools
      verbose if curtin invoked with verbose flags)
-- Scott Moser <email address hidden> Mon, 27 Oct 2014 20:58:43 -0400