UnicodeDecodeError with glade files in pbuilder

Bug #1017468 reported by marmuta
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
python-distutils-extra
Fix Released
Undecided
Unassigned
python-distutils-extra (Ubuntu)
Fix Released
Undecided
Martin Pitt
Precise
Fix Released
Undecided
Unassigned

Bug Description

[Test case]
Install the version from -proposed
1. have a package with an ui file with some UTF-8 characters using p-d-e
2. run pdebuild
-> no crash should occurs

[Regression potential]
Unlucky as the test suite covers other .ui file case

Hi, we're currently switching Onboard to Python 3. Building with python3-distutils-extra works fine from the terminal. However it fails in pbuilder:

pdebuild
...
running build_i18n
Traceback (most recent call last):
  File "setup.py", line 157, in <module>
    cmdclass = {'test': TestCommand},
  File "/usr/lib/python3/dist-packages/DistUtilsExtra/auto.py", line 100, in setup
    distutils.core.setup(**attrs)
  File "/usr/lib/python3.2/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.2/distutils/dist.py", line 917, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.2/distutils/dist.py", line 936, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.2/distutils/command/build.py", line 126, in run
    self.run_command(cmd_name)
  File "/usr/lib/python3.2/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.2/distutils/dist.py", line 936, in run_command
    cmd_obj.run()
  File "/usr/lib/python3/dist-packages/DistUtilsExtra/auto.py", line 607, in run
    contents = open(f).read()
  File "/usr/lib/python3.2/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 31523: ordinal not in range(128)

What seems to be happening is that pbuilder sets LANG=C, which causes Python 3 to assume a default encoding of 'ANSI_X3.4-1968' (ASCII). Our glade files are UTF-8 encoded, so read() is bound to fail.

As a workaround I've added this to Onboard's setup.py for now:
locale.getpreferredencoding = lambda: 'UTF-8'
With this the build runs through.

Onboard trunk
python3-distutils-extra 2.33-1
python3 3.2.3-0ubuntu1
Ubuntu 12.10, quantal

Revision history for this message
Martin Pitt (pitti) wrote :

Reproduced in test suite and fixed in trunk.

Changed in python-distutils-extra:
status: New → Fix Released
Changed in python-distutils-extra (Ubuntu):
status: New → Fix Committed
assignee: nobody → Martin Pitt (pitti)
description: updated
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-distutils-extra - 2.34-1

---------------
python-distutils-extra (2.34-1) unstable; urgency=low

  [ Martin Pitt ]
  * debian/rules: Run Python3 tests under C locale, to expose more unicode
    errors.
  * auto.py: Fix *.ui detection to be robust for non-ASCII files.
    (LP: #1017468)

  [ Didier Roche ]
  * auto.py: Fix a regression with python2 if files have some utf-8 directives
    (LP: #1021969)
 -- Martin Pitt <email address hidden> Mon, 09 Jul 2012 11:12:38 +0200

Changed in python-distutils-extra (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello marmuta, or anyone else affected,

Accepted python-distutils-extra into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/python-distutils-extra/2.34-0ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in python-distutils-extra (Ubuntu Precise):
status: New → Fix Committed
tags: added: verification-needed
Revision history for this message
marmuta (marmuta) wrote :

Precise:
Sorry, I can't really verify if this bug is fixed there, too many missing dependencies. Onboard builds for Python 3 only on Quantal.
The Python 2 build, using python-distutils-extra, still works with precise-proposed enabled in pbuilder.

Quantal:
Using python3-distutils-extra this bug is fixed for Onboard. Thanks for the quick response.

Unfortunately, without the above workaround, the build still fails in pbuilder. Different lines, but similar problems. I'll try to file bug reports later, but it boils down to any file read (auto.py:624) and print statement (auto.py:105 ) not being safe when run with LANG="C".

Revision history for this message
Sebastien Bacher (seb128) wrote :

seems the same error than bug #995653 "unicode error", the testcase provided there works fine with the current version

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (3.5 KiB)

This bug was fixed in the package python-distutils-extra - 2.34-0ubuntu0.1

---------------
python-distutils-extra (2.34-0ubuntu0.1) precise-proposed; urgency=low

  * Backport some fixes from latest release (and a regression fix):
    - debian/rules: Run Python3 tests under C locale, to expose more unicode
      errors.
    - auto.py: Fix *.ui detection to be robust for non-ASCII files.
      (LP: #1017468)
    - auto.py: Fix a regression with python2 if files have some utf-8 directives
      (LP: #1021969)

python-distutils-extra (2.34-1) unstable; urgency=low

  [ Martin Pitt ]
  * debian/rules: Run Python3 tests under C locale, to expose more unicode
    errors.
  * auto.py: Fix *.ui detection to be robust for non-ASCII files.
    (LP: #1017468)

  [ Didier Roche ]
  * auto.py: Fix a regression with python2 if files have some utf-8 directives
    (LP: #1021969)

python-distutils-extra (2.33-0ubuntu0.1) precise-proposed; urgency=low

  * Backport 2.33-1 to precise as an SRU (LP: #1020017):
    - auto.py: Fix crash when encountering binary files with Python 3.
      (LP: #995653)
    - auto.py: Fix crash when encountering an UTF-8 Python source code file.
    - debian/local/python-mkdebian cleanups: (LP: #706051)
      - Don't use {XS, XB}-Python-Version.
      - Generate an up to date Standards-Version.
      - Generate build dep on debhelper (>= 8) as debian/compat is set to '8'
        and debhelper overrides are used.
      - No need to generate a build dep on cdbs.
    - debian/local/python-mkdebian: For projects using --prefix, install
      binaries into a .../bin/ subdirectory instead of into the main project
      directory. This avoids potential name collisions with local Python
      modules. Thanks to Michael Terry for the patch!
    - debian/local/python-mkdebian: Create copyright format 1.0 compatible
      copyright files. Thanks Andrew Starr-Bochicchio! (LP: #1002379)
    - debian/copyright: Update to copyright 1.0 format.
    - debian/control: Bump Standards-Version to 3.9.3.
    - python-mkdebian: Fix wrong kde4 ui file detection regular expression.
      Thanks Angelo Compagnucci! (LP: #1002076)

python-distutils-extra (2.33-1) unstable; urgency=low

  * auto.py: Fix crash when encountering binary files with Python 3.
    (LP: #995653)
  * auto.py: Fix crash when encountering an UTF-8 Python source code file.

python-distutils-extra (2.32-5) unstable; urgency=low

  [ Andrew Starr-Bochicchio ]
  * debian/local/python-mkdebian cleanups: (LP: #706051)
    - Don't use {XS, XB}-Python-Version.
    - Generate an up to date Standards-Version.
    - Generate build dep on debhelper (>= 8) as debian/compat is set to '8'
      and debhelper overrides are used.
    - No need to generate a build dep on cdbs.

  [ Martin Pitt ]
  * debian/local/python-mkdebian: For projects using --prefix, install
    binaries into a .../bin/ subdirectory instead of into the main project
    directory. This avoids potential name collisions with local Python
    modules. Thanks to Michael Terry for the patch!

python-distutils-extra (2.32-4) unstable; urgency=low

  * debian/local/python-mkdebian: Create copyright format 1.0 compatible
    copyright fi...

Read more...

Changed in python-distutils-extra (Ubuntu Precise):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.