Unicode decode error during upgrade to 14.04 if sources.list contains non-ascii characters and locale is non-US

Bug #1309447 reported by Mekk
28
This bug affects 5 people
Affects Status Importance Assigned to Milestone
apt-clone (Ubuntu)
Fix Released
High
Brian Murray
Trusty
Fix Released
High
Michael Vogt
ubuntu-release-upgrader (Ubuntu)
Fix Released
High
Unassigned
Trusty
Triaged
High
Unassigned

Bug Description

TEST CASE:
1. On Saucy or Precise add non-ascii characters to the file /etc/apt/sources.list like:

# Français accentué
deb http://archive.ubuntu.com/ubuntu saucy main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu saucy-updates main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu saucy-security main restricted universe multiverse

2. Upgrade with a non-us locale
LANG=fr_FR.utf8 do-release-upgrade

ACTUAL RESULT
This crash:
Traceback (most recent call last):
  File "/tmp/ubuntu-release-upgrader-ui8dvi/trusty", line 10, in <module>
    sys.exit(main())
  File "/tmp/ubuntu-release-upgrader-ui8dvi/DistUpgrade/DistUpgradeMain.py", line 240, in main
    save_system_state(logdir)
  File "/tmp/ubuntu-release-upgrader-ui8dvi/DistUpgrade/DistUpgradeMain.py", line 133, in save_system_state
    scrub_sources=True)
  File "/tmp/ubuntu-release-upgrader-ui8dvi/DistUpgrade/apt_clone.py", line 150, in save_state
    self._write_state_sources_list(tar, scrub_sources)
  File "/tmp/ubuntu-release-upgrader-ui8dvi/DistUpgrade/apt_clone.py", line 231, in _write_state_sources_list
    "./etc/apt/sources.list")
  File "/tmp/ubuntu-release-upgrader-ui8dvi/DistUpgrade/apt_clone.py", line 251, in _add_file_to_tar_with_password_check
    for line in f.readlines():
  File "/usr/lib/python2.7/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)

== Original report ==
do-release-upgrade fails as below. I am using Polish utf-8 locale, so I suspect something called reports error in Polish with non-ascii characters...

File "/tmp/ubuntu-release-upgrader-j2xvu5/trusty", line 10, in
<module>
sys.exit(main())

File
"/tmp/ubuntu-release-upgrader-j2xvu5/DistUpgrade/DistUpgradeMain.py",
line 240, in main
save_system_state(logdir)

File
"/tmp/ubuntu-release-upgrader-j2xvu5/DistUpgrade/DistUpgradeMain.py",
line 133, in save_system_state
scrub_sources=True)

File "/tmp/ubuntu-release-upgrader-j2xvu5/DistUpgrade/apt_clone.py",
line 150, in save_state
self._write_state_sources_list(tar, scrub_sources)

File "/tmp/ubuntu-release-upgrader-j2xvu5/DistUpgrade/apt_clone.py",
line 231, in _write_state_sources_list
"./etc/apt/sources.list")

File "/tmp/ubuntu-release-upgrader-j2xvu5/DistUpgrade/apt_clone.py",
line 251, in _add_file_to_tar_with_password_check
for line in f.readlines():

File "/usr/lib/python2.7/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position
2604: ordinal not in range(128)

ProblemType: Bug
DistroRelease: Ubuntu 13.10
Package: ubuntu-release-upgrader-core 1:0.205.5
ProcVersionSignature: Ubuntu 3.11.0-19.33~ppa1-generic-tuxonice 3.11.10.5
Uname: Linux 3.11.0-19-generic-tuxonice x86_64
NonfreeKernelModules: fglrx
ApportVersion: 2.12.5-0ubuntu2.2
Architecture: amd64
CrashDB: ubuntu
Date: Fri Apr 18 11:54:23 2014
DistributionChannelDescriptor:
 # This is a distribution channel descriptor
 # For more information see http://wiki.ubuntu.com/DistributionChannelDescriptor
 canonical-oem-somerville-oneiric-amd64-20111116-1
EcryptfsInUse: Yes
InstallationDate: Installed on 2012-05-07 (711 days ago)
InstallationMedia: Ubuntu 11.10 "Oneiric" - Build amd64 LIVE Binary 20111116-18:24
MarkForUpload: True
PackageArchitecture: all
SourcePackage: ubuntu-release-upgrader
Symptom: release-upgrade
UpgradeStatus: Upgraded to saucy on 2014-04-18 (0 days ago)

Related branches

Revision history for this message
Mekk (marcin-kasperski) wrote :
Revision history for this message
Jean-Baptiste Lallement (jibel) wrote :

Thanks for your report.

Could you open a terminal and run the command:
find /etc/apt

and paste the output in this bug report? Thanks.

Changed in ubuntu-release-upgrader (Ubuntu):
importance: Undecided → High
status: New → Incomplete
Revision history for this message
Mekk (marcin-kasperski) wrote :

I found the reason: I had polish character in comment in /etc/apt/sources.list. Just sth like:

# Wyłączone w trakcie upgrade do saucy
deb ....

(after I removed this comment, the error went away, now I get „can't calculate upgrade” but that's another story)

Revision history for this message
Jean-Baptiste Lallement (jibel) wrote :

I cannot reproduce by just adding this comment, could you restore this file as it was when this crash occurred and attach it to this bug report?

Revision history for this message
Jean-Baptiste Lallement (jibel) wrote :

I can reproduce if I set a non-US locale like fr_FR.utf8. Setting to triage.

Changed in ubuntu-release-upgrader (Ubuntu):
status: Incomplete → Triaged
Revision history for this message
Mekk (marcin-kasperski) wrote :

(the latter caused by mate and cinnamon installed from PPA)

OK, this bug is about unicodedecodeerror when non-ascii comment is present in /etc/apt/sources.list

description: updated
Changed in ubuntu-release-upgrader (Ubuntu):
milestone: none → trusty-updates
assignee: nobody → Michael Vogt (mvo)
summary: - Unicode decode error during upgrade to 14.04
+ Unicode decode error during upgrade to 14.04 if sources.list contains
+ non-ascii characters and locale is non-US
description: updated
Revision history for this message
Brian Murray (brian-murray) wrote :

The crash is actually in apt-clone.

affects: ubuntu-release-upgrader (Ubuntu) → apt-clone (Ubuntu)
Revision history for this message
Brian Murray (brian-murray) wrote :

Here's a patch I don't have 100% confidence in although it does work. I'm cautious because of the work done to fix bug https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1212025.

Revision history for this message
Mekk (marcin-kasperski) wrote :

From the comment which I see in this patch ("open in unicode mode..."), it looks like somebody intended to go towards handling file content as unicode. Your patch reverts it towards binary. Maybe it would be better to attempt input decoding (guessing the encoding somehow, at least considering utf-8 and locale-preferred encoding)? I don't remember what/how this exactly works in python3, so feel free to ignore this remark.

(sidenote: I am fairly curious what would happen if I saved some config files as utf-16 ;-))

tags: added: patch
Revision history for this message
Michael Vogt (mvo) wrote :

Thanks Jibel and Brian - I can only reproduce this if I do not have "language-pack-fr" installed on precise.
If that is the case, then the command output is:
$ python -c 'import sys;print(sys.getfilesystemencoding())'
ANSI_X3.4-1968

and that leads python to fail with the above stacktrace.

Revision history for this message
Michael Vogt (mvo) wrote :

I pushed a branch lp:~mvo/apt-clone/lp1309447 that contains a testcase, however, I can only make it work if:

http://bazaar.launchpad.net/~mvo/apt-clone/lp1309447/view/head:/tests/test_lp1309447.py#L11

the locale is set to something unsupported or non-utf8, i.e. if the sys.getfilesystemencoding() is not utf8.

So its worthwhile to try to find out if something is unsetting LANG or LC_ALL when update-manager is run.
If its a common problem we should probably also add a workaround like:
http://bazaar.launchpad.net/~mvo/software-properties/lp1069019/revision/901

Revision history for this message
Michael Vogt (mvo) wrote :

I pushed a fix for the issues I was able to reproduce here:
  lp:~mvo/ubuntu-release-upgrader/lp1309447
it will use C.UTF-8 when the LANG is unset or set to C to ensure that the release upgrader does not fall over on a utf8 sources.list file.

Changed in apt-clone (Ubuntu):
status: Triaged → In Progress
Revision history for this message
Коренберг Марк (socketpair) wrote :

Bug #1243795 - the same exactly ?

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Mekk, or anyone else affected,

Accepted ubuntu-release-upgrader into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:0.220.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in ubuntu-release-upgrader (Ubuntu Trusty):
milestone: none → trusty-updates
importance: Undecided → High
Changed in ubuntu-release-upgrader (Ubuntu):
importance: Undecided → High
Changed in ubuntu-release-upgrader (Ubuntu Trusty):
status: New → Triaged
Changed in ubuntu-release-upgrader (Ubuntu):
status: New → Triaged
Changed in ubuntu-release-upgrader (Ubuntu Trusty):
status: Triaged → Fix Committed
tags: added: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ubuntu-release-upgrader - 1:14.10.2

---------------
ubuntu-release-upgrader (1:14.10.2) utopic; urgency=low

  * debian/test/control:
    - add missing pep8 test dependency
  * tests/test_xorg_fix_intrepid.py:
    -
 -- Michael Vogt <email address hidden> Sat, 03 May 2014 08:37:51 +0200

Changed in ubuntu-release-upgrader (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
Mathew Hodson (mhodson) wrote :

Still crashes in Trusty with http://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:0.220.3

I used the test case in the bug description and ran the command
LANG=fr_FR.utf8 do-release-upgrade -d

tags: added: verification-failed
removed: verification-needed
Revision history for this message
Brian Murray (brian-murray) wrote :

To test the version of the release upgrader from the -proposed repository you need to use 'do-release-upgrade -p'.

tags: added: verification-needed
removed: verification-failed
Revision history for this message
Brian Murray (brian-murray) wrote :

I've test this myself using the test case from the bug description and 'do-release-upgrade -p' and it did in fact fail.

I noticed the following traceback in my log file:

2014-05-05 10:59:56,198 INFO Using config files '['./DistUpgrade.cfg']'
2014-05-05 10:59:56,198 INFO uname information: 'Linux clean-saucy-amd64 3.11.0-19-generic #33-Ubuntu SMP Tue Mar 11 18:48:34 UTC 2014 x86_64'
2014-05-05 10:59:56,198 INFO apt version: '0.9.9.1~ubuntu1'
2014-05-05 10:59:56,198 INFO python version: '2.7.5+ (default, Feb 27 2014, 19:37:08)
[GCC 4.8.1]'
2014-05-05 10:59:56,199 INFO release-upgrader version '0.220.3' started
2014-05-05 10:59:56,199 ERROR getting the encoding failed
Traceback (most recent call last):
  File "/tmp/ubuntu-release-upgrader-cyewkp/DistUpgrade/DistUpgradeView.py", line 40, in <module>
    locale.setlocale(locale.LC_ALL, "")
  File "/usr/lib/python2.7/locale.py", line 547, in setlocale
    return _setlocale(category, locale)
Error: unsupported locale setting
2014-05-05 10:59:56,200 INFO locale: 'None' 'None'

tags: added: verification-failed
removed: verification-needed
Revision history for this message
Mathew Hodson (mhodson) wrote :

How do you test in Trusty? When I run

do-release-upgrade -p

I get "No new release found"

Revision history for this message
Brian Murray (brian-murray) wrote :

@Matthew - sorry if you are running trusty then you just use 'do-release-upgrade -d' as the fix is in the main pocket for utopic. If you are using saucy then you need to use 'do-release-upgrade -p' to download the release upgrader from the trusty -proposed pocket.

Mathew Hodson (mhodson)
tags: added: trusty
christy (clg991)
Changed in ubuntu-release-upgrader (Ubuntu Trusty):
status: Fix Committed → Fix Released
Changed in ubuntu-release-upgrader (Ubuntu Trusty):
status: Fix Released → Fix Committed
Revision history for this message
Brian Murray (brian-murray) wrote :

I confirm my own comment #18 that this really did fail.

Revision history for this message
Michael Vogt (mvo) wrote :

Thanks Brian and Mathew and sorry for the trouble. The issue is not actually the release upgrader but the apt-clone file its including. The push a branch to lp:~mvo/apt-clone/lp1324833 that fixes the bug. The release-upgrader needs to be rebuild/uploaded with this fix (and a updated build-dependency to ensure that really the latest version with the fix of apt-clone is made part of the release upgrader tarball).

Revision history for this message
fccoelho (fccoelho) wrote :

I am still getting the same error even when I start the upgrader with `do-release-upgrade -p` from saucy:

File
"/tmp/ubuntu-release-upgrader-qwht26/DistUpgrade/sourceslist.py",
line 254, in str
line += " #" + self.comment

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
27: ordinal not in range(128)
=== Command detached from window (Wed Jul 2 08:56:01 2014) ===
=== Command terminated with exit status 1 (Wed Jul 2 08:56:01 2014) ===

Revision history for this message
fccoelho (fccoelho) wrote :

I should also add that I have no-non ascii characters in my sources.list. I get this error even when I delete all the sources.

Mathew Hodson (mhodson)
Changed in ubuntu-release-upgrader (Ubuntu Trusty):
status: Fix Committed → In Progress
Changed in apt-clone (Ubuntu):
assignee: Michael Vogt (mvo) → Brian Murray (brian-murray)
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apt-clone - 0.3.1~ubuntu12

---------------
apt-clone (0.3.1~ubuntu12) utopic; urgency=medium

  [ Brian Murray ]
  * Add an option to replace the mirror used in /etc/apt/sources.list with one
    of your choosing. (LP: #1290584)
  * debian/tests/control: add in missing make dependency

  [ Michael Vogt ]
  * Resolve unicode decode error for sources.list files if locale is
    unsupported. (LP: #1309447)
 -- Brian Murray <email address hidden> Wed, 20 Aug 2014 13:54:48 -0700

Changed in apt-clone (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote :

@Michael - Do you have any ideas on how we can test the apt-clone part of this fix? We need to get the fixed apt-clone into trusty for the release-upgrader to use it, if I understand things correctly.

Revision history for this message
Chris J Arges (arges) wrote :

Hello Mekk, or anyone else affected,

Accepted apt-clone into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/apt-clone/0.3.1~ubuntu11.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in apt-clone (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: removed: verification-failed
tags: added: verification-needed
Revision history for this message
Brian Murray (brian-murray) wrote :

I've tested the apt-clone part of this fix by running "LANG='' test_lp1309447.py" on a 14.04 system with the version of apt-clone from the release pocket and observed a traceback. I then installed the version of apt-clone from -proposed and ran the same test and did not receive a traceback.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apt-clone - 0.3.1~ubuntu11.1

---------------
apt-clone (0.3.1~ubuntu11.1) trusty-proposed; urgency=medium

  * Resolve unicode decode error for sources.list files if locale is
    unsupported. (LP: #1309447)
 -- Brian Murray <email address hidden> Wed, 20 Aug 2014 14:21:21 -0700

Changed in apt-clone (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for apt-clone has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Mathew Hodson (mhodson) wrote :

Is there anything left to do for the ubuntu-release-upgrader task of this bug?

Michael said in comment #22, "The release-upgrader needs to be rebuild/uploaded with this fix (and a updated build-dependency to ensure that really the latest version with the fix of apt-clone is made part of the release upgrader tarball)."

Revision history for this message
Brian Murray (brian-murray) wrote :

Yes, we would need to update the Trusty version of ubuntu-release-upgrader for people to get the newest version of apt-clone. However, it is only necessary for people upgrading from Saucy to Trusty. I went looking for failures like this for people upgrading from Saucy to Trusty in the Ubuntu Error Tracker and was unable to find any because we stopped accepting crash reports from Saucy. Subsequently, we do not really have an idea of how many people are still affected by this specific bug report and therefore whether or not it is worth updating ubuntu-release-upgrader.

Revision history for this message
Mathew Hodson (mhodson) wrote :

I added bug #1278280 as a duplicate, because I think it is the same issue it has the same symptoms and test case.

Revision history for this message
Mathew Hodson (mhodson) wrote :

Looking closer, the traceback on those bugs is different. It's a similar Unicode issue, but maybe not the exact same one. I made them duplicates of bug #1243795 instead.

Mathew Hodson (mhodson)
Changed in ubuntu-release-upgrader (Ubuntu Trusty):
status: In Progress → Confirmed
Mathew Hodson (mhodson)
Changed in apt-clone (Ubuntu):
milestone: trusty-updates → none
Mathew Hodson (mhodson)
Changed in ubuntu-release-upgrader (Ubuntu Trusty):
status: Confirmed → Triaged
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.