software-center crashed with UnicodeEncodeError in get_dbus_message(): 'ascii' codec can't encode character u'\xfc' in position 65: ordinal not in range(128)

Looking at all the duplicate reports, this happens with any language that uses non-ascii characters (those above ASCII 127) - potentially every language but english.

However this only affects a few software packages; I was able to correctly install vlc and flash plugin using a german-language system, but installing mouse trap (the one that controls the mouse with the webcam) fails and reproduces the error.

Steps to reproduce:

0 - I'm using a fully-updated Oneiric installation with software-center 4.1.22.
1 - Install a language that may contain non-ascii characters (the duplicates indicate this may happen in spanish (mexico), portuguese (portugal) and I managed to reproduce it in german (austria)):
   sudo apt-get install language-pack-de language-pack-gnome-de
2- Launch software center from a terminal, using this installed language:
  LANG=de_AT.UTF-8 LANGUAGE=de_AT:de software-center
3- Search for and try to install "mousetrap" (the mouse controlling app, not the game)

Expected result:
- App installs with no errors

Actual result:
- As described in the original report, software center crashes with the given trace.

It looks like somewhere, there is a call to the "unicode" python function, which defaults to using the ascii encoder, so any character in the string to be encoded that can't be encoded as 7-bit ascii will throw the exception.

We saw something similar in bug 833747 and the solution was to explicitly specify the encoder in the call to unicode:

unicode(string,"utf8") instead of unicode(string)

Of course, tracing the actual call to unicode was the tricky part as this is sometimes done implicitly. Also this may cause "secondary effects" and need code changes elsewhere (in bug 833747 we also required some changes to the text sent to a Gtk.TextBuffer to force encoding the unicode into utf8).

I'm setting this bug to triaged and importance: Critical as it has a "severe impact in a large portion of Ubuntu users" - Unreliability and crashes in Software Center, one of Ubuntu's flagship apps, for potentially anyone not on an English locale seems pretty critical to me.

I did a bit more looking into this problem. This gets triggered if a transaction error occurs while trying to install the package. This generates an exception, and it is handling the exception's translated message string that the crash occurs, due to using the ascii codec to encode a unicode that contains characters above code 127. Mousetrap triggers this because it tries to install gnome-mousetrap which has a dependency problem. So I guess the common denominator in packages that trigger this problem is that they all generate an error while installing, due to dependency problems or possibly other factors.

This is visible in the terminal from which software-center is run when trying to install mousetrap:

2011-09-12 12:47:27,097 - softwarecenter.backend - ERROR - error in _on_trans_finished 'Fehler: Die Paketabhängigkeiten können nicht aufgelöst werden
Dieser Fehler könnte durch weitere erforderliche Software-Pakete, die jedoch fehlen oder nicht installiert werden können, verursacht worden sein. Außerdem könnte es einen Konflikt zwischen Paketen geben, die nicht zur selben Zeit installiert sein dürfen.

Die folgenden Pakete haben nicht erfüllte Abhängigkeiten:

gnome-mousetrap: Depends: python (>= 2.6) aber 2.7.2-5ubuntu1 soll installiert werden
                 Depends: python-support (>= 0.90.0) aber 1.0.13ubuntu1 soll installiert werden

Now, I tried setting the default encoding for the system in /usr/lib/python2.7/sitecustomize.py by adding:

import sys
sys.setdefaultencoding('utf8')

If I do this, I still see the above error message, as the dependency problem still remains, but I do not get the UnicodeEncodeError exception that was happening in software-center.

i'm not able to install flash-plugin on a german system

The error appears on the client side. Hopefully these errors will vanish with a port from dbus to gdbus and python3.

In the meantime we could just workarround this by using the Exception.message attribute which is deprecated.

Should be fixed with 697

This bug was fixed in the package aptdaemon - 0.43+bzr697

---------------
aptdaemon (0.43+bzr697) oneiric; urgency=low

  * New bzr bugfix version:
    - Don't block on lintian calls to avoid time outs on the DBus
      by making use of the excellent preexec_fn of subprocess
      (LP: #812023)
    - pyflakes based import fixes, this fixes a unneeded glib import
      that can cause crashes with gtk2/gtk3 (pygtk/pygi)
    - use the deprecated message attribute to avoid encoding errors
      (LP: #846044)
 -- Michael Vogt <email address hidden> Mon, 19 Sep 2011 14:01:15 +0200

Michael, why has this gone from "fix released" to "in progress"? Is it still in progress for oneiric?

This is fix released, sorry. I set this to in progress before the upload, looks like something was slow or I clicked on it accidentally.

When installing xsane and "dependence nonsatisfaite" with Fax.
PC HP DV7 2220 SF

Ubuntu 12.10. This affect me.

Ubuntu 12.10

This bug is reoccuring in Ubuntu12.10 64bit Quantal. Unfortunally launchpad let us not open up a new report, so we must stay here for handling.

Thank you very much !
Le 25 août 2012 14:50, "melchiaros" <email address hidden> a écrit :

> This bug is reoccuring in Ubuntu12.10 64bit Quantal. Unfortunally
> launchpad let us not open up a new report, so we must stay here for
> handling.
>
> ** Tags added: quantal
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (847897).
> https://bugs.launchpad.net/bugs/846044
>
> Title:
> software-center crashed with UnicodeEncodeError in get_dbus_message():
> 'ascii' codec can't encode character u'\xfc' in position 65: ordinal
> not in range(128)
>
> Status in “aptdaemon” package in Ubuntu:
> Fix Released
> Status in “aptdaemon” source package in Oneiric:
> Fix Released
>
> Bug description:
> I've tried to install an program. After clicking on "install" and
> entered password, the software center crashes.
>
> ProblemType: Crash
> DistroRelease: Ubuntu 11.10
> Package: software-center 4.1.22
> ProcVersionSignature: Ubuntu 3.0.0-10.16-generic-pae 3.0.4
> Uname: Linux 3.0.0-10-generic-pae i686
> NonfreeKernelModules: fglrx
> ApportVersion: 1.22.1-0ubuntu2
> Architecture: i386
> Date: Fri Sep 9 23:31:38 2011
> ExecutablePath: /usr/share/software-center/software-center
> InstallationMedia: Ubuntu 11.10 "Oneiric Ocelot" - Beta i386 (20110901)
> InterpreterPath: /usr/bin/python2.7
> PackageArchitecture: all
> ProcCmdline: /usr/bin/python /usr/bin/software-center
> ProcEnviron:
> PATH=(custom, no user)
> LANG=de_DE.UTF-8
> SHELL=/bin/bash
> PythonArgs: ['/usr/bin/software-center']
> SourcePackage: software-center
> Title: software-center crashed with UnicodeEncodeError in
> get_dbus_message(): 'ascii' codec can't encode character u'\xfc' in
> position 65: ordinal not in range(128)
> UpgradeStatus: No upgrade log present (probably fresh install)
> UserGroups: adm admin cdrom dialout lpadmin plugdev sambashare
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/aptdaemon/+bug/846044/+subscriptions
>

error appears after use of this "sudo dpkg -i --force-depends /tmp/fglrx_8.982-0ubuntu1_amd64.deb" in gkonsole

I use --force-depends
because
######
sudo apt-get -f install fglrx
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.
Statusinformationen werden eingelesen.... Fertig
Probieren Sie »apt-get -f install«, um dies zu korrigieren:
Die folgenden Pakete haben unerfüllte Abhängigkeiten:
 fglrx : Hängt ab von: xorg-video-abi-11
###

xorg-video-abi-11 does not ecist in 12.10

Should we open a new bug report for the reappered problem?

I opened a quantal task now.

I can reproduce this with:
$ LANGUAGE=de ./software-center 0ad:i386
and then clicking on "install". With some debug code added I get:

2012-10-08 09:07:52,447 - softwarecenter.backend - WARNING - _on_trans_error: 'UnicodeDecodeError('ascii', 'Transaction failed: Die Paketabh\xc3\xa4ngigkeiten k\xc3\xb6nnen nicht aufgel\xc3\xb6st werden\n', 32, 33, 'ordinal not in range(128)')' '<AptTransaction object at 0x562bdc0 (aptdaemon+client+AptTransaction at 0x2d126e0)>' '0ad:i386'

This bug was fixed in the package aptdaemon - 0.45+bzr861-0ubuntu8

---------------
aptdaemon (0.45+bzr861-0ubuntu8) quantal; urgency=low

  * debian/patches/lp846044.patch:
    - fix unicode decode error in py2 clients (LP: #846044)
 -- Michael Vogt <email address hidden> Mon, 08 Oct 2012 15:57:29 +0200

Happened after reboot. Ubuntu 12.10 Beta 2.

There is another releated issue here in dbus-python:

#!/usr/bin/python
# -*- coding: utf-8 -*-
import dbus
e = dbus.exceptions.DBusException(u"bäää")
msg= e.get_dbus_message()

will crash.

This is #3 on errors.ubuntu.com right now so we should SRU it once we have a good fix.

There is another way to reproduce this in software-center:
1. mkdir /tmp/äää
2. download google-chrome-stable (or another package that lintian compalins about)
3. copy deb to /tmp/äää
4. run software-center /tmp/ää/google-chrome-stable_current_amd64.deb
5. click on "install" and verify that there is a crash in the log

Barry, can you offer any assistance here? I guess you might be familiar with the encoding handling in the dbus-python code in question :)

On Oct 16, 2012, at 12:36 AM, Steve Langasek wrote:

>Barry, can you offer any assistance here? I guess you might be familiar
>with the encoding handling in the dbus-python code in question :)

Michael already did all the hard work of figuring out a patch. I reviewed it
and suggested he send it upstream, which he did, and got further comments from
the upstream maintainer. Michael posted an updated patch to the upstream
issue.

I think it is worth fixing for Quantal, although IIRC, Michael has a
workaround for software-center. In any case, since you targeted the bug task
to quantal-updates, I'll prepare a patch for quantal-proposed (testing it
now).

dbus-python (1.1.1-1ubuntu1) quantal-proposed; urgency=low

  * debian/patches/fdo-bug-55899.patch: Apply upstream patch for upstream
    bug 55899, support for Unicode DBusExceptions in Python 2.
    (LP: #846044)

 -- Barry Warsaw <email address hidden> Tue, 16 Oct 2012 10:04:27 -0400

Saying "Apply upstream patch" is rather misleading when I haven't merged it yet (because it fails the upstream regression tests)...

On Oct 16, 2012, at 05:19 PM, Simon McVittie wrote:

>Saying "Apply upstream patch" is rather misleading when I haven't merged
>it yet (because it fails the upstream regression tests)...

I'll fix the commit message (unless you make it true by the time I re-upload
it ;). In the meantime, I thought I did see a passing test suite, but I tried
it again in a clean tree to be sure, and indeed, I see the regressions too. I
asked the release team to reject my upload, which they've done. I'll respin
that once we have a patch that you like.

Thanks Barry and Simon!

I pushed a branch based on the latest fixes from Simon here:
  lp:~mvo/ubuntu/quantal/dbus-python/lp846044

Once that is merged upstream we should upload it (and/or adept if the final commit has changes too).

Unfortunately the workaround in software-center is not enough so we should fix this here would make software-center
get off #5 from errors.ubuntu.com for quantal :)

On Oct 17, 2012, at 07:37 AM, Michael Vogt wrote:

>Thanks Barry and Simon!
>
>I pushed a branch based on the latest fixes from Simon here:
> lp:~mvo/ubuntu/quantal/dbus-python/lp846044

Mine is at lp:~barry/ubuntu/quantal/dbus-python/lp846044

I haven't looked yet, but I'm guessing they're going to be functionally
equivalent.

>Once that is merged upstream we should upload it (and/or adept if the
>final commit has changes too).

+1

I uploaded this now to quantal-proposed.

This bug was fixed in the package dbus-python - 1.1.1-1ubuntu1

---------------
dbus-python (1.1.1-1ubuntu1) raring; urgency=low

  * merge lp:~barry/ubuntu/quantal/dbus-python/lp846044
    (LP: #846044)
 -- Michael Vogt <email address hidden> Mon, 05 Nov 2012 09:12:51 +0100

Hello! Unfortunately because this was accepted in raring first, we can't accept 1.1.1-1ubuntu1 into quantal (we can copy forward, but not backward AFAIK).

So, please re-upload to quantal-proposed with a version of 1.1.1-1ubuntu0.1

On Nov 07, 2012, at 10:19 PM, Clint Byrum wrote:

>Hello! Unfortunately because this was accepted in raring first, we can't
>accept 1.1.1-1ubuntu1 into quantal (we can copy forward, but not
>backward AFAIK).
>
>So, please re-upload to quantal-proposed with a version of
>1.1.1-1ubuntu0.1

Yep, saw that. No problem.

1.1.1-1ubuntu0.1 uploaded to quantal-proposed.

Hello Steven, or anyone else affected,

Accepted dbus-python into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/dbus-python/1.1.1-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This is the #2 most frequent recorded error for all Ubuntu users in the past day, and #4 for the past month.

I can confirm that the quantal update in -proposed fixes this for me.

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

This bug was fixed in the package dbus-python - 1.1.1-1ubuntu0.1

---------------
dbus-python (1.1.1-1ubuntu0.1) quantal-proposed; urgency=low

  * debian/patches/fdo-bug-55899.patch: Support Unicode DBusExceptions
    in Python 2. (LP: #846044)
 -- Barry Warsaw <email address hidden> Tue, 16 Oct 2012 15:25:00 -0400

Looking at the error page for this problem:

https://errors.ubuntu.com/bucket/?id=%2Fusr%2Fshare%2Fsoftware-center%2Fsoftware-center%3AUnicodeEncodeError%3Amsg_reply_handler%3A%3Clambda%3E%3Aget_native_exception%3Aget_dbus_message

It appears that very few Precise users are affected by this, as such I don't believe this is worth fixing in Precise at this point in time.

Right, I can't access the page but it seems the it's more oneiric era bug.