uncaught python exception : UnicodeDecodeError: 'utf-8' codec can't decode byte; invalid continuation byte

Bug #1766337 reported by Laurent Bonnaud
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
apport (Ubuntu)
Fix Released
High
Brian Murray
Bionic
Fix Released
High
Brian Murray

Bug Description

[Impact]
Ubuntu crash reports and package installation failures may not include DpkgTerminalLog.txt if there is a UnicodeDecodeError trying to read the dpkg log file.

[Test Case]
Because the fix is simple and its complicated to have a dpkg log file with unicode in it we'll just test that truncating of a dpkg log file still works.

1) Install 2vcard (so we have a recent dpkg log entry)
2) Run update-manager
3) Kill it with signal 11 e.g. kill -11 $PID of update-manager
4) Run ubuntu-bug /var/crash/_usr_bin_update-manager.1000.crash
5) Observe DpkgTerminalLog.txt only contains a log entry regarding the installation of 2vcard and matches the content in /var/log/apt/term.log.

[Regression Potential]
Its possible that the code to replace the DpkgTerminalLog contents of the report is wrong but the test case above ensures that it isn't.

[Original Description]
Hi,

here is a python exception I got while reporting a bug:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport/report.py", line 198, in _run_hook
    symb['add_info'](report, ui)
  File "/usr/share/apport/general-hooks/ubuntu.py", line 89, in add_info
    check_attachment_for_errors(report, attachment)
  File "/usr/share/apport/general-hooks/ubuntu.py", line 208, in check_attachment_for_errors
    trim_dpkg_log(report)
  File "/usr/share/apport/general-hooks/ubuntu.py", line 508, in trim_dpkg_log
    report['DpkgTerminalLog'] = '\n'.join([str(line.decode('utf-8')) for line in lines])
  File "/usr/share/apport/general-hooks/ubuntu.py", line 508, in <listcomp>
    report['DpkgTerminalLog'] = '\n'.join([str(line.decode('utf-8')) for line in lines])
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 3070: invalid continuation byte

It can also be seen here:

  https://launchpadlibrarian.net/367002573/HookError_ubuntu.txt

ProblemType: BugDistroRelease: Ubuntu 18.04
Package: apport 2.20.9-0ubuntu6
Uname: Linux 4.16.3-041603-generic x86_64
ApportLog:

ApportVersion: 2.20.9-0ubuntu6
Architecture: amd64
CurrentDesktop: MATE
Date: Mon Apr 23 20:13:24 2018
PackageArchitecture: all
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bashSourcePackage: apport
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Laurent Bonnaud (laurent-bonnaud) wrote :
tags: added: id-5ae35ea55ac10e9203cade8d
Changed in apport (Ubuntu):
status: New → In Progress
importance: Undecided → High
assignee: nobody → Brian Murray (brian-murray)
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apport - 2.20.10-0ubuntu1

---------------
apport (2.20.10-0ubuntu1) cosmic; urgency=medium

  * New upstream release:
    - backends/packaging-apt-dpkg.py: when using a permanent sandbox better
      check for the package being already available thereby reducing quantity
      of downloads.
    - apport/report.py: Use timeout parameter in subprocess to limit how long
      we'll wait for gdb to return information as gdb can hang with some core
      files.
    - data/apport: Fix PEP8 issues
    - apport/ui.py: Include ProblemType in reports which are updated as
      package hooks may expect the report to have a ProblemType. (LP: #1766794)
    - test/test_ui.py: modify run_crash_kernel test to account for the fact
      that linux-image-$kvers-$flavor is now built from the linux-signed
      source package on amd64 and ppc64el. (LP: #1766740)
  * data/general-hooks/ubuntu.py: Fix UnicodeDecodeError when processing
    DpkgTerminalLog. (LP: #1766337)
  * data/general-hooks/ubuntu.py: Don't display a messy error if python or
    python3 is not installed. (LP: #1769262)
  * debian/apport.links: source_linux-signed.py -> source_linux.py package
    hook (LP: #1766740)
  * data/package-hooks/source_linux.py: Redirect bugs filed about the
    linux-signed source package to linux. (LP: #1766740)

 -- Brian Murray <email address hidden> Thu, 10 May 2018 09:31:34 -0700

Changed in apport (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Laurent Bonnaud (laurent-bonnaud) wrote :

Thank you for the fix!

Changed in apport (Ubuntu Bionic):
status: New → In Progress
importance: Undecided → High
assignee: nobody → Brian Murray (brian-murray)
Changed in apport (Ubuntu Bionic):
status: In Progress → Triaged
assignee: Brian Murray (brian-murray) → nobody
Changed in apport (Ubuntu Bionic):
assignee: nobody → Brian Murray (brian-murray)
status: Triaged → In Progress
description: updated
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Laurent, or anyone else affected,

Accepted apport into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apport/2.20.9-0ubuntu7.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in apport (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-bionic
Revision history for this message
Brian Murray (brian-murray) wrote :

DpkgTerminalLog.txt still appears in an update-manager crash report using the version of apport from bionic-proposed.

bdmurray@clean-bionic-amd64:~$ grep -A4 DpkgTerminal /var/crash/_usr_bin_update-manager.1000.crash
DpkgTerminalLog.txt:
 Log started: 2019-03-05 09:00:54
(Reading database ... 166173 files and directories currently installed.)
 Removing apport-noui (2.20.9-0ubuntu7.6) ...
 Failed to stop apport-noui.path: Unit apport-noui.path not loaded.

bdmurray@clean-bionic-amd64:~$ apt-cache policy apport
apport:
  Installed: 2.20.9-0ubuntu7.6
  Candidate: 2.20.9-0ubuntu7.6

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apport - 2.20.9-0ubuntu7.6

---------------
apport (2.20.9-0ubuntu7.6) bionic; urgency=medium

  * data/general-hooks/ubuntu.py: Fix UnicodeDecodeError when processing
    DpkgTerminalLog. (LP: #1766337)
  * debian/apport-autoreport.service: Since this calls whoopsie-upload-all
    which fails if whoopsie isn't running, it should have a relationship with
    whoopsie.service. (LP: #1787729)

 -- Brian Murray <email address hidden> Wed, 06 Feb 2019 10:46:53 -0800

Changed in apport (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for apport has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.