At least one invalid signature was encountered.

Bug #1642386 reported by Seth Arnold
30
This bug affects 6 people
Affects Status Importance Assigned to Milestone
apt (Ubuntu)
Fix Released
High
Julian Andres Klode
Xenial
Fix Released
Undecided
Unassigned

Bug Description

[Summary]

A regression in apt in Xenial 1.2.15 causes "apt-get update" to fail with "At least one invalid signature was encountered." if there are files in /etc/apt/trusted.gpg.d/ that are not readable by the _apt user.

This has the consequence of getting apt "stuck"; it will not be able to download its own update that fixes the issue. This means that all affected users must apply the workaround; otherwise they will be stuck forever.

[Workaround]

Make sure all files in /etc/apt/trusted.gpg.d/ are world-readable. For example: "sudo chmod 644 /etc/apt/trusted.gpg.d/*". Then try "apt-get update" again.

Alternatively, you can manually install the fixed version of apt using dpkg.

[Impact]
Breaks update on systems with unreadable GPG keys

[Test case]
Run apt update with an unreadable GPG key file in trusted.gpg.d. This should work and (stretch goal) print a warning about the key being unreadable.

[Regression potential]
Low risk. We check that very situation in the automated test suite now like we did a lot of other situations before. The fix has been available in apt since 1.3_rc3 on Aug 30, and there have been no regressions reported since then.

[Original bug report]
Hello, a recent apt update appears to have broken apt entirely.

A coworker reported seeing troubles: http://paste.ubuntu.com/23487135/

To test, I upgraded my laptop then immediately re-ran apt-get update && apt-get -u dist-upgrade:

sarnold@hunt:~/Downloads$ sudo apt-get update && sudo apt-get -u dist-upgrade
Hit:1 http://mirrors.kernel.org/ubuntu xenial InRelease
Hit:2 http://mirrors.kernel.org/ubuntu xenial-updates InRelease
Hit:3 http://mirrors.kernel.org/ubuntu xenial-security InRelease
Ign:4 http://mirrors.kernel.org/ubuntu precise InRelease
Get:5 http://mirrors.kernel.org/ubuntu precise-updates InRelease [55.7 kB]
Get:6 http://mirrors.kernel.org/ubuntu precise-security InRelease [55.7 kB]
Get:7 http://mirrors.kernel.org/ubuntu precise-proposed InRelease [55.7 kB]
Ign:8 http://mirrors.kernel.org/ubuntu trusty InRelease
Get:9 http://mirrors.kernel.org/ubuntu trusty-updates InRelease [65.9 kB]
Hit:10 http://mirrors.kernel.org/ubuntu trusty-security InRelease
Get:11 http://mirrors.kernel.org/ubuntu trusty-proposed InRelease [65.9 kB]
Get:12 http://mirrors.kernel.org/ubuntu xenial-proposed InRelease [247 kB]
Err:1 http://mirrors.kernel.org/ubuntu xenial InRelease
  At least one invalid signature was encountered.
Hit:13 http://security.debian.org jessie/updates InRelease
Err:2 http://mirrors.kernel.org/ubuntu xenial-updates InRelease
  At least one invalid signature was encountered.
Get:14 http://mirrors.kernel.org/ubuntu yakkety InRelease [247 kB]
Err:3 http://mirrors.kernel.org/ubuntu xenial-security InRelease
  At least one invalid signature was encountered.
Hit:15 http://mirrors.kernel.org/ubuntu yakkety-updates InRelease
Get:16 http://mirrors.kernel.org/ubuntu yakkety-security InRelease [93.3 kB]
Hit:17 http://security.debian.org wheezy/updates InRelease
Get:18 http://mirrors.kernel.org/ubuntu yakkety-proposed InRelease [95.7 kB]
Hit:19 http://mirrors.kernel.org/ubuntu zesty InRelease
Hit:20 http://mirrors.kernel.org/ubuntu zesty-updates InRelease
Hit:21 http://mirrors.kernel.org/ubuntu zesty-security InRelease
Err:5 http://mirrors.kernel.org/ubuntu precise-updates InRelease
  At least one invalid signature was encountered.
Hit:22 http://mirrors.kernel.org/ubuntu zesty-proposed InRelease
Hit:23 http://mirrors.kernel.org/ubuntu precise Release
Hit:24 http://mirrors.kernel.org/ubuntu trusty Release
Ign:25 http://archive.canonical.com/ubuntu precise InRelease
Hit:26 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:27 http://ppa.launchpad.net/ci-train-ppa-service/stable-phone-overlay/ubuntu vivid InRelease
Hit:28 http://ftp.debian.org/debian unstable InRelease
Err:6 http://mirrors.kernel.org/ubuntu precise-security InRelease
  At least one invalid signature was encountered.
Err:7 http://mirrors.kernel.org/ubuntu precise-proposed InRelease
  At least one invalid signature was encountered.
Err:9 http://mirrors.kernel.org/ubuntu trusty-updates InRelease
  At least one invalid signature was encountered.
Ign:29 http://archive.canonical.com/ubuntu trusty InRelease
Hit:30 http://ppa.launchpad.net/snappy-dev/image/ubuntu vivid InRelease
Hit:31 http://ftp.debian.org/debian testing InRelease
Err:10 http://mirrors.kernel.org/ubuntu trusty-security InRelease
  At least one invalid signature was encountered.
Err:11 http://mirrors.kernel.org/ubuntu trusty-proposed InRelease
  At least one invalid signature was encountered.
Err:13 http://security.debian.org jessie/updates InRelease
  At least one invalid signature was encountered.
Hit:32 http://archive.canonical.com/ubuntu xenial InRelease
Hit:33 http://ppa.launchpad.net/ci-train-ppa-service/stable-phone-overlay/ubuntu xenial InRelease
Ign:34 http://ftp.debian.org/debian jessie InRelease
Err:12 http://mirrors.kernel.org/ubuntu xenial-proposed InRelease
  At least one invalid signature was encountered.
Err:14 http://mirrors.kernel.org/ubuntu yakkety InRelease
  At least one invalid signature was encountered.
Hit:35 http://archive.canonical.com/ubuntu yakkety InRelease
Err:15 http://mirrors.kernel.org/ubuntu yakkety-updates InRelease
  At least one invalid signature was encountered.
Hit:36 http://ftp.debian.org/debian jessie-updates InRelease
Err:16 http://mirrors.kernel.org/ubuntu yakkety-security InRelease
  At least one invalid signature was encountered.
Err:17 http://security.debian.org wheezy/updates InRelease
  At least one invalid signature was encountered.
Hit:37 http://archive.canonical.com/ubuntu zesty InRelease
Err:18 http://mirrors.kernel.org/ubuntu yakkety-proposed InRelease
  At least one invalid signature was encountered.
Ign:38 http://ftp.debian.org/debian wheezy InRelease
Err:19 http://mirrors.kernel.org/ubuntu zesty InRelease
  At least one invalid signature was encountered.
Err:20 http://mirrors.kernel.org/ubuntu zesty-updates InRelease
  At least one invalid signature was encountered.
Hit:39 http://archive.canonical.com/ubuntu precise Release
Err:21 http://mirrors.kernel.org/ubuntu zesty-security InRelease
  At least one invalid signature was encountered.
Hit:40 http://ftp.debian.org/debian wheezy-updates InRelease
Err:22 http://mirrors.kernel.org/ubuntu zesty-proposed InRelease
  At least one invalid signature was encountered.
Err:41 http://mirrors.kernel.org/ubuntu precise Release.gpg
  At least one invalid signature was encountered.
Hit:42 http://archive.canonical.com/ubuntu trusty Release
Err:43 http://mirrors.kernel.org/ubuntu trusty Release.gpg
  At least one invalid signature was encountered.
Hit:44 http://ftp.debian.org/debian jessie Release
Err:26 http://security.ubuntu.com/ubuntu xenial-security InRelease
  At least one invalid signature was encountered.
Err:27 http://ppa.launchpad.net/ci-train-ppa-service/stable-phone-overlay/ubuntu vivid InRelease
  At least one invalid signature was encountered.
Err:28 http://ftp.debian.org/debian unstable InRelease
  At least one invalid signature was encountered.
Hit:45 http://ftp.debian.org/debian wheezy Release
Err:30 http://ppa.launchpad.net/snappy-dev/image/ubuntu vivid InRelease
  At least one invalid signature was encountered.
Err:31 http://ftp.debian.org/debian testing InRelease
  At least one invalid signature was encountered.
Err:32 http://archive.canonical.com/ubuntu xenial InRelease
  At least one invalid signature was encountered.
Err:33 http://ppa.launchpad.net/ci-train-ppa-service/stable-phone-overlay/ubuntu xenial InRelease
  At least one invalid signature was encountered.
Err:35 http://archive.canonical.com/ubuntu yakkety InRelease
  At least one invalid signature was encountered.
Err:36 http://ftp.debian.org/debian jessie-updates InRelease
  At least one invalid signature was encountered.
Err:37 http://archive.canonical.com/ubuntu zesty InRelease
  At least one invalid signature was encountered.
Err:46 http://archive.canonical.com/ubuntu precise Release.gpg
  At least one invalid signature was encountered.
Err:40 http://ftp.debian.org/debian wheezy-updates InRelease
  At least one invalid signature was encountered.
Err:47 http://archive.canonical.com/ubuntu trusty Release.gpg
  At least one invalid signature was encountered.
Err:48 http://ftp.debian.org/debian jessie Release.gpg
  At least one invalid signature was encountered.
Err:49 http://ftp.debian.org/debian wheezy Release.gpg
  At least one invalid signature was encountered.
Fetched 981 kB in 1s (496 kB/s)
Reading package lists... Done
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.kernel.org/ubuntu xenial InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.kernel.org/ubuntu xenial-updates InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.kernel.org/ubuntu xenial-security InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.kernel.org/ubuntu precise-updates InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.kernel.org/ubuntu precise-security InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.kernel.org/ubuntu precise-proposed InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.kernel.org/ubuntu trusty-updates InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.kernel.org/ubuntu trusty-security InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.kernel.org/ubuntu trusty-proposed InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://security.debian.org jessie/updates InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.kernel.org/ubuntu xenial-proposed InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.kernel.org/ubuntu yakkety InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.kernel.org/ubuntu yakkety-updates InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.kernel.org/ubuntu yakkety-security InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://security.debian.org wheezy/updates InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.kernel.org/ubuntu yakkety-proposed InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.kernel.org/ubuntu zesty InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.kernel.org/ubuntu zesty-updates InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.kernel.org/ubuntu zesty-security InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.kernel.org/ubuntu zesty-proposed InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.kernel.org/ubuntu precise Release: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirrors.kernel.org/ubuntu trusty Release: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://security.ubuntu.com/ubuntu xenial-security InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://ppa.launchpad.net/ci-train-ppa-service/stable-phone-overlay/ubuntu vivid InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://ftp.debian.org/debian unstable InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://ppa.launchpad.net/snappy-dev/image/ubuntu vivid InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://ftp.debian.org/debian testing InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://archive.canonical.com/ubuntu xenial InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://ppa.launchpad.net/ci-train-ppa-service/stable-phone-overlay/ubuntu xenial InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://archive.canonical.com/ubuntu yakkety InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://ftp.debian.org/debian jessie-updates InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://archive.canonical.com/ubuntu zesty InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://archive.canonical.com/ubuntu precise Release: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://ftp.debian.org/debian wheezy-updates InRelease: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://archive.canonical.com/ubuntu trusty Release: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://ftp.debian.org/debian jessie Release: At least one invalid signature was encountered.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://ftp.debian.org/debian wheezy Release: At least one invalid signature was encountered.
W: Failed to fetch http://mirrors.kernel.org/ubuntu/dists/xenial/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://mirrors.kernel.org/ubuntu/dists/xenial-updates/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://mirrors.kernel.org/ubuntu/dists/xenial-security/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/xenial-security/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://ppa.launchpad.net/ci-train-ppa-service/stable-phone-overlay/ubuntu/dists/vivid/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://ppa.launchpad.net/snappy-dev/image/ubuntu/dists/vivid/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://ppa.launchpad.net/ci-train-ppa-service/stable-phone-overlay/ubuntu/dists/xenial/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://mirrors.kernel.org/ubuntu/dists/precise-updates/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://mirrors.kernel.org/ubuntu/dists/precise-security/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://mirrors.kernel.org/ubuntu/dists/precise-proposed/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://mirrors.kernel.org/ubuntu/dists/trusty-updates/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://mirrors.kernel.org/ubuntu/dists/trusty-security/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://mirrors.kernel.org/ubuntu/dists/trusty-proposed/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://mirrors.kernel.org/ubuntu/dists/xenial-proposed/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://archive.canonical.com/ubuntu/dists/xenial/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://mirrors.kernel.org/ubuntu/dists/yakkety/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://mirrors.kernel.org/ubuntu/dists/yakkety-updates/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://mirrors.kernel.org/ubuntu/dists/yakkety-security/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://mirrors.kernel.org/ubuntu/dists/yakkety-proposed/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://archive.canonical.com/ubuntu/dists/yakkety/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://mirrors.kernel.org/ubuntu/dists/zesty/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://mirrors.kernel.org/ubuntu/dists/zesty-updates/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://mirrors.kernel.org/ubuntu/dists/zesty-security/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://mirrors.kernel.org/ubuntu/dists/zesty-proposed/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://archive.canonical.com/ubuntu/dists/zesty/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://ftp.debian.org/debian/dists/unstable/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://ftp.debian.org/debian/dists/testing/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://ftp.debian.org/debian/dists/jessie-updates/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://security.debian.org/dists/jessie/updates/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://ftp.debian.org/debian/dists/wheezy-updates/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://security.debian.org/dists/wheezy/updates/InRelease At least one invalid signature was encountered.
W: Failed to fetch http://mirrors.kernel.org/ubuntu/dists/precise/Release.gpg At least one invalid signature was encountered.
W: Failed to fetch http://mirrors.kernel.org/ubuntu/dists/trusty/Release.gpg At least one invalid signature was encountered.
W: Failed to fetch http://archive.canonical.com/ubuntu/dists/precise/Release.gpg At least one invalid signature was encountered.
W: Failed to fetch http://archive.canonical.com/ubuntu/dists/trusty/Release.gpg At least one invalid signature was encountered.
W: Failed to fetch http://ftp.debian.org/debian/dists/jessie/Release.gpg At least one invalid signature was encountered.
W: Failed to fetch http://ftp.debian.org/debian/dists/wheezy/Release.gpg At least one invalid signature was encountered.
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  linux-headers-4.4.0-21 linux-headers-4.4.0-21-generic linux-headers-4.4.0-22 linux-headers-4.4.0-22-generic
  linux-headers-4.4.0-24 linux-headers-4.4.0-24-generic linux-headers-4.4.0-28 linux-headers-4.4.0-28-generic
  linux-headers-4.4.0-31 linux-headers-4.4.0-31-generic linux-headers-4.4.0-34 linux-headers-4.4.0-34-generic
  linux-headers-4.4.0-36 linux-headers-4.4.0-36-generic linux-headers-4.4.0-38 linux-headers-4.4.0-38-generic
  linux-image-4.4.0-21-generic linux-image-4.4.0-22-generic linux-image-4.4.0-24-generic
  linux-image-4.4.0-28-generic linux-image-4.4.0-31-generic linux-image-4.4.0-34-generic
  linux-image-4.4.0-36-generic linux-image-4.4.0-38-generic
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

I'll attach the full terminal log soon.

Thanks

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: apt 1.2.15
ProcVersionSignature: Ubuntu 4.4.0-45.66-generic 4.4.21
Uname: Linux 4.4.0-45-generic x86_64
NonfreeKernelModules: zfs zunicode zcommon znvpair zavl
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
CurrentDesktop: Unity
Date: Wed Nov 16 12:29:54 2016
InstallationDate: Installed on 2012-10-18 (1490 days ago)
InstallationMedia: Ubuntu 12.04.1 LTS "Precise Pangolin" - Release amd64 (20120823.1)
SourcePackage: apt
UpgradeStatus: Upgraded to xenial on 2016-04-30 (200 days ago)

Revision history for this message
Seth Arnold (seth-arnold) wrote :
Revision history for this message
Seth Arnold (seth-arnold) wrote :

The full run, showing apt working a few seconds before it fails, and no errors in dmesg.

Revision history for this message
Seth Arnold (seth-arnold) wrote :

I was too hasty -- apt appears to still function (e.g. apt-get install expat, apt-get purge expat, installed an expat from the -updates pocket and removed it again). It's just insanely ugly warnings on the apt-get update step, and maybe(?) new lists can't be downloaded.

Anyway it's more nuanced than "broken entirely". Sorry.

Revision history for this message
Julian Andres Klode (juliank) wrote :

It works perfectly fine for me. Can you re-run this with:

 -o Debug::Acquire::gpgv=1

Changed in apt (Ubuntu):
status: New → Incomplete
Changed in apt (Ubuntu):
importance: Undecided → High
Revision history for this message
Julian Andres Klode (juliank) wrote :

Your coworker seems to have caught a crash in appstream, BTW, not in apt. Not sure why the files fail their hashes or have no sections in them, though.

Changed in apt (Ubuntu):
assignee: nobody → Julian Andres Klode (juliank)
Revision history for this message
Julian Andres Klode (juliank) wrote :

Oh, you may also want to try moving lists/ out of the way and running apt update again.

And perhaps send me a tarball of /var/lib/apt and /etc/apt - then I might have luck reproducing it.

If you want to, you could try bisecting this in the apt git repo, starting with

git bisect start
git bisect good 1.2.12
git bisect bad 1.2.15
git bisect run sh ./script.sh

where script.sh is:

#!/bin/sh
make fast || exit 125
sudo LD_LIBRARY_PATH=$PWD/build/bin/ ./build/bin/apt-get update -o Dir::Bin::Methods="$PWD/build/bin/methods/" 2>&1 | tee update.log

if grep "was encountered" update.log; then
  exit 1
fi

exit 0

Revision history for this message
Seth Arnold (seth-arnold) wrote :

apt-get update -o Debug::Acquire::gpgv=1

Revision history for this message
Seth Arnold (seth-arnold) wrote :

Tarball of /etc/apt and /var/lib/apt

Revision history for this message
Seth Arnold (seth-arnold) wrote :

393c61b8e29bd1923a5fe8abf4690c24e7f498aa8a4f5954a6a87da7d05a0bef apt-etc-and-var.tar.xz

Revision history for this message
Seth Arnold (seth-arnold) wrote :
Download full text (39.6 KiB)

sarnold@hunt:/var/lib/apt$ sudo mv lists lists.old
sarnold@hunt:/var/lib/apt$ sudo apt-get update
Get:1 http://mirrors.kernel.org/ubuntu xenial InRelease [247 kB]
Get:2 http://mirrors.kernel.org/ubuntu xenial-updates InRelease [95.7 kB]
Get:3 http://mirrors.kernel.org/ubuntu xenial-security InRelease [94.5 kB]
Ign:4 http://mirrors.kernel.org/ubuntu precise InRelease
Get:5 http://mirrors.kernel.org/ubuntu precise-updates InRelease [55.7 kB]
Get:6 http://mirrors.kernel.org/ubuntu precise-security InRelease [55.7 kB]
Get:7 http://mirrors.kernel.org/ubuntu precise-proposed InRelease [55.7 kB]
Ign:8 http://mirrors.kernel.org/ubuntu trusty InRelease
Get:9 http://mirrors.kernel.org/ubuntu trusty-updates InRelease [65.9 kB]
Get:10 http://mirrors.kernel.org/ubuntu trusty-security InRelease [65.9 kB]
Get:11 http://mirrors.kernel.org/ubuntu trusty-proposed InRelease [65.9 kB]
Get:12 http://mirrors.kernel.org/ubuntu xenial-proposed InRelease [247 kB]
Get:13 http://mirrors.kernel.org/ubuntu yakkety InRelease [247 kB]
Get:14 http://mirrors.kernel.org/ubuntu yakkety-updates InRelease [94.5 kB]
Get:15 http://mirrors.kernel.org/ubuntu yakkety-security InRelease [93.3 kB]
Get:16 http://mirrors.kernel.org/ubuntu yakkety-proposed InRelease [95.7 kB]
Get:17 http://mirrors.kernel.org/ubuntu zesty InRelease [247 kB]
Ign:1 http://mirrors.kernel.org/ubuntu xenial InRelease
Get:18 http://security.debian.org jessie/updates InRelease [63.1 kB]
Get:19 http://mirrors.kernel.org/ubuntu zesty-updates InRelease [92.1 kB]
Get:20 http://mirrors.kernel.org/ubuntu zesty-security InRelease [92.2 kB]
Get:21 http://mirrors.kernel.org/ubuntu zesty-proposed InRelease [95.6 kB]
Get:22 http://mirrors.kernel.org/ubuntu precise Release [49.6 kB]
Get:23 http://mirrors.kernel.org/ubuntu trusty Release [58.5 kB]
Get:24 http://mirrors.kernel.org/ubuntu xenial/main Sources [868 kB]
Ign:2 http://mirrors.kernel.org/ubuntu xenial-updates InRelease
Get:25 http://mirrors.kernel.org/ubuntu xenial/restricted Sources [4,808 B]
Get:26 http://mirrors.kernel.org/ubuntu xenial/...

Revision history for this message
Seth Arnold (seth-arnold) wrote :

And the results of git bisect, thanks for the excellent instruction and script!

cde5b485c9cdf0bfd5b6ea8e4973abe378270e60 is the first bad commit
commit cde5b485c9cdf0bfd5b6ea8e4973abe378270e60
Author: David Kalnischkies <email address hidden>
Date: Fri May 20 09:37:24 2016 +0200

    fail instead of segfault on unreadable config files

    The report mentions "apt list --upgradable", but there are others which
    have inconsistent behavior ranging from segfaulting to doing something
    with the partial (and hence incomplete) data. We had a recent report
    about sources.list (#818628), this one mentions prefences, the obvious
    next step is conf files… so the testcase is adapted to check for all
    three in file and directory versions and run a bunch of commands each
    time which should all have more or less the same behavior in such a case
    (aka error out).

    Closes: 824503
    (cherry picked from commit fdf9eef4d96a18d0167708499c993e1174251e88)

:040000 040000 04f4856e0a9313f9f51a5a6dc56c9af005ac54f4 98d34296e4f9212a124515095f6ed9afd5739111 M apt-pkg
:040000 040000 d48745d59ec9dd40087de492197c7a1060d1451d a5038f3b3730d8ba9e070f68a60af11fbe0e7ac3 M apt-private
:040000 040000 cfb7d9b7f8130e98173cf09a56eea0c232cc75fa c2f0ed862e98021705643c42a1b549306b648c15 M cmdline
:040000 040000 24d65ae746427999b728b85acb6969f509885cb1 82f28b27a48c3a6cf5b9caaf6434512a4a5dd79f M test
bisect run success

Revision history for this message
Seth Arnold (seth-arnold) wrote :
Download full text (26.1 KiB)

I'm skeptical of the git bisect results:

$ git checkout cde5b485c9cdf0bfd5b6ea8e4973abe378270e60^
Previous HEAD position was cde5b48... fail instead of segfault on unreadable config files
HEAD is now at 235347e... Release 1.2.12
$ make fast
Compiling cachefile.cc to ../build/obj/apt-pkg/cachefile.opic
Compiling policy.cc to ../build/obj/apt-pkg/policy.opic
Building shared library ../build/bin/libapt-pkg.so.5.0.0
Compiling contrib/arfile.cc to ../build/obj/apt-inst/arfile.opic
Compiling contrib/extracttar.cc to ../build/obj/apt-inst/extracttar.opic
Compiling deb/debfile.cc to ../build/obj/apt-inst/debfile.opic
Compiling dirstream.cc to ../build/obj/apt-inst/dirstream.opic
Compiling extract.cc to ../build/obj/apt-inst/extract.opic
Compiling filelist.cc to ../build/obj/apt-inst/filelist.opic
Building shared library ../build/bin/libapt-inst.so.2.0.0
Compiling private-cacheset.cc to ../build/obj/apt-private/private-cacheset.opic
Compiling private-list.cc to ../build/obj/apt-private/private-list.opic
Compiling private-depends.cc to ../build/obj/apt-private/private-depends.opic
Compiling private-show.cc to ../build/obj/apt-private/private-show.opic
Building shared library ../build/bin/libapt-private.so.0.0.0
Compiling apt-mark.cc to ../build/obj/cmdline/apt-mark.o
Building program ../build/bin/apt-mark
Must have libdb to build apt-ftparchive
$ sudo rm -rf /var/lib/apt/lists
sarnold@hunt:~/trees/apt$ sudo LD_LIBRARY_PATH=$PWD/build/bin/ ./build/bin/apt-get update -o Dir::Bin::Methods="$PWD/build/bin/methods/" 2>&1 | tee update.log
Get:1 http://mirrors.kernel.org/ubuntu xenial InRelease [247 kB]
Get:2 http://mirrors.kernel.org/ubuntu xenial-updates InRelease [95.7 kB]
Get:3 http://security.debian.org jessie/updates InRelease [63.1 kB]
Get:4 http://mirrors.kernel.org/ubuntu xenial-security InRelease [94.5 kB]
Ign:5 http://mirrors.kernel.org/ubuntu precise InRelease
Get:6 http://mirrors.kernel.org/ubuntu precise-updates InRelease [55.7 kB]
Get:7 http://mirrors.kernel.org/ubuntu precise-security InRelease [55.7 kB]
Get:8 http://mirrors.kernel.org/ubuntu precise-proposed InRelease [55.7 kB]
Ign:9 http://mirrors.kernel.org/ubuntu trusty InRelease
Get:10 http://mirrors.kernel.org/ubuntu trusty-updates InRelease [65.9 kB]
Get:11 http://mirrors.kernel.org/ubuntu trusty-security InRelease [65.9 kB]
Get:12 http://mirrors.kernel.org/ubuntu trusty-proposed InRelease [65.9 kB]
Get:13 http://mirrors.kernel.org/ubuntu xenial-proposed InRelease [247 kB]
Get:14 http://mirrors.kernel.org/ubuntu yakkety InRelease [247 kB]
Get:15 http://security.debian.org wheezy/updates InRelease [40.6 kB]
Get:16 http://mirrors.kernel.org/ubuntu yakkety-updates InRelease [94.5 kB]
Get:17 http://mirrors.kernel.org/ubuntu yakkety-security InRelease [93.3 kB]
Get:18 http://mirrors.kernel.org/ubuntu yakkety-proposed InRelease [95.7 kB]
Get:19 http://mirrors.kernel.org/ubuntu zesty InRelease [247 kB]
Ign:1 http://mirrors.kernel.org/ubuntu xenial InRelease
Get:20 http://mirrors.kernel.org/ubuntu zesty-updates InRelease [92.1 kB]
Get:21 http://mirrors.kernel.org/ubuntu zesty-security InRelease [92.2 kB]
Get:22 http://mirrors.kernel.org/ubuntu zesty-proposed InRelease [95.6 kB]
Get:23 http://m...

Changed in apt (Ubuntu):
status: Incomplete → New
Revision history for this message
Julian Andres Klode (juliank) wrote :

First thanks for the files. Unfortunately, I can't reproduce it with the files either. It must be a fairly system-specific bug.

> HEAD is now at 235347e... Release 1.2.12
> $ make fast
> [... with errors]

Whoa, if it happens with the 1.2.12 checkout too, then something else seems wrong. I picked that as the "good" commit, as it apparently worked in your first log file. But with 1.2.12 failing in the checkout as well, that seems really weird. You could try older 1.2 versions if you want, specifying bad as 1.2.12, and good as 1.2.10, for example (1.2.10 is basically the version xenial shipped with).

So this does not really look like a regression because it apparently happens with the previously installed version as well now (at least when built via git).

Revision history for this message
Julian Andres Klode (juliank) wrote :

What we see from your debug output is that gpgv is not returning any sensible information:

Summary:
  Good:
  Bad:
  Worthless:
  SoonWorthless:
  NoPubKey:

You could try running apt-key verify manually on a few InRelease files (like apt-key verify /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_xenial_InRelease) and see if that gives us any clue - or at least look at the files (especially in partial/), maybe they are garbage.

Revision history for this message
Seth Arnold (seth-arnold) wrote :
Download full text (14.7 KiB)

Based on the funny git results, I re-installed the packages:

3243aa899fcf2f09b910b7429eeae6205a71c379a45c0e8e31723836bb094163 apt_1.2.12~ubuntu16.04.1_amd64.deb
5b9a82b1dc1f82fc3655038336d099410d643d5188629aba475050d7f9bd99c3 apt-transport-https_1.2.12~ubuntu16.04.1_amd64.deb
25af186c488f2b7f31dcef15776bfe4dd1a7a3c98a1d378937f07365eb9aa95a libapt-inst2.0_1.2.12~ubuntu16.04.1_amd64.deb
b84273b8bfddea9aa5be26b2dd2e7ed449503a93c92ac5522fdfa74ae6f61c22 libapt-pkg5.0_1.2.12~ubuntu16.04.1_amd64.deb

With these packages installed, apt-get update works as I expect:

sarnold@hunt:/mnt/ubuntu/.zfs/snapshot/rsync.30/pool/main/a/apt$ sudo apt-get update
Hit:1 http://mirrors.kernel.org/ubuntu xenial InRelease
Hit:2 http://mirrors.kernel.org/ubuntu xenial-updates InRelease
Hit:3 http://mirrors.kernel.org/ubuntu xenial-security InRelease
Ign:4 http://mirrors.kernel.org/ubuntu precise InRelease
Hit:5 http://security.debian.org jessie/updates InRelease
Hit:6 http://mirrors.kernel.org/ubuntu precise-updates InRelease
Hit:7 http://mirrors.kernel.org/ubuntu precise-security InRelease
Hit:8 http://mirrors.kernel.org/ubuntu precise-proposed InRelease
Hit:9 http://security.debian.org wheezy/updates InRelease
Ign:10 http://mirrors.kernel.org/ubuntu trusty InRelease
Hit:11 http://mirrors.kernel.org/ubuntu trusty-updates InRelease
Hit:12 http://mirrors.kernel.org/ubuntu trusty-security InRelease
Hit:13 http://mirrors.kernel.org/ubuntu trusty-proposed InRelease
Hit:14 http://mirrors.kernel.org/ubuntu xenial-proposed InRelease
Hit:15 http://mirrors.kernel.org/ubuntu yakkety InRelease
Ign:16 http://archive.canonical.com/ubuntu precise InRelease
Hit:17 http://ppa.launchpad.net/ci-train-ppa-service/stable-phone-overlay/ubuntu vivid InRelease
Hit:18 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:19 http://mirrors.kernel.org/ubuntu yakkety-updates InRelease
Hit:20 http://ftp.debian.org/debian unstable InRelease
Hit:21 http://mirrors.kernel.org/ubuntu yakkety-security InRelease
Hit:22 http://mirrors.kernel.org/ubuntu yakkety-proposed InRelease
Hit:23 http://mirrors.kernel.org/ubuntu zesty InRelease ...

Revision history for this message
Seth Arnold (seth-arnold) wrote :
Download full text (32.5 KiB)

I reinstalled the latest xenial packages:

ii apt 1.2.15 amd64 commandline package manager
ii apt-transport-https 1.2.15 amd64 https download transport for APT
ii libapt-inst2.0:amd64 1.2.15 amd64 deb package format runtime library
ii libapt-pkg-perl 0.1.29build7 amd64 Perl interface to libapt-pkg
ii libapt-pkg5.0:amd64 1.2.15 amd64 package management runtime library

and now apt-get update is unhappy again:

sarnold@hunt:/mnt/ubuntu/.zfs/snapshot/rsync.30/pool/main/a/apt$ sudo apt-get update
Hit:1 http://mirrors.kernel.org/ubuntu xenial InRelease
Hit:2 http://security.debian.org jessie/updates InRelease
Hit:3 http://security.debian.org wheezy/updates InRelease
Hit:4 http://mirrors.kernel.org/ubuntu xenial-updates InRelease
Hit:5 http://mirrors.kernel.org/ubuntu xenial-security InRelease
Ign:6 http://mirrors.kernel.org/ubuntu precise InRelease
Hit:7 http://mirrors.kernel.org/ubuntu precise-updates InRelease
Err:1 http://mirrors.kernel.org/ubuntu xenial InRelease
  At least one invalid signature was encountered.
Hit:8 http://mirrors.kernel.org/ubuntu precise-security InRelease
Hit:9 http://mirrors.kernel.org/ubuntu precise-proposed InRelease
Ign:10 http://mirrors.kernel.org/ubuntu trusty InRelease
Err:2 http://security.debian.org jessie/updates InRelease
  At least one invalid signature was encountered.
Hit:11 http://mirrors.kernel.org/ubuntu trusty-updates InRelease
Hit:12 http://mirrors.kernel.org/ubuntu trusty-security InRelease
Err:3 http://security.debian.org wheezy/updates InRelease
  At least one invalid signature was encountered.
Hit:13 http://mirrors.kernel.org/ubuntu trusty-proposed InRelease
Hit:14 http://mirrors.kernel.org/ubuntu xenial-proposed InRelease
Hit:15 http://ppa.launchpad.net/ci-train-ppa-service/stable-phone-overlay/ubuntu vivid InRelease
Ign:16 http://archive.canonical.com/ubuntu precise InRelease
Get:17 http://security.ubuntu.com/ubuntu xenial-security InRelease [94.5 kB]
Err:4 http://mirrors.kernel.org/ubuntu xenial-updates InRelease
  At least one invalid signature was encountered.
Hit:18 h...

Revision history for this message
Seth Arnold (seth-arnold) wrote :
Download full text (4.0 KiB)

The files in partial/ don't look too damning:

root@hunt:/var/lib/apt/lists/partial# file *
ftp.debian.org_debian_dists_jessie-updates_contrib_source_Sources: empty
ftp.debian.org_debian_dists_wheezy-updates_contrib_source_Sources: empty
mirrors.kernel.org_ubuntu_dists_xenial-proposed_restricted_source_Sources: empty
mirrors.kernel.org_ubuntu_dists_yakkety-proposed_multiverse_source_Sources: empty
mirrors.kernel.org_ubuntu_dists_yakkety-proposed_restricted_source_Sources: empty
mirrors.kernel.org_ubuntu_dists_yakkety-security_restricted_source_Sources: empty
mirrors.kernel.org_ubuntu_dists_yakkety-updates_restricted_source_Sources: empty
mirrors.kernel.org_ubuntu_dists_zesty-proposed_restricted_source_Sources: empty
security.debian.org_dists_jessie_updates_contrib_source_Sources: ASCII text, with very long lines
root@hunt:/var/lib/apt/lists/partial# cat security.debian.org_dists_jessie_updates_contrib_source_Sources
Package: virtualbox
Binary: virtualbox-qt, virtualbox, virtualbox-dbg, virtualbox-dkms, virtualbox-source, virtualbox-guest-dkms, virtualbox-guest-source, virtualbox-guest-x11, virtualbox-guest-utils
Version: 4.3.36-dfsg-1+deb8u1
Maintainer: Debian Virtualbox Team <email address hidden>
Uploaders: Ritesh Raj Sarraf <email address hidden>, Gianfranco Costamagna <email address hidden>
Build-Depends: bzip2, debhelper (>= 9), default-jdk, dh-python, dkms (>= 2.1.1.1), docbook-xml, docbook-xsl, dpkg-dev (>= 1.15.6~), g++-multilib, genisoimage, gsoap (>= 2.8.16), iasl, imagemagick, kbuild (>= 1:0.1.9998svn2695), libasound2-dev, libcap-dev, libcurl4-gnutls-dev, libdevmapper-dev, libdrm-dev, libgl1-mesa-dev, libglu1-mesa-dev, libidl-dev, libpam0g-dev, libpixman-1-dev, libpng-dev, libpulse-dev, libqt4-dev (>= 4.4.0), libqt4-network (>= 4.4.0), libqt4-opengl-dev (>= 4.4.0), libsdl1.2-dev, libssl-dev, libvncserver-dev, libvpx-dev, libx11-dev, libxcomposite-dev, libxcursor-dev, libxdamage-dev, libxext-dev, libxi-dev, libxinerama-dev, libxml2-dev, libxmu-dev, libxrandr-dev, libxrender-dev, libxslt1-dev, libxt-dev, lsb-release, lynx-cur, makeself, module-assistant, python-dev (>= 2.6.6-3~), texlive-fonts-extra, texlive-fonts-recommended, texlive-latex-extra, texlive-latex-recommended, uuid-dev, x11proto-gl-dev, x11proto-xf86dri-dev, xserver-xorg-dev, xsltproc, yasm (>= 0.7.0), zlib1g-dev
Architecture: amd64 i386 all
Standards-Version: 3.9.6
Format: 3.0 (quilt)
Files:
 a21ddb4a21ad729519508d28b14e20b5 3696 virtualbox_4.3.36-dfsg-1+deb8u1.dsc
 1423337a5a9970dda72e60fcaa0f8d05 47713148 virtualbox_4.3.36-dfsg.orig.tar.xz
 8010c3b4e28f7910e44d9ec9ea9376ef 75292 virtualbox_4.3.36-dfsg-1+deb8u1.debian.tar.xz
Vcs-Browser: http://anonscm.debian.org/gitweb/?p=pkg-virtualbox/virtualbox.git
Vcs-Git: git://anonscm.debian.org/pkg-virtualbox/virtualbox.git
Checksums-Sha1:
 686d044b04e48816db96db89b425f3758ba80e26 3696 virtualbox_4.3.36-dfsg-1+deb8u1.dsc
 b458c3c7ce0f1e9081dbcde9e39325653962a449 47713148 virtualbox_4.3.36-dfsg.orig.tar.xz
 55a72ca9a4ddcd19fa983b31f2a519273eea51b8 75292 virtualbox_4.3.36-dfsg-1+deb8u1.debian.tar.xz
Checksums-Sha256:
 f5703f0247ad06c375f529ace969cee15627a4d670fc948f13e8c9eb...

Read more...

Revision history for this message
Seth Arnold (seth-arnold) wrote :
Download full text (7.6 KiB)

On a whim I ran apt-get update through valgrind:

==22064==
==22064== HEAP SUMMARY:
==22064== in use at exit: 695,606 bytes in 7,018 blocks
==22064== total heap usage: 67,584 allocs, 60,566 frees, 18,503,180 bytes allocated
==22064==
==22064== LEAK SUMMARY:
==22064== definitely lost: 0 bytes in 0 blocks
==22064== indirectly lost: 0 bytes in 0 blocks
==22064== possibly lost: 0 bytes in 0 blocks
==22064== still reachable: 695,606 bytes in 7,018 blocks
==22064== suppressed: 0 bytes in 0 blocks
==22064== Rerun with --leak-check=full to see details of leaked memory
==22064==
==22064== For counts of detected and suppressed errors, rerun with: -v
==22064== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==20846== Conditional jump or move depends on uninitialised value(s)
==20846== at 0x4F4DA00: pkgCache::ReMap(bool const&) (in /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0.0)
==20846== by 0x4F55E68: ??? (in /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0.0)
==20846== by 0x4F580B4: ??? (in /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0.0)
==20846== by 0x4EBAA51: pkgCacheFile::BuildCaches(OpProgress*, bool) (in /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0.0)
==20846== by 0x51EE809: DoUpdate(CommandLine&) (in /usr/lib/x86_64-linux-gnu/libapt-private.so.0.0.0)
==20846== by 0x4ECEA25: CommandLine::DispatchArg(CommandLine::Dispatch const*, bool) (in /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0.0)
==20846== by 0x51C0E62: DispatchCommandLine(CommandLine&, std::vector<CommandLine::Dispatch, std::allocator<CommandLine::Dispatch> > const&) (in /usr/lib/x86_64-linux-gnu/libapt-private.so.0.0.0)
==20846== by 0x10BB38: ??? (in /usr/bin/apt-get)
==20846== by 0x59B682F: (below main) (libc-start.c:291)
==20846== Uninitialised value was created by a stack allocation
==20846== at 0x4F55B4D: ??? (in /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0.0)
==20846==
==20846== Syscall param write(buf) points to uninitialised byte(s)
==20846== at 0x5A8C6E0: __write_nocancel (syscall-template.S:84)
==20846== by 0x4ED6B13: FileFd::Write(void const*, unsigned long long) (in /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0.0)
==20846== by 0x4E7A460: ??? (in /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0.0)
==20846== by 0x4F58146: ??? (in /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0.0)
==20846== by 0x4EBAA51: pkgCacheFile::BuildCaches(OpProgress*, bool) (in /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0.0)
==20846== by 0x51EE809: DoUpdate(CommandLine&) (in /usr/lib/x86_64-linux-gnu/libapt-private.so.0.0.0)
==20846== by 0x4ECEA25: CommandLine::DispatchArg(CommandLine::Dispatch const*, bool) (in /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0.0)
==20846== by 0x51C0E62: DispatchCommandLine(CommandLine&, std::vector<CommandLine::Dispatch, std::allocator<CommandLine::Dispatch> > const&) (in /usr/lib/x86_64-linux-gnu/libapt-private.so.0.0.0)
==20846== by 0x10BB38: ??? (in /usr/bin/apt-get)
==20846== by 0x59B682F: (below main) (libc-start.c:291)
==20846== Address 0xd003007 is in a rw- anonymous segment
==20846==
==20846== Syscall param write(buf) points to uninitialised byte(s)
==20846== at 0x5A8C6E0: __write_...

Read more...

Revision history for this message
Seth Arnold (seth-arnold) wrote :
Download full text (9.1 KiB)

Turns out the valgrind messages aren't regressions either.

Here's the older apt packages again which seemed to work okay:

==25043==
==25043== HEAP SUMMARY:
==25043== in use at exit: 13,118,211 bytes in 170,033 blocks
==25043== total heap usage: 626,066 allocs, 456,033 frees, 69,255,845 bytes allocated
==25043==
==25043== LEAK SUMMARY:
==25043== definitely lost: 0 bytes in 0 blocks
==25043== indirectly lost: 0 bytes in 0 blocks
==25043== possibly lost: 0 bytes in 0 blocks
==25043== still reachable: 13,118,211 bytes in 170,033 blocks
==25043== suppressed: 0 bytes in 0 blocks
==25043== Rerun with --leak-check=full to see details of leaked memory
==25043==
==25043== For counts of detected and suppressed errors, rerun with: -v
==25043== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==23213== Conditional jump or move depends on uninitialised value(s)
==23213== at 0x4F4D240: pkgCache::ReMap(bool const&) (in /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0.0)
==23213== by 0x4F55598: ??? (in /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0.0)
==23213== by 0x4F577E4: ??? (in /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0.0)
==23213== by 0x4EBA081: pkgCacheFile::BuildCaches(OpProgress*, bool) (in /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0.0)
==23213== by 0x51EC5A1: DoUpdate(CommandLine&) (in /usr/lib/x86_64-linux-gnu/libapt-private.so.0.0.0)
==23213== by 0x4ECDFA5: CommandLine::DispatchArg(CommandLine::Dispatch const*, bool) (in /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0.0)
==23213== by 0x51BEDF2: DispatchCommandLine(CommandLine&, std::vector<CommandLine::Dispatch, std::allocator<CommandLine::Dispatch> > const&) (in /usr/lib/x86_64-linux-gnu/libapt-private.so.0.0.0)
==23213== by 0x10BB38: ??? (in /usr/bin/apt-get)
==23213== by 0x59B482F: (below main) (libc-start.c:291)
==23213== Uninitialised value was created by a stack allocation
==23213== at 0x4F5527D: ??? (in /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0.0)
==23213==
==23213== Syscall param write(buf) points to uninitialised byte(s)
==23213== at 0x5A8A6E0: __write_nocancel (syscall-template.S:84)
==23213== by 0x4ED5CAB: FileFd::Write(void const*, unsigned long long) (in /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0.0)
==23213== by 0x4E7A150: ??? (in /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0.0)
==23213== by 0x4F57876: ??? (in /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0.0)
==23213== by 0x4EBA081: pkgCacheFile::BuildCaches(OpProgress*, bool) (in /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0.0)
==23213== by 0x51EC5A1: DoUpdate(CommandLine&) (in /usr/lib/x86_64-linux-gnu/libapt-private.so.0.0.0)
==23213== by 0x4ECDFA5: CommandLine::DispatchArg(CommandLine::Dispatch const*, bool) (in /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0.0)
==23213== by 0x51BEDF2: DispatchCommandLine(CommandLine&, std::vector<CommandLine::Dispatch, std::allocator<CommandLine::Dispatch> > const&) (in /usr/lib/x86_64-linux-gnu/libapt-private.so.0.0.0)
==23213== by 0x10BB38: ??? (in /usr/bin/apt-get)
==23213== by 0x59B482F: (below main) (libc-start.c:291)
==23213== Address 0xe401007 is in a rw- anonymous segment
==23213==
==23213== Syscal...

Read more...

Revision history for this message
Julian Andres Klode (juliank) wrote :

Yeah, valgrind is a bit noisy always because we are building the cache in memory before (1) writing it to the disk and the write includes unused regions and (2) we are hashing the entire thing before writing it, including the uninitialised bytes. So that means while we do have a few uninitialized bytes, it's actually safe.

This bug is fairly strange. Especially git vs packages. The only thing different when building via git is that hardening flags are not used. You could export those:

  DEB_BUILD_MAINT_OPTIONS=hardening=+all dpkg-buildflags
  export CXXFLAGS LDFLAGS CPPFLAGS

but I seriously doubt that's the problem (if git always worked instead of always failed, this might have made sense).
If I had a user account on an affected machine, where

If I had a user account on an affected machine (it needs to be reproducible by creating a fake root directory, and copying etc/apt and var/lib/apt to it, then I can use -o Dir=$PATH_TO_FAKE_ROOT instead of needing root), where I can build apt and have tools like valgrind, gdb; I could (try to) debug that myself.

Revision history for this message
Seth Arnold (seth-arnold) wrote :

Julian, thanks for your patience. I'm not able to offer a shell on the affected machine, so debugging this is just going to have to go at a snail's pace.

I read strace and ltrace logs from both 1.2.12-ish and 1.2.15 apt packages and narrowed it down to /usr/bin/apt-key.

When I use the /usr/bin/apt-key from apt_1.2.12~ubuntu16.04.1_amd64.deb (but everything else from 1.2.15 packaging) my apt-get update runs as I expect.

I'll attach the diff between the two (NOT A FIX).

Thanks

Revision history for this message
Julian Andres Klode (juliank) wrote :

Ah, apt-key explains it. We did not catch that in the bisect correctly, as I forgot to specify -o Dir::Bin::apt-key=$PWD/build/bin/apt-key :/

This means it is a regression introduced in:

commit b515fe3a0012c1f155dbf6a4199e919fec102578
Author: David Kalnischkies <email address hidden>
Date: Thu Jun 2 11:12:39 2016 +0200

    apt-key: change to / before find to satisfy its CWD needs

    First seen on hurd, but easily reproducible on all systems by removing
    the 'execution' bit from the current working directory and watching some
    tests (mostly the no-output expecting tests) fail due to find printing:
    "find: Failed to restore initial working directory: …"

    Samuel Thibault says in the bugreport:
    | To do its work, find first records the $PWD, then goes to
    | /etc/apt/trusted.gpg.d/ to find the files, and then goes back to $PWD.
    |
    | On Linux, getting $PWD from the 700 directory happens to work by luck
    | (POSIX says that getcwd can return [EACCES]: Search permission was denied
    | for the current directory, or read or search permission was denied for a
    | directory above the current directory in the file hierarchy). And going
    | back to $PWD fails, and thus find returns 1, but at least it emitted its
    | output.
    |
    | On Hurd, getting $PWD from the 700 directory fails, and find thus aborts
    | immediately, without emitting any output, and thus no keyring is found.
    |
    | So, to summarize, the issue is that since apt-get update runs find as a
    | non-root user, running it from a 700 directory breaks find.

    Solved as suggested by changing to '/' before running find, with some
    paranoia extra care taking to ensure the paths we give to find are really
    absolute paths first (they really should, but TMPDIR=. or a similar
    Dir::Etc::trustedparts setting could exist somewhere in the wild).

    The commit takes also the opportunity to make these lines slightly less
    error ignoring and the two find calls using (mostly) the same parameters.

    Thanks: Samuel Thibault for 'finding' the culprit!
    Closes: 826043
    (cherry picked from commit 0cfec3ab589c6309bf284438d2148c7742cdaf10)

Revision history for this message
Julian Andres Klode (juliank) wrote :

OK, the reason this happens is that some of your key files are not readable (I'm not sure, but it might be running as _apt). The commit mentioned introduced a regresssion in that it does not ignore failures from unreadable key files.

This was fixed in 1.3~rc3 in commit 105503b4b470c124bc0c271bd8a50e25ecbe9133. I cherry-picked that change in my for-1.2/apt-key branch in https://github.com/julian-klode/apt.

You should be able to verify this by adding -o Dir::Bin::Apt-Key="$PWD/build/bin/apt-key" to the apt-get invocation in the script and then running it once with the normal 1.2.y branch and once with my for-1.2/apt-key branch.

The test suite currently fails, as the new tests added depend on some other changes, once I got those merged I can upload it as 1.2.17 (1.2.16 is already in the unapproved queue for -proposed, it fixes bugs with localized strings in protocols).

Changed in apt (Ubuntu Xenial):
status: New → Triaged
Changed in apt (Ubuntu):
status: New → Fix Released
Revision history for this message
Myk Dowling (politas) wrote :

Is there something we Xenial users need to do to resolve this?

Revision history for this message
Julian Andres Klode (juliank) wrote :

You can fix your permissions on your trusted.gpg and trusted.gpg.d files in /etc/apt, so that the files are world-readable (chmod ugo+r /etc/apt/trusted.gpg /etc/apt/trusted.gpg.d -R) [or give access to root and _apt via acls].

You don't have to do that, though - it will start "working" again in 1.2.17. "working" in the sense that unreadable files are ignored (or warned about, not sure yet).

Revision history for this message
Julian Andres Klode (juliank) wrote :

But of course: If you safely want to get apt 1.2.17 via apt 1.2.15, you have to have correct permissions first - otherwise your old apt won't see the new apt.

Revision history for this message
Seth Arnold (seth-arnold) wrote :

Julian, looks like you win this year's remote-debugging-via-crystal-ball award!

$ find /etc/apt -ls | grep sarnold
  2572875 4 -rw------- 1 sarnold sarnold 1740 Mar 23 2016 /etc/apt/trusted.gpg.d/ddebs.gpg

Well done :D

Thanks

Revision history for this message
Seth Arnold (seth-arnold) wrote :

Julian, I modified the script.sh to include the following line:

sudo LD_LIBRARY_PATH=$PWD/build/bin/ ./build/bin/apt-get update -o Dir::Bin::Apt-Key="$PWD/build/bin/apt-key" -o Dir::Bin::Methods="$PWD/build/bin/methods/" 2>&1 | tee update.log

When run from remotes/julian/for-1.2/apt-key I get the usual progress I expect.
When run from remotes/origin/1.2.y I get the errors as described above.

Oddly enough I don't see the text "The key(s) in the keyring $1 are ignored as the file is not readable by user '$USER' executing apt-key." in the output. (But I don't think I'm currently configured to download sources from ddebs.ubuntu.com, which is the host corresponding to the unreadable key.)

Thanks

Revision history for this message
Julian Andres Klode (juliank) wrote :

Thanks for verifying. I should have the final update ready within the next 16-48 hours.

Revision history for this message
Myk Dowling (politas) wrote :

>You can fix your permissions on your trusted.gpg and trusted.gpg.d files in /etc/apt, so that the files are world-readable (chmod ugo+r /etc/apt/trusted.gpg /etc/apt/trusted.gpg.d -R) [or give access to root and _apt via acls].

That has been successful for me.

Revision history for this message
Julian Andres Klode (juliank) wrote :

Uploaded 1.2.17 to xenial-proposed

apt (1.2.17) xenial; urgency=medium

  [ David Kalnischkies ]
  * apt-key: warn instead of fail on unreadable keyrings (LP: #1642386)
  * show apt-key warnings in apt update (Closes: 834973)

  [ Julian Andres Klode ]
  * test-releasefile-verification: installaptold: Clean up before run

 -- Julian Andres Klode <email address hidden> Wed, 23 Nov 2016 20:09:27 +0100

Changed in apt (Ubuntu Xenial):
status: Triaged → In Progress
description: updated
Revision history for this message
Julian Andres Klode (juliank) wrote :

Notably the first commit just causes the thing to fail silently, the second one makes apt forward the warning on an update.

Robie Basak (racb)
description: updated
tags: added: regression-update
Revision history for this message
Adam Conrad (adconrad) wrote : Please test proposed package

Hello Seth, or anyone else affected,

Accepted apt into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.2.18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in apt (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Seth Arnold (seth-arnold) wrote :

Thanks Julian!

I tested the apt 1.2.18 packages and found the results far more pleasing:

...
Fetched 1,688 kB in 2s (685 kB/s)
Reading package lists... Done
W: http://mirrors.kernel.org/ubuntu/dists/xenial/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ddebs.gpg are ignored as the file is not readable by user 'root' executing apt-key.
W: http://mirrors.kernel.org/ubuntu/dists/xenial-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ddebs.gpg are ignored as the file is not readable by user 'root' executing apt-key.
...

It may be a bit verbose, one line per configured source, but for the average user it shouldn't be overwhelming, and it very clearly points to the cause of the problem and the solution.

Thanks for working through this with me, I know it took a lot of time and effort.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apt - 1.2.18

---------------
apt (1.2.18) xenial; urgency=high

  * SECURITY UPDATE: gpgv: Check for errors when splitting files (CVE-2016-1252)
    Thanks to Jann Horn, Google Project Zero for reporting the issue
    (LP: #1647467)
  * gpgv: Flush the files before checking for errors

apt (1.2.17) xenial; urgency=medium

  [ David Kalnischkies ]
  * apt-key: warn instead of fail on unreadable keyrings (LP: #1642386)
  * show apt-key warnings in apt update (Closes: 834973)

  [ Julian Andres Klode ]
  * test-releasefile-verification: installaptold: Clean up before run

apt (1.2.16) xenial; urgency=medium

  [ David Kalnischkies ]
  * avoid changing the global LC_TIME for Release writing
  * use de-localed std::put_time instead rolling our own
  * accept only the expected UTC timezones in date parsing (Closes: 819697)
  * avoid std::get_time usage to sidestep libstdc++6 bug (LP: #1593583)
  * imbue datetime parsing with C.UTF-8 locale (Closes: 828011)
  * prevent C++ locale number formatting in text APIs (try 2) (Closes: 832044)
  * prevent C++ locale number formatting in text APIs (try 3) (LP: #1611010)
    (LP: #1592817)
  * imbue .diff/Index parsing with C.UTF-8 as well

  [ Julian Andres Klode ]
  * Use C locale instead of C.UTF-8 for protocol strings
  * Add shippable.yml for CI on Shippable
  * Revert "if the FileFd failed already following calls should fail, too"
    (LP: #1641905)

 -- Julian Andres Klode <email address hidden> Thu, 08 Dec 2016 15:28:08 +0100

Changed in apt (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for apt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.