Captive portals may corrupt apt translation indices

The attached file was the content of vim /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_natty_main_i18n_Translation-en .

Here's the error message of apt-get:

pacaud@lappc-p348:~$ sudo apt-get update
[sudo] password for pacaud:
Sorry, try again.
[sudo] password for pacaud:
Ign http://archive.canonical.com natty InRelease
Ign http://archive.ubuntu.com natty InRelease
Ign http://archive.ubuntu.com natty-updates InRelease
Ign http://archive.ubuntu.com natty-security InRelease
Réception de : 1 http://archive.canonical.com natty Release.gpg [198 B]
Ign http://archive.ubuntu.com natty-proposed InRelease
Ign http://archive.ubuntu.com natty-backports InRelease
Réception de : 2 http://archive.ubuntu.com natty Release.gpg [198 B]
Réception de : 3 http://archive.canonical.com natty Release [5 916 B]
Réception de : 4 http://archive.ubuntu.com natty-updates Release.gpg [198 B]
Réception de : 5 http://archive.ubuntu.com natty-security Release.gpg [198 B]
Réception de : 6 http://archive.ubuntu.com natty-proposed Release.gpg [198 B]
Réception de : 7 http://archive.ubuntu.com natty-backports Release.gpg [198 B]
Réception de : 8 http://archive.canonical.com natty/partner amd64 Packages [4 393 B]
Réception de : 9 http://archive.ubuntu.com natty Release [39,8 kB]
Ign http://archive.canonical.com natty/partner TranslationIndex
Réception de : 10 http://archive.ubuntu.com natty-updates Release [23,2 kB]
Réception de : 11 http://archive.ubuntu.com natty-security Release [23,2 kB]
Réception de : 12 http://archive.ubuntu.com natty-proposed Release [23,2 kB]
Réception de : 13 http://archive.ubuntu.com natty-backports Release [23,2 kB]
Réception de : 14 http://archive.ubuntu.com natty/main Sources [864 kB]
Ign http://archive.canonical.com natty/partner Translation-fr_FR
Ign http://archive.canonical.com natty/partner Translation-fr
Ign http://archive.canonical.com natty/partner Translation-en
Réception de : 15 http://archive.ubuntu.com natty/restricted Sources [4 117 B]
Réception de : 16 http://archive.ubuntu.com natty/universe Sources [4 381 kB]
Réception de : 17 http://archive.ubuntu.com natty/multiverse Sources [155 kB]
Réception de : 18 http://archive.ubuntu.com natty/main amd64 Packages [1 557 kB]
Réception de : 19 http://archive.ubuntu.com natty/restricted amd64 Packages [9 027 B]
Réception de : 20 http://archive.ubuntu.com natty/universe amd64 Packages [6 010 kB]
Réception de : 21 http://archive.ubuntu.com natty/multiverse amd64 Packages [180 kB]
Ign http://archive.ubuntu.com natty/main TranslationIndex
Ign http://archive.ubuntu.com natty/multiverse TranslationIndex
Ign http://archive.ubuntu.com natty/restricted TranslationIndex
Ign http://archive.ubuntu.com natty/universe TranslationIndex
Atteint http://archive.ubuntu.com natty-updates/main Sources
Atteint http://archive.ubuntu.com natty-updates/restricted Sources
Atteint http://archive.ubuntu.com natty-updates/universe Sources
Atteint http://archive.ubuntu.com natty-update...

mama@zeuza:~$ sudo wget http://mamalibre.no-ip.org/mamalibre-lucid.list -O /etc/apt/sources.list.d/mamalibre.list && sudo apt-get update
--2011-04-17 21:10:01-- http://mamalibre.no-ip.org/mamalibre-lucid.list
Resolviendo mamalibre.no-ip.org... 190.19.160.85
Conectando a mamalibre.no-ip.org|190.19.160.85|:80... conectado.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 54 [application/octet-stream]
Guardando en: «/etc/apt/sources.list.d/mamalibre.list»

100%[====================================================================================>] 54 --.-K/s en 0s

2011-04-17 21:10:02 (4,79 MB/s) - «/etc/apt/sources.list.d/mamalibre.list» guardado [54/54]

Des:1 http://mamalibre.no-ip.org lucid InRelease [4.147 B]
Ign http://mamalibre.no-ip.org lucid InRelease
Des:2 http://mamalibre.no-ip.org lucid/main i386 Packages [5.689 B]
Des:3 http://mamalibre.no-ip.org lucid/main TranslationIndex [4.147 B]
Des:4 http://mamalibre.no-ip.org lucid/main Translation-es_AR [4.147 B]
Des:5 http://mamalibre.no-ip.org lucid/main Translation-es [4.147 B]
81% [4 Translation-es_AR bzip2 0 B] [Conectando a mirrors.us.kernel.org (149.20.20.135)] [Conectando a extras.ubuntu.com (91.bzip2: (stdin) is not a bzip2 file.
81% [5 Translation-es bzip2 0 B] [Conectando a mirrors.us.kernel.org (149.20.20.135)] [Conectando a extras.ubuntu.com (91.189bzip2: (stdin) is not a bzip2 file.
Des:6 http://mamalibre.no-ip.org lucid/main Translation-en [4.147 B]
84% [6 Translation-en bzip2 0 B] [Conectando a mirrors.us.kernel.org (149.20.20.135)] [Conectando a extras.ubuntu.com (91.189bzip2: (stdin) is not a bzip2 file.
84% [4 Translation-es_AR xz 0 B] [Esperando las cabeceras] [Conectando a extras.ubuntu.com (91.189.88.33)] [Conectando a ppa./usr/bin/xz: (stdin): Formato de archivo no reconocido
84% [5 Translation-es xz 0 B] [Esperando las cabeceras] [Conectando a extras.ubuntu.com (91.189.88.33)] [Conectando a ppa.lau/usr/bin/xz: (stdin): Formato de archivo no reconocido
84% [4 Translation-es_AR lzma 0 B] [6 Translation-en xz 0 B] [Esperando las cabeceras] [Conectando a extras.ubuntu.com (91.18/usr/bin/xz: (stdin): Formato de archivo no reconocido
84% [4 Translation-es_AR lzma 0 B] [Esperando las cabeceras] [Conectando a extras.ubuntu.com (91.189.88.33)] [Conectando a pp/usr/bin/lzma: Decoder error
84% [5 Translation-es lzma 0 B] [Esperando las cabeceras] [Conectando a extras.ubuntu.com (91.189.88.33)] [Esperando las cabe/usr/bin/lzma: Decoder error
84% [6 Translation-en lzma 0 B] [Esperando las cabeceras] [Esperando las cabeceras] [Esperando las cabeceras] [Esperando las /usr/bin/lzma: Decoder error
Ign http://mirrors.us.kernel.org natty InRelease
Ign http://mirrors.us.kernel.org natty-updates InRelease ...

I inserted the content of the attached file into
archive.ubuntu.com_ubuntu_dists_precise-updates_restricted_i18n_Translation-en
on Precise and the problem still exists:

Ign http://archive.ubuntu.com lucid/universe Translation-en_US
Ign http://archive.ubuntu.com lucid/universe Translation-en
Reading package lists... Error!
E: Encountered a section with no Package: header
E: Problem with MergeList /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_precise-updates_restricted_i18n_Translation-en
E: The package lists or status file could not be parsed or opened.

I don't think apt has an onboard method to clean files in the lists directory, they need to be removed manually based on the error above.

Still present in 12.04 LTS, Precise running apt 0.8.16

In my case the corrupted package list files in /var/lib/apt/lists are caused by the router redirecting to an internal help page when it realises that its internet connection is down. So, when a fetch is attempted from, say gb.archive.ubuntu.com/ubuntu/dists/precise-updates/universe/binary-i386/Packages when checking for updates what comes back is the html source from the router's help page (example attached -- line 52 contains the requested url).

It would appear that no sanity check is done on the returned data leaving subsequent parse attempts to choke. The corrupted files remain and may propagate (???) causing other update failures.

On a security note, it occurs to me that an attacker in control of the router could return crafted files in place of apt's package lists to introduce malware as part of the normal automated update process. I trust checks are in place to prevent this???

See also Bug #1001209, Bug #1034834

See also Bug #1055614

See also Bug #756317

This is indeed still a bug in 12.04. When doing an apt-get update behind a captive portal, the lists files get corrupted and won't be fixed even after the portal is gone. The only fix is to remove those files.

And I've been wondering what causes these MergeList issues so often (not for me, but for other Mint users)...

Yes, this bug is a PITA. I can't see why something as important as an
update list isn't cryptographically verified. Heck, even a quick md5sum
check would catch this 99.99999% of the time.

Bryan Harris, PE
Research Engineer
Structures and Materials Evaluation Group
University of Dayton Research Institute
<email address hidden>
http://www.udri.udayton.edu/
(937) 229-5561

On Thu, Mar 20, 2014 at 2:17 PM, Monsta <email address hidden> wrote:

> ** Bug watch added: Debian Bug tracker #710229
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=710229
>
> ** Also affects: apt (Debian) via
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=710229
> Importance: Unknown
> Status: Unknown
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1055614).
> https://bugs.launchpad.net/bugs/756317
>
> Title:
> Captive portals may corrupt apt package lists
>
> Status in "apt" package in Ubuntu:
> Confirmed
> Status in "apt" package in Debian:
> Unknown
>
> Bug description:
> I have an adsl modem which returns an html page if the adsl link is
> broken. This page ends as the content of the apt cache files stored in
> /var/lib/apt/lists, which breaks apt.
>
> The only way to make apt work again is to delete all the files stored
> in /var/lib/apt/lists.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/apt/+bug/756317/+subscriptions
>

Even ignoring that fact that this is a huge security issue, a computer
connecting to free wifi at Starbucks should not irreversibly corrupt the
update process requiring manual intervention.

Bryan Harris, PE
Research Engineer
Structures and Materials Evaluation Group
University of Dayton Research Institute
<email address hidden>
http://www.udri.udayton.edu/
(937) 229-5561

On Thu, Mar 20, 2014 at 3:01 PM, Bryan Harris <email address hidden>wrote:

> Yes, this bug is a PITA. I can't see why something as important as an
> update list isn't cryptographically verified. Heck, even a quick md5sum
> check would catch this 99.99999% of the time.
>
> Bryan Harris, PE
> Research Engineer
> Structures and Materials Evaluation Group
> University of Dayton Research Institute
> <email address hidden>
> http://www.udri.udayton.edu/
> (937) 229-5561
>
>
> On Thu, Mar 20, 2014 at 2:17 PM, Monsta <email address hidden> wrote:
>
>> ** Bug watch added: Debian Bug tracker #710229
>> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=710229
>>
>> ** Also affects: apt (Debian) via
>> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=710229
>> Importance: Unknown
>> Status: Unknown
>>
>> --
>> You received this bug notification because you are subscribed to a
>> duplicate bug report (1055614).
>> https://bugs.launchpad.net/bugs/756317
>>
>> Title:
>> Captive portals may corrupt apt package lists
>>
>> Status in "apt" package in Ubuntu:
>> Confirmed
>> Status in "apt" package in Debian:
>> Unknown
>>
>> Bug description:
>> I have an adsl modem which returns an html page if the adsl link is
>> broken. This page ends as the content of the apt cache files stored in
>> /var/lib/apt/lists, which breaks apt.
>>
>> The only way to make apt work again is to delete all the files stored
>> in /var/lib/apt/lists.
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/ubuntu/+source/apt/+bug/756317/+subscriptions
>>
>
>

I'm not sure why I couldn't convince the security team that this is a
security issue. The ability for an attacker to write arbitrary information
to your software update database sounds like a pretty darn big security
flaw.

Bryan Harris, PE
Research Engineer
Structures and Materials Evaluation Group
University of Dayton Research Institute
<email address hidden>
http://www.udri.udayton.edu/
(937) 229-5561

On Thu, Mar 20, 2014 at 3:04 PM, Bryan Harris <email address hidden>wrote:

> Even ignoring that fact that this is a huge security issue, a computer
> connecting to free wifi at Starbucks should not irreversibly corrupt the
> update process requiring manual intervention.
>
> Bryan Harris, PE
> Research Engineer
> Structures and Materials Evaluation Group
> University of Dayton Research Institute
> <email address hidden>
> http://www.udri.udayton.edu/
> (937) 229-5561
>
>
> On Thu, Mar 20, 2014 at 3:01 PM, Bryan Harris <email address hidden>wrote:
>
>> Yes, this bug is a PITA. I can't see why something as important as an
>> update list isn't cryptographically verified. Heck, even a quick md5sum
>> check would catch this 99.99999% of the time.
>>
>> Bryan Harris, PE
>> Research Engineer
>> Structures and Materials Evaluation Group
>> University of Dayton Research Institute
>> <email address hidden>
>> http://www.udri.udayton.edu/
>> (937) 229-5561
>>
>>
>> On Thu, Mar 20, 2014 at 2:17 PM, Monsta <email address hidden>wrote:
>>
>>> ** Bug watch added: Debian Bug tracker #710229
>>> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=710229
>>>
>>> ** Also affects: apt (Debian) via
>>> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=710229
>>> Importance: Unknown
>>> Status: Unknown
>>>
>>> --
>>> You received this bug notification because you are subscribed to a
>>> duplicate bug report (1055614).
>>> https://bugs.launchpad.net/bugs/756317
>>>
>>> Title:
>>> Captive portals may corrupt apt package lists
>>>
>>> Status in "apt" package in Ubuntu:
>>> Confirmed
>>> Status in "apt" package in Debian:
>>> Unknown
>>>
>>> Bug description:
>>> I have an adsl modem which returns an html page if the adsl link is
>>> broken. This page ends as the content of the apt cache files stored in
>>> /var/lib/apt/lists, which breaks apt.
>>>
>>> The only way to make apt work again is to delete all the files stored
>>> in /var/lib/apt/lists.
>>>
>>> To manage notifications about this bug go to:
>>> https://bugs.launchpad.net/ubuntu/+source/apt/+bug/756317/+subscriptions
>>>
>>
>>
>

As I said in a duplicate bug, I think this is a serious issue both from a user viewpoint (i've seen people switching back to Windows due to this bug, since a "normal" user doesn't know how to fix this) and a security viewpoint. The captive portal owner could easily insert a malicious repository into the APT configuration.

I believe we already fixed that, there are even test cases for something like this in APT. Probably someone missed something.

There is no security issue here. We verify that package indices match the checksums specified in the (signed) Release file. If the Release file is corrupt, we issue a warning that we cannot verify the packages.

'No security issue' overstates the position I'd say.

Joe User will be used to ignoring installation warnings. How often does a Windows user click Carry on Regardless when installing drivers that have not been WHQL certified despite a similar warning? Every time, probably.

Of course this is not fixed anywhere.

E: Encountered a section with no Package: header
E: Problem with MergeList /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_saucy-security_main_i18n_Translation-en
E: The package lists or status file could not be parsed or opened.
E: _cache->open() failed, please report.

APT cannot recover from it automatically, and that is the serious problem for the unsuspecting users: they have no idea that they need to remove /var/lib/apt/lists/* manually and then update the package index to make APT work again.

This problem happens so often these days that the Linux Mint developers even had to add the "Fix MergeList problem" button to their software sources management tool.

Well, there is a magical word which helps in getting everything you want and the word is: "patch", but I don't see one here… doesn't seem THAT important after all it seems.

That recent reporters all complain about Translation-* files is btw only possible because not all repositories (read: nobody expect Debian itself) provide checksums for these files in the (In)Release file [some provide a i18n/Index file with similar content which was used for a while as a stopgap, but it was deemed to hard to support unsigned, stopgap and signed properly in apt and as everyone could just switch to the real thing the stopgap was dropped…]. If they would, they wouldn't need to implement a "Fix MergeList problem" button, but buttons are easier than fixing the repository creation process used internally by your own distribution of course… just saying…
(it would solve a few more issues of course, but given that not even this problem is helping the adoption I doubt any other would… I at least gave up years ago)
[Well, you would still see this with repositories without a Release file of course]

Oh, and while at it tell all your captive portal providers to do their job properly and report that they are one (e.g. "511 Network Authentication Required" and similar such instead of "200 OK" while nothing is "okay", okay?).

For both things you probably can't provide a patch, so that is were you should complain, not here, were you could provide one but nobody does. APT is just the messenger here and you should never kill the messenger for the content of the message… even if it is a HTML message and you expected a deb822 one.

> "patch"

That would certainly be useful.

But seriously, complaining over semi-broken captive portals? You need a vacation.

Fixing an unknown number, but probably hundreds of thousands, broken routers
mostly operated by non-tech-savvy people is not going to happen in a timely manner.
They will get replaced when they fail and the replacements will have a new set of
bugs.

So where do we stand?

1. APT cannot recover from receiving broken files. This is *not* just the result of
    captive portals. Truncated files -- even zero-length files -- seem to cause it
    trouble too.

2. Anyone with a router can stop a user from getting security updates from then on.
    Just hand out an IP address and serve a broken file. Yes, that really is a security
    issue.

*You* need to stop blaming the messengers. The problem here is cutting corners in
the design: putting that amount of trust on the network is not "best practices" and
hasn't been for 3-4 decades.

I probably shouldn't write all this without being constructive myself, so here goes:

Item 1 seems to be fixable with a basic syntax check on the file. If the check fails,
toss the file and life goes on.

Item 2 is much trickier. A full fix probably requires signatures or strong checksums, i.e.,
it cannot happen in APT alone, but APT could certainly issue a "HEAD" request and
verify basic things like file length.

APT does basic syntax checks before moving the files in on most indices. Optional indices are excluded from that, and Translation indices seem to be optional. I do not know why, but David probably does.

APT does check signatures for everything important. I don't know whether that includes translation indices, it does not seem to be the case if those errors happen. But that is not problematic apart from the fact that mergelist fails. It would only get problematic if we crashed or executed code we should not execute.

It's problematic for one simple reason: APT ceases to function normally until you remove the offending files and update the package index again.

On 21/03/14 16:41, Monsta wrote:
> It's problematic for one simple reason: APT ceases to function normally
> until you remove the offending files and update the package index again.
>

Simply, it is a Denial of Service.

Quite a neat way to create a window of opportunity for so-called zero-day vulnerabilities too.

I'm willing to spend some time working on a patch.

Does anybody have an idea where in the code a syntax check should go?

Bryan Harris, PE
Research Engineer
Structures and Materials Evaluation Group
University of Dayton Research Institute
<email address hidden>
http://www.udri.udayton.edu/
(937) 229-5561

On Fri, Mar 21, 2014 at 4:41 PM, Monsta <email address hidden> wrote:

> It's problematic for one simple reason: APT ceases to function normally
> until you remove the offending files and update the package index again.
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1055614).
> https://bugs.launchpad.net/bugs/756317
>
> Title:
> Captive portals may corrupt apt translation indices
>
> Status in "apt" package in Ubuntu:
> Confirmed
> Status in "apt" package in Debian:
> New
>
> Bug description:
> I have an adsl modem which returns an html page if the adsl link is
> broken. This page ends as the content of the apt cache files stored in
> /var/lib/apt/lists, which breaks apt.
>
> The only way to make apt work again is to delete all the files stored
> in /var/lib/apt/lists.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/apt/+bug/756317/+subscriptions
>

I see that there are http and https methods for downloading items. What if
we just removed http download of files and defaulted to https? This would
neatly guarantee that a captive portal (either broken or malicious) cannot
get away with pretending to be a download server. Assuming https is
implemented properly, we would have authentication and encryption of all
downloads.

Bryan Harris, PE
Research Engineer
Structures and Materials Evaluation Group
University of Dayton Research Institute
<email address hidden>
http://www.udri.udayton.edu/
(937) 229-5561

On Fri, Mar 21, 2014 at 6:32 PM, Bryan Harris <email address hidden>wrote:

> I'm willing to spend some time working on a patch.
>
> Does anybody have an idea where in the code a syntax check should go?
>
> Bryan Harris, PE
> Research Engineer
> Structures and Materials Evaluation Group
> University of Dayton Research Institute
> <email address hidden>
> http://www.udri.udayton.edu/
> (937) 229-5561
>
>
> On Fri, Mar 21, 2014 at 4:41 PM, Monsta <email address hidden> wrote:
>
>> It's problematic for one simple reason: APT ceases to function normally
>> until you remove the offending files and update the package index again.
>>
>> --
>> You received this bug notification because you are subscribed to a
>> duplicate bug report (1055614).
>> https://bugs.launchpad.net/bugs/756317
>>
>> Title:
>> Captive portals may corrupt apt translation indices
>>
>> Status in "apt" package in Ubuntu:
>> Confirmed
>> Status in "apt" package in Debian:
>> New
>>
>> Bug description:
>> I have an adsl modem which returns an html page if the adsl link is
>> broken. This page ends as the content of the apt cache files stored in
>> /var/lib/apt/lists, which breaks apt.
>>
>> The only way to make apt work again is to delete all the files stored
>> in /var/lib/apt/lists.
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/ubuntu/+source/apt/+bug/756317/+subscriptions
>>
>
>

I already wrote it: The check already exist. It is in pkgAcqIndex::Done() in apt-pkg/acquire-item.cc, it is just disabled in commit c5f661b715fbd86fcbca694c44bb8422f01db267 for some type of indices (optional ones like translations). I do not know why.

This is not a denial of service. APT clearly informs you that your lists are broken, you can delete them, and things start working again. Unless you hit one of those stupid portals again.

This bug is not for discussing thing. We know that the bug exists, but we cannot fix this immediately.

We know that we need to enable the check again, but we first need to figure out why we disabled it in the first place.

Thank you.

Would be really nice if you'd fixed it before Ubuntu 14.04 LTS is finally released...

This is too complicated to fix before the trusty release. Any fix for this needs proper testing, and we cannot do this in such a short time frame. Most people do not use APT behind such portals, and we need to be sure to not break things for them.

With the current situation, you do not even have to actively use APT behind such a portal for the bug to be triggered.
When my main internet connection was working with such a portal, APT just stopped working seemingly randomly because there apparently is some kind of auto fetch daemon running by default on Ubuntu. I had to delete the contents of that directory very regularly, because APT often wasn't working when I needed it. Of course I would only use APT myself once I verified that my connection was working properly.

And I'm sad to say it, but it seems such portals are very popular in this part of the world :(

If this is not going to be fixed for Trusty, at least add a note to the release notes which explains how to disable this automatic fetching of package lists. That would significantly reduce the problem for many people.

The problem is that most people don't use apt at all. Their computer goes
out to see if there are updates and breaks silently. It happens without
user knowledge or intervention.
On Mar 23, 2014 6:01 AM, "Monsta" <email address hidden> wrote:

> Would be really nice if you'd fixed it before Ubuntu 14.04 LTS is
> finally released...
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1055614).
> https://bugs.launchpad.net/bugs/756317
>
> Title:
> Captive portals may corrupt apt translation indices
>
> Status in "apt" package in Ubuntu:
> Confirmed
> Status in "apt" package in Debian:
> New
>
> Bug description:
> I have an adsl modem which returns an html page if the adsl link is
> broken. This page ends as the content of the apt cache files stored in
> /var/lib/apt/lists, which breaks apt.
>
> The only way to make apt work again is to delete all the files stored
> in /var/lib/apt/lists.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/apt/+bug/756317/+subscriptions
>

I pushed a package for trusty with this fixed to

ppa:deity/sid

(currently building)

This is almost completely untested, it just re-enables verification for all indices.

You can try it out and report back if this produces any error.

(The version number is 0.9.15.4ubuntu2+jak1)

You mean this PPA?
https://launchpad.net/~deity/+archive/ubuntu-tests

Exactly.

How to test it without an access to a captive portal or too clever adsl modem?

If I corrupt any of Translation files manually, APT still behaves like before (i.e. ceases to function). Should I do some other test?

Just checking that normal use without a portal does not break also helps.

The normal use does not break, apt-get update && apt-get dist-upgrade, and also some usage of apt-cache - all went successfully.

But shouldn't the reading of the corrupted files be fixed somehow?

No. Corrupted files in /var/lib/apt/lists will still produce errors. APT downloads files to a "partial" subdirectory, and moves them to the lists directory if they verify correctly (contain a Package: field). Those, corrupt files cannot end up in the lists directory.

Tim Hortons' captive portals break apt. Do they have Tim Horton's coffee
shop in your neck of the woods?

Bryan Harris, PE
Research Engineer
Structures and Materials Evaluation Group
University of Dayton Research Institute
<email address hidden>
http://www.udri.udayton.edu/
(937) 229-5561

On Tue, Mar 25, 2014 at 1:53 PM, Julian Andres Klode <email address hidden>wrote:

> No. Corrupted files in /var/lib/apt/lists will still produce errors. APT
> downloads files to a "partial" subdirectory, and moves them to the lists
> directory if they verify correctly (contain a Package: field). Those,
> corrupt files cannot end up in the lists directory.
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1055614).
> https://bugs.launchpad.net/bugs/756317
>
> Title:
> Captive portals may corrupt apt translation indices
>
> Status in "apt" package in Ubuntu:
> In Progress
> Status in "apt" package in Debian:
> New
>
> Bug description:
> I have an adsl modem which returns an html page if the adsl link is
> broken. This page ends as the content of the apt cache files stored in
> /var/lib/apt/lists, which breaks apt.
>
> The only way to make apt work again is to delete all the files stored
> in /var/lib/apt/lists.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/apt/+bug/756317/+subscriptions
>

Are you sure it's alright to not have any protection from the corrupted files in /var/lib/apt/lists directory?

You may have fixed this particular bug, but there may be others related to the corruption of the lists.

So when will this fix get into Trusty?

This bug was fixed in the package apt - 1.0.4ubuntu5

---------------
apt (1.0.4ubuntu5) utopic; urgency=medium

  [ Michael Vogt ]
  * Try not to parse invalid translation files (LP: #756317)

  [ Iain Lane ]
  * Remove stray *.debhelper files.
 -- Iain Lane <email address hidden> Fri, 04 Jul 2014 10:15:28 +0100

Ok great, and when will the fix be backported into Trusty?

I open a possible duplicate bug #1343205, and it was marked as a duplicate by Monsta, but...

The symptoms of this in my case are different.

1. It's always the same file that is corrupted. /var/lib/apt/lists/za.archive.ubuntu.com_ubuntu_dists_trusty-updates_main_i18n_Translation-en
2. This happens on multiple machines running 14.04 and earlier on different internet connections.
3. I have actually run tests and the link does not drop to cause this, it happens while the internet connection is stable.

Is this the same issue or not?

> It's always the same file that is corrupted. /var/lib/apt/lists/za.archive.ubuntu.com_ubuntu_dists_trusty-updates_main_i18n_Translation-en

That's enough to mark it as a duplicate. APT was vulnerable precisely to the corrupted translation files.
If you don't believe (and have enough time), look at the comments above.

Seeing a bunch of reports about Ubiquity crashing and showing the same MergeList error in translation files, I insist that the fix for this issue *must* be backported to Trusty.

Come on guys, Trusty users still encounter the issue.

I believe I have just experienced this bug in xenial with apt 1.2.32.
Files in /var/lib/apt/lists contained HTML from a captive portal, breaking apt until they were removed.

Unfortunately, I cannot reproduce the bug by deliberately running apt update while the captive portal is in effect.

I think I must have hit some obscure edge case that isn't covered by the existing fix. Maybe something to do with switching networks during an automatic update? Who knows. Clearly I haven't got enough information to resolve this.

Unfortunately I did not retain a copy of the files I removed, so they cannot be analysed, though I have saved a copy of the current captive portal page if anyone wants me to upload it. (It might differ from the one that triggered the bug for me though.)

Apologies for double comment.