Ubuntu Archive translations are missing Index metadata file

--- push --

Status changed to 'Confirmed' because the bug affects multiple users.

Opening a task as requested on the Launchpad project.

There are sort of two separate problems here. One is that the Translations-* files generated by custom DDTP uploads aren't indexed. The other is that the Translations-en files generated by apt-ftparchive by request of Launchpad for Ubuntu Oneiric onward aren't indexed. The latter is what's caused a regression recently, because at least English package descriptions used to be available from the output of debmirror, but they aren't any more for Oneiric.

The reason I think we need to consider these separately is that I rather suspect that Launchpad doesn't really know internally about custom uploads at the point when it would need to generate these Index files; they get stuffed onto its filesystem when the custom uploads are first published, but thereafter it more or less forgets about them. The only files that normally go into Launchpad-generated indexes are those that are actually recorded in the Launchpad database.

So, I propose that we fix this in two stages, of which only the first is urgent (because it's fixing a regression). First, generate i18n/Index files for distroseries where include_long_descriptions is True, and include them in Release; these index files should initially only refer to Translations-en.bz2. Secondly, once the files from DDTP custom uploads are recorded somehow in the database rather than merely on the filesystem, they can be added to i18n/Index.

Incidentally, the i18n/Index files are a bit odd. Debian only generates them with a SHA1 section, and only includes the .bz2 versions. debmirror's parser doesn't look desperately robust against getting more sections, so I suggest that at least for now we should simply mimic Debian's behaviour here.

In fact, on reflection, generating the Index file by inspecting files on the filesystem really isn't much different from generating the Release file by inspecting files on the filesystem, and we already do the latter; so maybe we should just do the whole lot in one go.

I've filed http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=644609 for the robustness problems in debmirror.

Fixed in stable r14123 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/14123>.

This wasn't a debmirror bug in the first place (although I understand why you filed it there), so I'll close that part.

The Launchpad change has been deployed and seems to be working properly. We aren't going to republish Natty and earlier now, I'm afraid, but (as of the next mirror push) Oneiric and beyond will have i18n/Index files that can be used by debmirror.

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/572128

This actually seems to happen again with 16.04 xenial mirrors. I know it's still beta, but we are testing already!

will there be a patch before Final Release?

Patrick, please could you urgently provide details of exactly what you're seeing? If you're referring to output from apt, please use the -oDebug::Acquire::http=true option.

I'm sorry I interrupted you. Seems to be a local mirror problem. debmirror (1:2.16ubuntu1) doesn't correctly download these files. So if there is a bug, it will be a separate bug which has nothing to do with this one.

We're debugging the problem at the moment. So I can't tell what's the problem, yet.

You may want to try debmirror 1:2.16ubuntu1.1 in trusty-proposed, then.
See bug 1565585.