dapper indices files still being regenerated but shouldn't be

Bug #48948 reported by James Troup
10
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
High
Julian Edwards

Bug Description

The dapper indices files (Packages, Sources) are still being regenerated. This is really bad because:

 a) it means the millions of users now running Dapper are seeing "new" Packages files every hour and redownloading them
 b) it means the guarantees for a 'frozen' and 'released' distribution aren't guarantees at all.

Please fix. Seriously.

James Troup (elmo)
Changed in launchpad-publisher:
importance: Untriaged → Critical
Revision history for this message
Daniel Silverstone (dsilvers) wrote :

It's not in the apt.conf we pass to apt-ftparchive so if it's being regenerated I don't see where it can be happening.

Revision history for this message
Daniel Silverstone (dsilvers) wrote :

Sorry, I should have added to that "But I do see it is happening"

I'm still chasing it down

Changed in launchpad-publisher:
assignee: nobody → dsilvers
status: Unconfirmed → Confirmed
Revision history for this message
Daniel Silverstone (dsilvers) wrote :

Because the Packages files are identical to Edgy's the dsync deduping is deduping them and unfortunately picking the edgy one as the primary and thus making dapper's match edgy's

The symptom will go away when we have updated packages in edgy. Unfortunately the bug is deeper than that

Changed in launchpad-publisher:
assignee: dsilvers → nobody
Revision history for this message
Celso Providelo (cprov) wrote :

Malcc, have you done something in this land in you p-d-o branch ?

Changed in launchpad-publisher:
assignee: nobody → malcolmcleaton
Revision history for this message
Christian Reis (kiko) wrote :

We might be better off not running dsync on the dists/ tree.

Revision history for this message
Malcolm Cleaton (malcolmcleaton) wrote :

No, I haven't touched this problem.

The only solution I can see, assuming Daniel's analysis is correct, is to not run dsync in the dists tree.

Dsync appears to assume that it's ok to make two files with identical contents hardlinks to the same underlying file, but this can't be done without changing timestamps; if timestamps can't be changed in the dists tree, running dsync on the dists tree is not safe.

As far as we know, we'll have this problem again if we were to open edgy+1 with today's Soyuz.

Revision history for this message
Malcolm Cleaton (malcolmcleaton) wrote :

Discussed this with James, who pointed out that not running dsync on the dists tree would be bad, because it also contains all the installer stuff, which needs de-duplicating.

The alternative suggestion is, modify dsync to steer clear of Sources, Packages and Release files.

Revision history for this message
Malcolm Cleaton (malcolmcleaton) wrote :

A look was taken, and dsync may already do this:

<cjwatson> malcc: I think you can just pass extra options to dsync-flist generate: -e 'Packages*' -e 'Sources*' -e 'Release*'
<malcc> cjwatson: Cool!
<cjwatson> just check that that doesn't end up deleting files or whatever, but I don't think it will
<cjwatson> if it does, dsync-flist has other filter options - let me know and I'll look into them for you

I need to do some testing.

Revision history for this message
Malcolm Cleaton (malcolmcleaton) wrote :

We've worked around this for the feisty release by disabling dsync link-dups; we'll re-enable it once feisty diverges from edgy.

Revision history for this message
Malcolm Cleaton (malcolmcleaton) wrote :

Reduced in priority to high because of existence of workaround.

Changed in soyuz:
importance: Critical → High
Changed in soyuz:
assignee: malcolmcleaton → nobody
Revision history for this message
Celso Providelo (cprov) wrote :

James,

Does dsync still disabled as mentioned ?

Have we implemented anything towards improving the dsync filtering options to still catching dups on installer but not in archive indexes ?

Joey Stanford (joey)
Changed in soyuz:
assignee: nobody → julian-edwards
Revision history for this message
Christian Reis (kiko) wrote :

Pushing off so we survive 1.1.7.

Celso Providelo (cprov)
Changed in soyuz:
assignee: julian-edwards → cprov
Changed in soyuz:
assignee: cprov → julian-edwards
status: Confirmed → In Progress
Revision history for this message
Julian Edwards (julian-edwards) wrote :

Fix in RF 4885.

Changed in soyuz:
status: In Progress → Fix Committed
Changed in soyuz:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.