please add cnf support to debmirror

Thanks for this Jamie. I just had the same problem with my local mirror here at home.

I had to make some small changes to your patch for it to work here. I guess you were using an un-split (archive vs. ports) mirror to test? The Release files advertise Commands-* for all architectures even though they aren't actually available to download, so we need to limit to the arches requested.

This change uses perl smartmatching, which is apparently still(!) experimental. Colin might not like that. :-)

Yes Iain, I don't mirror ports. Thanks for the updated patch. :)

Status changed to 'Confirmed' because the bug affects multiple users.

For what its worth this worked on my Ubuntu 18.04 with debmirror installed.

Thanks. I applied Jamie's patch, and took a slightly different approach for the problem that Iain reported:

  https://salsa.debian.org/debian/debmirror/commit/292e9e9c999dbfa9cd0269c5b7c8f82af09300fc
  https://salsa.debian.org/debian/debmirror/commit/39055366f9cd4f161e451cf0621d52f7dc0d5126
  https://salsa.debian.org/debian/debmirror/commit/39055366f9cd4f161e451cf0621d52f7dc0d5126

Status changed to 'Confirmed' because the bug affects multiple users.

Would be nice to have this into Ubuntu 20.04 so it can mirror itself...

Please change status back to Confirmed

I applied this patch and it still doesn't work. Still no cnf directories.

The patch is not correct.

In the function "sub cnf_from_release", there is a line which reads...

next if $filename =~ /^Commands-/;

This should actually read...

next if $filename =~ /^Commands-/ and $filename !~ /^Commands-$all_arches\./;

Please update the patch to take this into account.

Thanks

This may be patched upstream, but it is not present in the version which today installs on Focal 20.04 (reported as 1:2.27ubuntu1).

One reason might be that https://code.launchpad.net/~vcs-imports/debmirror/trunk (which is about this package, right?) reports that "Last successful import was on 2014-11-07. "

Possibly this is caused by the fact that the upstream repo has moved from git://git.debian.org/collab-maint/debmirror.git (listed on that page) to https://salsa.debian.org/debian/debmirror as shown in comment #6.

Another reason why Focal cannot mirror itself successfully is reported in this bug: https://bugs.launchpad.net/ubuntu/+source/debmirror/+bug/1879348

How do i patch the debmirror file please??

For completeness sake here, there is a merge request at https://bugs.launchpad.net/ubuntu/+source/debmirror/+bug/1892110 which rebases the Ubuntu changes on top of the latest upstream debmirror version, in turn including the required `command-not-found` changes.

This bug was fixed in the package debmirror - 1:2.33ubuntu1

---------------
debmirror (1:2.33ubuntu1) hirsute; urgency=medium

  * Resynchronise with Debian. (LP: #1892110) Remaining changes:
    - debian/{control,rules}: Add quilt for patch management.
    - Debian bug 455082:
      + default-settings.patch: change rsync defaults.
      + silence-errors.patch: throw away find errors.
      + drop-redundant-rsync.patch: clean up logic in rsync batching.
      + check_file-return.patch: report why a file needs to sync.
    - Debian bug 576576:
      + skip-installer.patch: allow specific releases to be skipped.
    - ubuntu-settings.patch: Changed defaults to download Ubuntu, replaced
      most instances of Debian with Ubuntu in the documentation.

debmirror (1:2.33) unstable; urgency=medium

  * Use debhelper-compat instead of debian/compat.
  * Strip trailing spaces and tabs from the content of clearsigned files
    before verifying the signature, in accordance with RFC 4880 section 7.1
    (closes: #943970).

debmirror (1:2.32) unstable; urgency=medium

  [ Jamie Strandboge ]
  * Support downloading command-not-found metadata (LP: #1821251).

debmirror (1:2.31) unstable; urgency=medium

  [ Ondřej Nový ]
  * d/copyright: Use https protocol in Format field
  * d/changelog: Remove trailing whitespaces

  [ Colin Watson ]
  * Fetch Packages-all and Contents-all if they exist (closes: #904927).
  * Fix mirroring of suites that have InRelease but not Release.gpg.
  * Improve GPG verification: accept a signature file as long as there's at
    least one good signature from a trusted key, and borrow code from APT to
    explicitly verify the structure of InRelease files (closes: #918304).

debmirror (1:2.30) unstable; urgency=medium

  * Make the error message when mkdir fails more helpful.
  * Restart partial HTTP downloads (closes: #629922).
  * Use /usr/share/dpkg/pkg-info.mk rather than equivalent hand-written
    code.
  * Upgrade to debhelper v9.

debmirror (1:2.29) unstable; urgency=medium

  [ Stefan Kisdaroczi ]
  * Add version string and --version option.
  * Add additional trace info (closes: #863194).

  [ Colin Watson ]
  * Automatically update $version at binary package build time.

debmirror (1:2.28) unstable; urgency=medium

  * Change priority to optional, since "Priority: extra" is now deprecated.
  * Move VCS to salsa.debian.org.
  * Fix use of uninitialised value $buf in name_release.
  * Depend on xz-utils (LP: #1614779); xz is needed for all suites that
    publish .xz but not uncompressed index files, or .gz but not .xz without
    --slow-cpu, so I think that's common enough to rise above the level of a
    Recommends.
  * Always use "exists $file_lists{foo}{size}" rather than just "exists
    $file_lists{foo}". The latter can return false positives due to
    autovivification (closes: #868675).

 -- Valters Jansons <email address hidden> Sat, 22 Aug 2020 15:18:46 +0300

please add the usual SRU details in the description, thanks

This was handled as a merge in #1892110 and is being handled as an SRU in #1906541 with the patch provided in that SRU ticket including further upstream commits.

> This was handled as a merge in #1892110 and is being handled as an SRU
> in #1906541 with the patch provided in that SRU ticket including
> further upstream commits.

The bug linked in the changelog of the SRU is this one, not bug #1906541. The SRU template needs to be on this bug (and, in general, we should not be opening separate bugs for an SRU).

> The bug linked in the changelog of the SRU is this one, not bug #1906541.

Indeed, I am not sure of the proper formatting rules, considering this was my first merge request followed by SRU request of upstream changes. The upstream changelog included the details of the original owner along with this LP# considering the fix originated here.

> The SRU template needs to be on this bug

How should this be handled as of now? By who?

> in general, we should not be opening separate bugs for an SRU

In this case, this bug ticket is marked for multiple packages. It seemed more straight-forward to me for the individual changes to be split off in a separate ticket for the SRU itself. Would, instead, multiple SRU templates be expected on this ticket - one for each package? Or would the SRU potentially change depending on the changes required in the individual packages?

This process is fairly scary from the outside perspective as these real life scenarios are not covered. Considering this bug has not been fixed for one-and-a-half years, with a patch laying around, I was trying to get the ball rolling. I do ask once again for actionable items as of now, or clarity of whether something should be done whatsoever (as it may instead be handled in other communication channels without information visible to the 'outside world').

Hey! Ok, so let me help out with the formalities here and then accept the SRU. I'll update this bug with the description of LP: #1906541, as this is the bug number that is referenced in the upload. I'll then mark the other one as a duplicate of this bug. We should be good then!

It does seem scary but it's not that much really! It's just that we need SRU's to link in the changelog to the bug with the appropriate SRU information. This is crucial as it's necessary for bugfix validation and, in case of regressions, knowing about what's going on. Since not everyone has the power to update bug descriptions, I can do it for you. Normally we'd just expect the package upload to reference LP: #1906541 instead.

Anyway, on it now.

I noticed that the version number is not perfect. Then again, I don't think it makes sense to re-upload just because of that. Let me accept it as is.

Hello Jamie, or anyone else affected,

Accepted debmirror into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/debmirror/1:2.27ubuntu2.20.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Jamie, or anyone else affected,

Accepted debmirror into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/debmirror/1:2.27ubuntu2.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

TEST CASE:
1. Installed current non-SRU package: apt install debmirror=1:2.27ubuntu1
2. Created a 20.04 Focal mirror: /usr/bin/debmirror -a amd64,i386 -d focal,focal-security,focal-updates,focal-backports -s main,universe,restricted,multiverse --nosource --passive --method=http -h archive.ubuntu.com -r ubuntu --keyring=/usr/share/keyrings/ubuntu-archive-keyring.gpg --progress /opt/debmirror
3. Exposed /opt/debmirror as webroot on test.example.com
4. On a separate node, configured APT to use the mirror, via sources.list such as:
   deb http://test.example.com/ubuntu/ focal main universe restricted multiverse
   deb http://test.example.com/ubuntu/ focal-security main universe restricted multiverse
   deb http://test.example.com/ubuntu/ focal-updates main universe restricted multiverse
5. Tried to use mirror on the newly configured client: apt update
   E: Failed to fetch http://test.example.com/ubuntu/dists/focal/main/cnf/Commands-amd64 404 Not Found [IP: 10.10.10.10 80]
   E: Failed to fetch http://test.example.com/ubuntu/dists/focal-security/main/cnf/Commands-amd64 404 Not Found [IP: 10.10.10.10 80]
   E: Failed to fetch http://test.example.com/ubuntu/dists/focal-updates/main/cnf/Commands-amd64 404 Not Found [IP: 10.10.10.10 80]
   E: Some index files failed to download. They have been ignored, or old ones used instead.
6. Installed SRU package: apt install debmirror=1:2.27ubuntu2.20.04.1
7. Repeated `debmirror` invocation from step 2, in order to pull in cnf files.
8. Repeated `apt update` invocation as from step 5, in order to try using the mirror.

VERIFICATION DONE
APT client succeeds, and mirrored repository can be successfully used.

TEST CASE:
1. Deploy fresh (up-to-date) Ubuntu 20.10 (Groovy) machine.
2. apt install -qqy debmirror=1:2.27ubuntu1
3. debmirror -a amd64 -d groovy -s main --nosource --passive --method=http -h archive.ubuntu.com -r ubuntu --keyring=/usr/share/keyrings/ubuntu-archive-keyring.gpg --progress /opt/debmirror
4. echo 'deb file:///opt/debmirror groovy main' > /etc/apt/sources.list.d/file.list
5. apt update # expected E: failed to fetch file:/opt/debmirror/dists/groovy/main/cnf/Commands-amd64 File not found
6. apt install -qqy debmirror=1:2.27ubuntu2.20.10.1
7. debmirror -a amd64 -d groovy -s main --nosource --passive --method=http -h archive.ubuntu.com -r ubuntu --keyring=/usr/share/keyrings/ubuntu-archive-keyring.gpg --progress /opt/debmirror
8. echo 'deb file:///opt/debmirror groovy main' > /etc/apt/sources.list.d/file.list
9. apt update # All packages are up to date.

VERIFICATION DONE
Mirror is usable with updated `debmirror` package.

This bug was fixed in the package debmirror - 1:2.27ubuntu2.20.10.1

---------------
debmirror (1:2.27ubuntu2.20.10.1) groovy; urgency=medium

  * Cherry-pick upstream command-not-found changes (version 1:2.32).

  [ Jamie Strandboge ]
  * Support downloading command-not-found metadata (LP: #1821251).

 -- Valters Jansons <email address hidden> Wed, 02 Dec 2020 18:36:09 +0200

The verification of the Stable Release Update for debmirror has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package debmirror - 1:2.27ubuntu2.20.04.1

---------------
debmirror (1:2.27ubuntu2.20.04.1) focal; urgency=medium

  * Cherry-pick upstream command-not-found changes (version 1:2.32).

  [ Jamie Strandboge ]
  * Support downloading command-not-found metadata (LP: #1821251).

 -- Valters Jansons <email address hidden> Wed, 02 Dec 2020 18:36:09 +0200