magic-proxy broke with iptables 1.8.7-1ubuntu2

I tried to reproduce this in an up-to-date bionic VM as follows:

# inside the bionic VM
sudo snap install lxd
sudo lxd init # accept defauls
sudo lxc launch ubuntu-daily:hirsute hirsute
sudo lxc exec hirsute /bin/bash

# then inside the hirsute container install livecd-rootfs
apt update
apt install livecd-rootfs

# http works as expected with no changes
wget -q www.google.com -O/dev/null && echo Working || echo Failed
Working # works as expected with no iptables rule

# add iptables rule manually
iptables -t nat -A OUTPUT -p tcp --dport 80 -m owner ! --uid-owner daemon \
           -j REDIRECT --to 8080

# now we expect it to fail as there is no magic-proxy running yet
wget -q www.google.com -O/dev/null && echo Working || echo Failed
Failed

# start the magic-proxy manually
/usr/share/livecd-rootfs/magic-proxy \
       --address="127.0.0.1" \
       --port=8080 \
       --run-as=daemon \
       --cutoff-time=0 \
       --log-file=livecd.magic-proxy.log \
       --pid-file=magic-proxy.pid \
       --background \
       --setsid

# wget works as expected via the proxy
wget -q www.google.com -O/dev/null && echo Working || echo Failed
Working

# kill the proxy
killall magic-proxy

# fails again
wget -q www.google.com -O/dev/null && echo Working || echo Failed
Failed

# remove iptables rule
iptables -t nat -D OUTPUT -p tcp --dport 80 -m owner ! --uid-owner daemon \
           -j REDIRECT --to 8080

# works as normal
wget -q www.google.com -O/dev/null && echo Working || echo Failed
Working

@alex

Thanks for trying this. And yes, we have been unable to reproduce this outside of launchpad.

However I do not think that google.com is a good test, as it will try to redirect to https, no? and magic proxy only does things with http. We are failing to reach http ftpmaster.internal.

I am now trying to rewrite bits of magic-proxy to use more of urllib and surface more HTTP and IO errors. Maybe it will tell us something more.

Good point re google.com - I just repeated the above test but replacing www.google.com with http://neverssl.com and verified it worked as expected so it doesn't look like http->https redirect affected the results.

Hmmm perhaps there is something else at play compared to when testing locally vs on launchpad - with your original test-case, does using `iptables -L -t nat` behave any differently than `iptables -S -t nat` in terms of working around this? Perhaps there is something in the existing iptables setup on launchpad that is not present in our local testing which may be needed to reproduce this?

i think either -L or -S "made everything work".

Note that there is no iptables installs in the lxd container, and we install iptables on the fly.

nftables are not installed either, because that's not in main.

So when lxd container started, nothing did "restore" of any default chains I don't think. But also have no idea if any are needed, or if I can tune any debugging in the kernel to notice if iptables are poked in some odd ways inside the network namespace that we are in.

Also note all the extra setup that launchpad-buildd does:
https://git.launchpad.net/launchpad-buildd/tree/lpbuildd/target/lxd.py

in terms of creating lxd profile, and doing stuff to the bridge with iptables.

I wonder if I must always use legacy iptables, if that's what used on the host to configure the bridge of the lxd container one is in.

We do some slightly fiddly stuff in launchpad-buildd to set up a bridge - see the `iptables` and `start_bridge` methods in https://git.launchpad.net/launchpad-buildd/tree/lpbuildd/target/lxd.py#n233. Might this be relevant?

The nat fiddles are not visible inside the container network namespace. Thus I am wondering if there is an odd interaction between namespace, nftables based iptables vs legacy iptables. I.e. whilst the host is configured using legacy iptables, maybe the lxd guests must be using legacy iptables too.

I'll experiment to see if forcing to simply only use iptables-legacy inside the lxd guest is good enough for now. Despite the hosts getting upgraded to bionic. Cause it's only groovy that started to use nftables based iptables.

This bug was fixed in the package livecd-rootfs - 2.721

---------------
livecd-rootfs (2.721) impish; urgency=medium

  [ Dimitri John Ledkov ]
  * 999-cpc-fixes: enable more code on grub2 armhf & arm64 (LP: #1925780)
  * Add support for generic preinstalled images. LP: #1923832
  * Change iptables calls, to query rules and quickly check that
    connectivity works after transparent proxy has been installed. LP:
    #1917920
  * magic-proxy: replace http.client with urllib calls
  * buildd: Re-enable merged /usr by default, as it is now required

  [ Michael Hudson-Doyle ]
  * remove the workaround for bug #1893818

  [ Łukasz 'sil2100' Zemczak ]
  * Start using the ubuntu-server-raspi seeds for preinstalled raspi server
    images, similarly to how it's done for desktop-preinstalled images.
  * Drop the unused imx6 support.

 -- Dimitri John Ledkov <email address hidden> Wed, 05 May 2021 19:08:21 +0100

The livecd-rootfs SRU for Bionic, Focal and Hirsute is currently blocked by another SRU in progress.

Hello Dimitri, or anyone else affected,

Accepted livecd-rootfs into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.664.32 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted livecd-rootfs (2.664.32) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-image/1.11+20.04ubuntu1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#livecd-rootfs

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Bionic is not affected

Hello Dimitri, or anyone else affected,

Accepted livecd-rootfs into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.664.33 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted livecd-rootfs (2.664.33) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-image/1.11+20.04ubuntu1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#livecd-rootfs

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

I did a testbuild with repo-stamp enabled against the livecd-rootfs version from proposed (2.664.33). See
- https://launchpadlibrarian.net/566970666/buildlog_ubuntu_focal_amd64_amd64-tarball_proposed_BUILDING.txt.gz
- https://launchpad.net/~toabctl/+livefs/ubuntu/focal/proposed/+build/307101

That looks good.

Hello Dimitri, or anyone else affected,

Accepted livecd-rootfs into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.719.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package livecd-rootfs - 2.664.33

---------------
livecd-rootfs (2.664.33) focal; urgency=medium

  * Install cloud-initramfs-growroot to actually enable rootfs resize.
  * Fix a grub error by making sure the unicode.pf2 font is installed in the
    right path for preinstalled amd64 desktop images.

livecd-rootfs (2.664.32) focal; urgency=medium

  * 099-ubuntu-image-customization.chroot: fix a typo in it.

livecd-rootfs (2.664.31) focal; urgency=medium

  [ Łukasz 'sil2100' Zemczak ]
  * Add the 099-ubuntu-image-customization.chroot for
    desktop-preinstalled images similar to what we have in groovy+ (for the pi
    desktop), but improved for amd64 platforms. We need it to generate a valid
    grub.cfg on the rootfs (similar to ubuntu-cpc) and then use that instead
    of a static configuration locked on the boot partition (LP: #1949102).

  [ Brian Murray ]
  * Properly check ARCH when setting the intel-iot model.

livecd-rootfs (2.664.30) focal; urgency=medium

  [ Thomas Bechtold ]
  * magic-proxy: Replace http.client with urllib calls. live-build/auto/build:
    change iptables calls to query rules and quickly check that connectivity
    works after transparent proxy has been installed. (LP: #1917920)
  * magic-proxy: fix TypeError when trying to call get_uri() (LP: #1944906)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Fri, 29 Oct 2021 15:33:34 +0200

SRU verification for hirsute:

I did a testbuild with repo-stamp enabled against the livecd-rootfs version from proposed (2.719.3). See
- https://launchpad.net/~toabctl/+livefs/ubuntu/hirsute/proposed/+build/308173
- https://launchpadlibrarian.net/567803112/buildlog_ubuntu_hirsute_amd64_amd64-tarball_proposed_BUILDING.txt.gz

That looks good.

This bug was fixed in the package livecd-rootfs - 2.719.3

---------------
livecd-rootfs (2.719.3) hirsute; urgency=medium

  [ Thomas Bechtold ]
  * magic-proxy: Replace http.client with urllib calls. live-build/auto/build:
    change iptables calls to query rules and quickly check that connectivity
    works after transparent proxy has been installed. (LP: #1917920)
  * magic-proxy: fix TypeError when trying to call get_uri() (LP: #1944906)

 -- Brian Murray <email address hidden> Thu, 21 Oct 2021 11:17:58 -0700