neutron-linuxbridge-agent fails to start with iptables 1.8.5

Bug #1898547 reported by Albert Damen
24
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Fix Released
Undecided
Skipper Bug Screeners
iptables (Ubuntu)
Fix Released
High
Alex Murray
Groovy
Fix Released
High
Alex Murray
Hirsute
Fix Released
High
Alex Murray
neutron (Ubuntu)
Invalid
Undecided
Unassigned
Groovy
Invalid
Undecided
Unassigned
Hirsute
Invalid
Undecided
Unassigned

Bug Description

[Impact]

With iptables 1.8.5 neutron-linuxbridge-agent fails to properly start.

The log file shows many errors like:

2020-10-05 10:20:37.998 551 ERROR neutron.plugins.ml2.drivers.agent._common_agent ; Stdout: ; Stderr: iptables-restore: line 29 failed

This can be demonstrated with a simple test case:

iptables-restore <<EOF
*filter
:INPUT - [0:0]
COMMIT
EOF

This fails with iptables 1.8.5 and is a known upstream bug that was subsequently fixed in upstream commit https://git.netfilter.org/iptables/commit/?id=0bd7a8eaf3582159490ab355b1217a4e42ed021f

As such, neutron-linuxbridge-agent is not able to be used successfully on groovy. This fix to iptables is required to allow neutron-linuxbridge-agent to successfully run.

In hirsute, iptables 1.8.5-3ubuntu3 has been uploaded which fixes this bug by backporting the upstream fix from commit 0bd7a8eaf3582159490ab355b1217a4e42ed021f above. This is currently sitting in hirsute-proposed waiting for autopkgtests to complete to finish migration.

For groovy, iptables 1.8.5-3ubuntu2.20.10.1 is sitting in Unapproved and is the subject of this SRU (this is simply 1.8.5-3ubuntu3 packaged for groovy)

[Test Case]

This can be reproduced by the test case.

[Regression Potential]

 * This is a low risk update since it only affects the behaviour when a policy of '-' is specified and so does not affect any users of iptables that specify an explicit policy (like ACCEPT, REJECT etc). Since this '-' behaviour is currently broken it has a very low chance of causing a regression as it does not affect any code paths the use an explicit policy. One possible regression would be if any users of iptables-restore
were relying on this failing behaviour, but since this has only failed for
groovy and no other Ubuntu releases this is highly unlikely. The other
possibility is that the patch introduces some other failure, however
as stated above, close analysis of the patch shows it only introduces
new behaviour when the policy is specified as '-' - so this should be
impossible.

 * In the event of a regression, iptables can be reverted back to a rebuild of 1.8.5-3ubuntu1 by simply backing out this patch.

[Other Info]

 * Details regarding an explicit test verification of neutron-linuxbridge-agent will be added soon.

Revision history for this message
Albert Damen (albrt) wrote :
Albert Damen (albrt)
Changed in neutron (Ubuntu):
status: New → Invalid
Revision history for this message
Albert Damen (albrt) wrote :

This issue was fixed in iptables git master commit dac904bdcd9a18aabafee7275ccf0c2bd53800f3

I guess the actual fix may have been "iptables-nft: fix basechain policy configuration", commit 0bd7a8eaf3582159490ab355b1217a4e42ed021f

Revision history for this message
Albert Damen (albrt) wrote :

I could reproduce the issue by building git v1.8.5 and the issue was fixed after cherry-picking "iptables-nft: fix basechain policy configuration"

$ git log
commit 8d985eb4eb7a23fd98b75d71179af40169144cc5 (HEAD -> bug1898547)
Author: Pablo Neira Ayuso <email address hidden>
Date: Fri Oct 2 13:44:36 2020 +0200

    iptables-nft: fix basechain policy configuration

    Previous to this patch, the basechain policy could not be properly
    configured if it wasn't explictly set when loading the ruleset, leading
    to iptables-nft-restore (and ip6tables-nft-restore) trying to send an
    invalid ruleset to the kernel.

    Signed-off-by: Arturo Borrero Gonzalez <email address hidden>
    Signed-off-by: Pablo Neira Ayuso <email address hidden>

commit 14ac250946289e280fb09ef978a45042871275b0 (tag: v1.8.5)
Author: Pablo Neira Ayuso <email address hidden>
Date: Wed Jun 3 11:37:52 2020 +0200

    configure: bump version for 1.8.5 release

    Signed-off-by: Pablo Neira Ayuso <email address hidden>

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in iptables (Ubuntu):
status: New → Confirmed
Revision history for this message
James Page (james-page) wrote :
Changed in neutron (Ubuntu Groovy):
status: New → Invalid
James Page (james-page)
Changed in iptables (Ubuntu Hirsute):
importance: Undecided → High
Changed in iptables (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in iptables (Ubuntu Hirsute):
status: Confirmed → Triaged
Alex Murray (alexmurray)
Changed in iptables (Ubuntu Groovy):
assignee: nobody → Alex Murray (alexmurray)
Changed in iptables (Ubuntu Hirsute):
assignee: nobody → Alex Murray (alexmurray)
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

FYI, 1.8.5-3ubuntu3 was uploaded to hirsute-proposed yesterday. 1.8.5-3ubuntu2.20.10.1 is in the unapproved queue for groovy-proposed. Alex said he'd do the SRU paperwork.

Changed in iptables (Ubuntu Hirsute):
status: Triaged → Fix Committed
Alex Murray (alexmurray)
Changed in iptables (Ubuntu Groovy):
status: Triaged → In Progress
Alex Murray (alexmurray)
description: updated
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Albert, or anyone else affected,

Accepted iptables into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/iptables/1.8.5-3ubuntu2.20.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in iptables (Ubuntu Groovy):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-groovy
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: New → Fix Committed
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (iptables/1.8.5-3ubuntu2.20.10.1)

All autopkgtests for the newly accepted iptables (1.8.5-3ubuntu2.20.10.1) for groovy have finished running.
The following regressions have been reported in tests triggered by the package:

sshuttle/1.0.4-1ubuntu4 (arm64)
firewalld/0.9.1-1ubuntu1 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/groovy/update_excuses.html#iptables

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Alex Murray (alexmurray)
description: updated
Revision history for this message
Alex Murray (alexmurray) wrote :

FYI the two autopkgtest failures for arm64 (sshuttle & firewalld) both appear to be transient failures so these are currently being retried...

Revision history for this message
Albert Damen (albrt) wrote :

I have verified the fixed package, see attached terminal output.

Steps taken:
- check package version
- verify nf_tables is used
- check default chains have not been created yet
- run test case
- check if default chain has been created

After that I upgraded the iptables packages on my neutron and compute hosts and rebooted them. Without any manual intervention the linuxbridge-agent was started and I could start a new instance which entered the "Running" state and had network connectivity.
The linuxbridge-agent logs did not contain errors regarding iptables after the reboot.

tags: added: verification-done-groovy
removed: verification-needed-groovy
Revision history for this message
Andrew McLeod (admcleod) wrote :

I've tested this (s390x, groovy) and im able to launch an instance with the specific version of iptables mentioned above

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package iptables - 1.8.5-3ubuntu3

---------------
iptables (1.8.5-3ubuntu3) hirsute; urgency=medium

  * Fix regression when configuring basechain policy (LP: #1898547)
    - d/p/9003-iptables_nft_fix_basechain_policy_configuration.patch: Backport
      patch from upstream to fix basechain policy configuration when it
      wasn't explicitly set.

 -- Alex Murray <email address hidden> Tue, 03 Nov 2020 11:57:59 +1030

Changed in iptables (Ubuntu Hirsute):
status: Fix Committed → Fix Released
Alex Murray (alexmurray)
tags: removed: verification-needed
Revision history for this message
Alex Murray (alexmurray) wrote :

jdstrand sponsored this to groovy-proposed and autopkgtests have all passed - ~ubuntu-sru - could you please review?

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package iptables - 1.8.5-3ubuntu2.20.10.1

---------------
iptables (1.8.5-3ubuntu2.20.10.1) groovy; urgency=medium

  * Fix regression when configuring basechain policy (LP: #1898547)
    - d/p/9003-iptables_nft_fix_basechain_policy_configuration.patch: Backport
      patch from upstream to fix basechain policy configuration when it
      wasn't explicitly set.

 -- Alex Murray <email address hidden> Tue, 03 Nov 2020 11:57:59 +1030

Changed in iptables (Ubuntu Groovy):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for iptables has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Changed in ubuntu-z-systems:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.