networkd: Don't reset forwarding unless told to do so in config

Bug #1500992 reported by Stéphane Graber
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
systemd (Ubuntu)
Fix Released
Wishlist
Martin Pitt

Bug Description

It's been reported by several LXC users that systemd-networkd will turn off per-interface forwarding for all network interfaces when it starts.

Presumably upstream expects users to go and manually edit their config to allow it when needed.

This breaks LXC, libvirt, ... anything which ships a bridge that then NAT or route outgoing traffic. Requiring the user to do the config change would be a massive regression in user friendliness and having lxc, libvirt, ... do it for the user would be a policy violation.

As a result, I'd recommend we patch systemd to not interfere with forwarding unless explicitly configured by the user. This will allow all our existing scripts to keep setting things up themselves and have it all run fine.

Revision history for this message
Martin Pitt (pitti) wrote :

Can you please be more specific what exactly happens here?

$ cat /proc/sys/net/ipv4/conf/tun0/forwarding
1
$ sudo systemctl start systemd-networkd
$ cat /proc/sys/net/ipv4/conf/tun0/forwarding
1

Is that the setting you mean? How to reproduce the disabling of forwarding? Apparently it's more than just "start networkd" (I don't have any actual *.network config on this machine). Or do you mean "all interfaces that have a config for networkd" instead of "all interfaces"? It seems reasonable that networkd puts the devices it configures itself into a defined state. You wouldn't use it to manage tun0 for LXC, as LXC itself already sets that up?

Changed in systemd (Ubuntu):
status: New → Incomplete
Revision history for this message
Stéphane Graber (stgraber) wrote :

Oh, then maybe it's only on networkd managed interfaces.

The case I saw was on Mike MacCracken's machine where he had a basic dhcp config for eth0 which would then lead networkd to set /proc/sys/net/ipv4/conf/eth0/forwarding to off and IIRC same happened to /proc/sys/net/ipv6/conf/eth0/forwarding

So to clarify, I'd expect that unless I explictly turn off IPForward in networkd's config that it doesn't go and alter the interface state and instead leaves it at the default value set by the kernel.

This means that we can then set /proc/sys/net/ipv4/ip_forward to 1 and get the expected behavior (which is what most scripts, firewalls, ... do).

Revision history for this message
Stéphane Graber (stgraber) wrote :

To provide internet to VMs and containers, the interface behind which the default gateway sits, must have forwarding enabled.
The way this is typically done is that whichever tool sets up a bridge will also setup a NAT entry for IPv4 and will make sure /proc/sys/net/ipv4/ip_forward is set to 1.

This is the standard way of doing things because then you don't have to know what interface packets will be heading out of, forwarding is just enabled on all of them and if your NAT rule is properly written (and ours are), they'll apply to all interfaces too.

Now this all breaks if the outgoing interface is using a non-default config which disables forwarding at the interface level since that then ignores the global /proc/sys/net/ipv4/ip_forward knob.

Having the bridge setup tool, modify the configuration of all potential outgoing interface to allow forwarding seems completely wrong to me (why would the script setting up lxcbr0 go and edit your eth0 and wlan0 interfaces?), not to mention, it's not actually possible to know for sure what interfaces will be outgoing interfaces. You may have some that don't exist yet (usb stick?).

Even if we somehow did change the bridge setup scripts to go and set /forwarding to 1 on all interfaces which we guess may be used as outgoing interfaces, this still wouldn't change the fact that networkd would go ahead and reset them all to 0 on daemon restart (package update for example).

Revision history for this message
Martin Pitt (pitti) wrote :

Ah, thanks for confirming. This indeed applies to interfaces which networkd configures; man systemd.network documents this quite clearly.

> Having the bridge setup tool, modify the configuration of all potential outgoing interface to allow forwarding seems completely wrong to me

Full ack. Such scripts should not touch the configuration of interfaces that they didn't set up. This is unexpected, leads to race conditions, and isn't reliable as you wrote.

Setting to wishlist for now as it's currently behaving as documented. I'll raise this upstream and see if they are willing to change the default to "kernel".

Changed in systemd (Ubuntu):
status: Incomplete → Triaged
importance: Undecided → Wishlist
Revision history for this message
Martin Pitt (pitti) wrote :

Raising priority; on second thought this is actually interfering with configuration. Forwarded to https://github.com/systemd/systemd/issues/1411 , I'd like to change this upstream if at all possible.

Changed in systemd (Ubuntu):
importance: Wishlist → Medium
Revision history for this message
Martin Pitt (pitti) wrote :

Upstream reply/justification for current behaviour: https://github.com/systemd/systemd/issues/1411#issuecomment-144338862

Changed in systemd (Ubuntu):
importance: Medium → Wishlist
Revision history for this message
Martin Pitt (pitti) wrote : Re: Provide script to enable IP forwarding

Adjusting bug title after IRC discussion, see upstream issue summary. We need something like http://paste.ubuntu.com/12625613/ .

summary: - networkd: Don't reset forwarding unless told to do so in config
+ Provide script to enable IP forwarding
Martin Pitt (pitti)
Changed in systemd (Ubuntu):
status: Triaged → In Progress
assignee: nobody → Martin Pitt (pitti)
Revision history for this message
Martin Pitt (pitti) wrote :

It is much simpler to just set /proc/sys/net/ipv{4,6}/conf/all/forwarding to 1, as that updates all conf/iface/forwarding and /proc/sys/net/ipv4/ip_forward. I. e. we just need

for f in /proc/sys/net/ipv*/conf/all/forwarding; do
    echo 1 > $f
done

Revision history for this message
Martin Pitt (pitti) wrote :

This approach doesn't work after all, see
https://github.com/systemd/systemd/issues/1411#issuecomment-144975442

So changing back to the original title.

summary: - Provide script to enable IP forwarding
+ networkd: Don't reset forwarding unless told to do so in config
Revision history for this message
Martin Pitt (pitti) wrote :
Changed in systemd (Ubuntu):
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 225-1ubuntu7

---------------
systemd (225-1ubuntu7) wily; urgency=medium

  * Don't restart logind on upgrades any more. This kills X.org (#798097)
    while logind doesn't save/restore its open fds (issue #1163), and also
    gets confused about being idle in between (LP: #1473800)
  * debian/extra/initramfs-tools/hooks/udev: Copy all
    /etc/udev/rules.d/*.rules rules which are not merely overriding the one in
    /lib/, not just 70-persistent-net.rules. They might contain network names
    or other bits which are relevant for the initramfs. (Closes: #795494)
  * ifup@.service: Drop PartOf=network.target; we don't want to stop these
    units during shutdown. Stopping networking.service already shuts down the
    interfaces, but contains the safeguard for NFS or other network file
    systems. Isolating emergency.target still keeps working as before as well,
    as this also stops networking.service. (Closes: #761909, LP: #1492546)
  * networkd: Change IPForward= default to "kernel". This keeps compatibility
    with lots of packages which expect to be able to enable global forwarding
    in /proc/sys/net/ipv4/ip_forward. (LP: #1500992)

 -- Martin Pitt <email address hidden> Mon, 05 Oct 2015 15:53:26 +0200

Changed in systemd (Ubuntu):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.