System with DNS server in /etc/network/interfaces has bogus systemd-resolved config after upgrade to 22.04

Bug #1981103 reported by Uwe Schindler
50
This bug affects 9 people
Affects Status Importance Assigned to Milestone
ifupdown (Ubuntu)
Fix Released
Medium
Heitor Alves de Siqueira
Focal
Fix Released
Low
Heitor Alves de Siqueira
Jammy
Fix Released
Low
Heitor Alves de Siqueira
Lunar
Fix Released
Low
Heitor Alves de Siqueira

Bug Description

[ Impact ]
ifupdown configurations will stop having a working DNS server on upgrade

[ Test Plan ]
Upgrading from an Ubuntu 18.04 or earlier release, bounce a network interface with configured DNS server:

root@pangaea-pm:~# ifdown ens13; ifup ens13
/etc/network/if-down.d/resolved: 12: mystatedir: not found
/etc/network/if-up.d/resolved: 12: mystatedir: not found
/etc/network/if-up.d/resolved: 71: DNS: not found
/etc/network/if-up.d/resolved: 1: /run/network/ifupdown-inet-ens13: DNS=134.102.20.20 134.102.200.14: not found
/etc/network/if-up.d/resolved: 2: /run/network/ifupdown-inet-ens13: DOMAINS=marum.de: not found
Failed to parse DNS server address: DNS
Failed to set DNS configuration: Invalid argument

This should result in no errors, and in a valid network configuration with working DNS resolution.
It's likely that existing netplan configuration needs to be removed and migrated to classic ifupdown scripts before running the test above.

[ Where problems could occur ]
Although we're only fixing the resolved scripts, more complex network setups might see regressions in DNS resolution for some cases. We should pay special attention to bridges or virtual interfaces that have been configured through ifupdown, as well as validate the other scripts under /etc/network/if-up.d/

[ Other Info ]
Description: Ubuntu 22.04 LTS
Release: 22.04

ifupdown:
  Installed: 0.8.36+nmu1ubuntu3
  Candidate: 0.8.36+nmu1ubuntu3
  Version table:
 *** 0.8.36+nmu1ubuntu3 500
        500 http://de.archive.ubuntu.com/ubuntu jammy/universe amd64 Packages
        100 /var/lib/dpkg/status

After upgrading a server with classic ifupdown configuration after reboot the machine had no valid dns servers anymore.

The problem is that the state file created by ifupdown using /etc/network/if-up.d/resolved looks like this:

root@pangaea-pm:~# cat /run/network/ifupdown-inet-ens13
"DNS"="134.102.20.20 134.102.200.14"
"DOMAINS"="marum.de"

The script later sources this file and causes the following errors, easy to see when you execute this:

root@pangaea-pm:~# ifdown ens13; ifup ens13
/etc/network/if-down.d/resolved: 12: mystatedir: not found
/etc/network/if-up.d/resolved: 12: mystatedir: not found
/etc/network/if-up.d/resolved: 71: DNS: not found
/etc/network/if-up.d/resolved: 1: /run/network/ifupdown-inet-ens13: DNS=134.102.20.20 134.102.200.14: not found
/etc/network/if-up.d/resolved: 2: /run/network/ifupdown-inet-ens13: DOMAINS=marum.de: not found
Failed to parse DNS server address: DNS
Failed to set DNS configuration: Invalid argument

This happened to me on three different servers, so this is a serious bug and should be fixed before 22.04 upgrades are allowed for everybody. Most servers provided by data centers like Hetzner (Germany) are configure like that. After a do-release-upgrade you have no working DNS anymore, unless you disable systemd-resolved. I don't want to use netplan, so changing to this is no option.

The fix is easy - remove the quotes in the script on the left side "$DNS" => $DNS; same for DOMAINS:

    if [ -n "$NEW_DNS" ]; then
        cat <<EOF >"$mystatedir/ifupdown-${ADDRFAM}-$interface"
$DNS="$NEW_DNS"
EOF
        if [ -n "$NEW_DOMAINS" ]; then
            cat <<EOF >>"$mystatedir/ifupdown-${ADDRFAM}-$interface"
$DOMAINS="$NEW_DOMAINS"
EOF
        fi
    fi

Related branches

Revision history for this message
Uwe Schindler (uwe-thetaphi) wrote :

One addition, line 71 in script also looks strange:

    if ! cmp --silent "$oldstate" "$newstate" 2>/dev/null; then
        DNS DNS6 DOMAINS DOMAINS6 DEFAULT_ROUTE

This causes:
/etc/network/if-up.d/resolved: 71: DNS: not found

If you are on it, please fix this, too.

Revision history for this message
Uwe Schindler (uwe-thetaphi) wrote :

I think this should unset the variables, then error goes away and it makes sense!

    if ! cmp --silent "$oldstate" "$newstate" 2>/dev/null; then
        unset DNS DNS6 DOMAINS DOMAINS6 DEFAULT_ROUTE

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ifupdown (Ubuntu):
status: New → Confirmed
Revision history for this message
Chris Nicolson (morphiusfaydal) wrote :

This is affecting me too. It's only happening on systems that started life as Ubuntu 18.04 and earlier - any server that was installed as 20.04 is upgrading to 22.04 without problem. Once I make the same change as Uwe, systemd-resolved is getting configured correctly.

Revision history for this message
Neil (broomfn) wrote :

+1, can I simply remove ifupdown (e.g. apt remove ifupdown), do I still need it with 22.04.1?

Revision history for this message
Uwe Schindler (uwe-thetaphi) wrote :

If you changed away from /etc/network/interfaces file to netplan.io configuration (/etc/netplan/...) then you won't need ifupdown. There is a configuartion tool, with simple configs it works: See this info how to convert your network config. But be sure to have physical access to the machine's console: https://gist.github.com/mss/7a8e048dd51e5ef928039f1450ba8f31

I did this for my systems and removed and purged ifupdown. For some configs it may be very complex, especially if you have many tunnels, bridges or other special stuff in /etc/network/interfaces. But for simple single ethernet/wifi device it is easy to migrate using the above GIST.

Revision history for this message
Paul Wright (paulwright76) wrote :

The same error also happened in the ifdown script for resolved. Weird.

Revision history for this message
Paul Wright (paulwright76) wrote :

Deleted my incorrect comment about the use of "local", that doesn't work. https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1907878 has the correct fix.

Lukas Märdian (slyon)
tags: added: foundations-triage-discuss
Changed in ifupdown (Ubuntu):
status: Confirmed → Triaged
importance: Undecided → Medium
Revision history for this message
David Kowis (dkowis) wrote :

https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1910273

This one seems to have the full patch repair in the bug description. None of the other bugs I've found relating to this have the full solution.

Revision history for this message
Uwe Schindler (uwe-thetaphi) wrote : Re: [Bug 1981103] Re: System with DNS server in /etc/network/interfaces has bogus systemd-resolved config after upgrade to 22.04

*This* bug has the full information in the first post including all needed to fix the problem. Actually there are more problems in the script regarding syntax, so more issues exist, leading to unwanted behavior.

Uwe

Am 22. Oktober 2022 22:05:59 MESZ schrieb David Kowis <email address hidden>:
>https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1910273
>
>This one seems to have the full patch repair in the bug description.
>None of the other bugs I've found relating to this have the full
>solution.
>
>--
>You received this bug notification because you are subscribed to the bug
>report.
>https://bugs.launchpad.net/bugs/1981103
>
>Title:
> System with DNS server in /etc/network/interfaces has bogus systemd-
> resolved config after upgrade to 22.04
>
>Status in ifupdown package in Ubuntu:
> Triaged
>
>Bug description:
> Description: Ubuntu 22.04 LTS
> Release: 22.04
>
> ifupdown:
> Installed: 0.8.36+nmu1ubuntu3
> Candidate: 0.8.36+nmu1ubuntu3
> Version table:
> *** 0.8.36+nmu1ubuntu3 500
> 500 http://de.archive.ubuntu.com/ubuntu jammy/universe amd64 Packages
> 100 /var/lib/dpkg/status
>
> After upgrading a server with classic ifupdown configuration after
> reboot the machine had no valid dns servers anymore.
>
> The problem is that the state file created by ifupdown using
> /etc/network/if-up.d/resolved looks like this:
>
> root@pangaea-pm:~# cat /run/network/ifupdown-inet-ens13
> "DNS"="134.102.20.20 134.102.200.14"
> "DOMAINS"="marum.de"
>
> The script later sources this file and causes the following errors,
> easy to see when you execute this:
>
> root@pangaea-pm:~# ifdown ens13; ifup ens13
> /etc/network/if-down.d/resolved: 12: mystatedir: not found
> /etc/network/if-up.d/resolved: 12: mystatedir: not found
> /etc/network/if-up.d/resolved: 71: DNS: not found
> /etc/network/if-up.d/resolved: 1: /run/network/ifupdown-inet-ens13: DNS=134.102.20.20 134.102.200.14: not found
> /etc/network/if-up.d/resolved: 2: /run/network/ifupdown-inet-ens13: DOMAINS=marum.de: not found
> Failed to parse DNS server address: DNS
> Failed to set DNS configuration: Invalid argument
>
> This happened to me on three different servers, so this is a serious
> bug and should be fixed before 22.04 upgrades are allowed for
> everybody. Most servers provided by data centers like Hetzner
> (Germany) are configure like that. After a do-release-upgrade you have
> no working DNS anymore, unless you disable systemd-resolved. I don't
> want to use netplan, so changing to this is no option.
>
> The fix is easy - remove the quotes in the script on the left side
> "$DNS" => $DNS; same for DOMAINS:
>
> if [ -n "$NEW_DNS" ]; then
> cat <<EOF >"$mystatedir/ifupdown-${ADDRFAM}-$interface"
> $DNS="$NEW_DNS"
> EOF
> if [ -n "$NEW_DOMAINS" ]; then
> cat <<EOF >>"$mystatedir/ifupdown-${ADDRFAM}-$interface"
> $DOMAINS="$NEW_DOMAINS"
> EOF
> fi
> fi
>
>To manage notifications about this bug go to:
>https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1981103/+subscriptions
>

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de

Revision history for this message
Lukas Märdian (slyon) wrote (last edit ):

Thank you for describing the solution in this bug report.

Could somebody come up with a patch/debdiff, so that any MOTU could test & sponsor the ifupdown fix into universe?

tags: removed: foundations-triage-discuss
Changed in ifupdown (Ubuntu):
assignee: nobody → Heitor Alves de Siqueira (halves)
status: Triaged → In Progress
Revision history for this message
Heitor Alves de Siqueira (halves) wrote :
Revision history for this message
Heitor Alves de Siqueira (halves) wrote :

I've attached a debdiff with the patches for fixing this one. I've subscribed ubuntu-sponsors to kindly request sponsorship for mantic.

Thanks!

Revision history for this message
Benjamin Drung (bdrung) wrote :

Thanks. The 0.8.41 Debian package contains this fix already (so no forwarding needed). So uploaded to mantic (included some trailing space removals).

Changed in ifupdown (Ubuntu):
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ifupdown - 0.8.36+nmu1ubuntu5

---------------
ifupdown (0.8.36+nmu1ubuntu5) mantic; urgency=medium

  [ Uwe Kleine-König ]
  * Fix if-up.d/resolved hook to properly work with nameservers and search
    domains (LP: #1981103)

 -- Heitor Alves de Siqueira <email address hidden> Wed, 06 Sep 2023 13:23:34 +0000

Changed in ifupdown (Ubuntu):
status: Fix Committed → Fix Released
description: updated
Changed in ifupdown (Ubuntu Focal):
assignee: nobody → Heitor Alves de Siqueira (halves)
Changed in ifupdown (Ubuntu Jammy):
assignee: nobody → Heitor Alves de Siqueira (halves)
Changed in ifupdown (Ubuntu Lunar):
assignee: nobody → Heitor Alves de Siqueira (halves)
Changed in ifupdown (Ubuntu Focal):
status: New → In Progress
Changed in ifupdown (Ubuntu Jammy):
status: New → In Progress
Changed in ifupdown (Ubuntu Lunar):
status: New → In Progress
Changed in ifupdown (Ubuntu Focal):
importance: Undecided → Low
Changed in ifupdown (Ubuntu Lunar):
importance: Undecided → Critical
Changed in ifupdown (Ubuntu Jammy):
importance: Undecided → Low
Changed in ifupdown (Ubuntu Lunar):
importance: Critical → Low
Revision history for this message
Heitor Alves de Siqueira (halves) wrote :

Focal isn't affected by this, so marking it as Fix Released.

Changed in ifupdown (Ubuntu Focal):
status: In Progress → Fix Released
Revision history for this message
Timo Aaltonen (tjaalton) wrote : Please test proposed package

Hello Uwe, or anyone else affected,

Accepted ifupdown into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ifupdown/0.8.36+nmu1ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in ifupdown (Ubuntu Lunar):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-lunar
Changed in ifupdown (Ubuntu Jammy):
status: In Progress → Fix Committed
tags: added: verification-needed-jammy
Revision history for this message
Timo Aaltonen (tjaalton) wrote :

Hello Uwe, or anyone else affected,

Accepted ifupdown into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ifupdown/0.8.36+nmu1ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Heitor Alves de Siqueira (halves) wrote :

Validated for Jammy according to test case from description. No error messages were thrown, and networking has been correctly set up.

root@halves-lp1981103-j:~# ifdown enp5s0; ifup enp5s0
Killed old client process
Internet Systems Consortium DHCP Client 4.4.1
Copyright 2004-2018 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/enp5s0/00:16:3e:bb:4d:57
Sending on LPF/enp5s0/00:16:3e:bb:4d:57
Sending on Socket/fallback
DHCPRELEASE of 10.125.177.27 on enp5s0 to 10.125.177.1 port 67 (xid=0x4969e444)
Internet Systems Consortium DHCP Client 4.4.1
Copyright 2004-2018 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/enp5s0/00:16:3e:bb:4d:57
Sending on LPF/enp5s0/00:16:3e:bb:4d:57
Sending on Socket/fallback
DHCPDISCOVER on enp5s0 to 255.255.255.255 port 67 interval 3 (xid=0x787e8804)
DHCPOFFER of 10.125.177.27 from 10.125.177.1
DHCPREQUEST for 10.125.177.27 on enp5s0 to 255.255.255.255 port 67 (xid=0x4887e78)
DHCPACK of 10.125.177.27 from 10.125.177.1 (xid=0x787e8804)
bound to 10.125.177.27 -- renewal in 1648 seconds.

root@halves-lp1981103-j:~# dpkg -l | grep ifupdown
ii ifupdown 0.8.36+nmu1ubuntu3.1 amd64 high level tools to configure network interfaces

tags: added: verification-done-jammy
removed: verification-needed-jammy
Revision history for this message
Heitor Alves de Siqueira (halves) wrote :

Validated for Lunar according to test case from description. No error messages were thrown, and networking has been correctly set up.

root@halves-lp1981103-l:~# ifdown enp5s0; ifup enp5s0
Killed old client process
Internet Systems Consortium DHCP Client 4.4.3-P1
Copyright 2004-2022 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/enp5s0/00:16:3e:c7:2c:be
Sending on LPF/enp5s0/00:16:3e:c7:2c:be
Sending on Socket/fallback
xid: warning: no netdev with useable HWADDR found for seed's uniqueness enforcement
xid: rand init seed (0x65355010) built using gethostid
DHCPRELEASE of 10.125.177.42 on enp5s0 to 10.125.177.1 port 67 (xid=0x1adc66da)
Internet Systems Consortium DHCP Client 4.4.3-P1
Copyright 2004-2022 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/enp5s0/00:16:3e:c7:2c:be
Sending on LPF/enp5s0/00:16:3e:c7:2c:be
Sending on Socket/fallback
xid: warning: no netdev with useable HWADDR found for seed's uniqueness enforcement
xid: rand init seed (0x65355010) built using gethostid
DHCPDISCOVER on enp5s0 to 255.255.255.255 port 67 interval 3 (xid=0xda66dc1a)
DHCPOFFER of 10.125.177.42 from 10.125.177.1
DHCPREQUEST for 10.125.177.42 on enp5s0 to 255.255.255.255 port 67 (xid=0x1adc66da)
DHCPACK of 10.125.177.42 from 10.125.177.1 (xid=0xda66dc1a)
bound to 10.125.177.42 -- renewal in 1683 seconds.

root@halves-lp1981103-l:~# dpkg -l | grep ifupdown
ii ifupdown 0.8.36+nmu1ubuntu4.1 amd64 high level tools to configure network interfaces

tags: added: verification-done verification-done-lunar
removed: verification-needed verification-needed-lunar
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ifupdown - 0.8.36+nmu1ubuntu4.1

---------------
ifupdown (0.8.36+nmu1ubuntu4.1) lunar; urgency=medium

  [ Uwe Kleine-König ]
  * Fix if-up.d/resolved hook to properly work with nameservers and search
    domains (LP: #1981103)

 -- Heitor Alves de Siqueira <email address hidden> Mon, 16 Oct 2023 14:34:24 +0000

Changed in ifupdown (Ubuntu Lunar):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for ifupdown has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ifupdown - 0.8.36+nmu1ubuntu3.1

---------------
ifupdown (0.8.36+nmu1ubuntu3.1) jammy; urgency=medium

  [ Uwe Kleine-König ]
  * Fix if-up.d/resolved hook to properly work with nameservers and search
    domains (LP: #1981103)

  [ Heinrich Schuchardt ]
  * Remove invalid lines from resolved integration scripts (LP: #1907878)

 -- Joao Andre Simioni <email address hidden> Mon, 16 Oct 2023 14:41:24 +0000

Changed in ifupdown (Ubuntu Jammy):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.