systemd-resolved: please do not use Google public DNS by default

Bug #1449001 reported by Malcolm Scott
68
This bug affects 13 people
Affects Status Importance Assigned to Milestone
systemd
New
Undecided
Unassigned
systemd (Debian)
Fix Released
Unknown
systemd (Ubuntu)
Fix Released
Low
Dimitri John Ledkov
Zesty
Fix Released
Low
Unassigned
Artful
Fix Released
Low
Dimitri John Ledkov

Bug Description

[Impact]
systemd-resolved will fall back to Google public DNS (8.8.8.8, etc.) in the absence of other configured DNS servers.

systemd-resolved is not enabled by default in Ubuntu 15.04, but it is installed by default and will behave in this way if enabled by the user.

$ cat /etc/systemd/resolved.conf
(...)
# Entries in this file show the compile time defaults.
(...)
#FallbackDNS=8.8.8.8 8.8.4.4 2001:4860:4860::8888 2001:4860:4860::8844

This raises privacy concerns since in the event of accidental misconfiguration DNS queries will be sent unencrypted across the internet, and potentially also security concerns given systemd-resolved does not perform DNSSEC validation and is not particularly well hardened against malicious responses e.g. from a MITM (http://www.openwall.com/lists/oss-security/2014/11/12/5).

I believe that it would be better to fail safe if no DNS server is configured -- i.e. have DNS lookups fail; it's better that the user is aware of their misconfiguration, rather than silently sending their queries to Google. The user can intentionally opt to use Google public DNS if they wish.

[Testcase]
Steps to reproduce:
1. Remove existing DNS configuration (from /etc/network/interfaces, /etc/resolv.conf, /etc/resolvconf/resolv.conf.d/*)
2. Reboot, or otherwise clear relevant state
3. sudo service systemd-resolved start
4. Note that Google's servers are listed in /run/systemd/resolve/resolv.conf
5. If systemd-resolved is enabled in /etc/nsswitch.conf (it isn't by default), observe that DNS lookups probably still work, and queries are being sent to one of Google's servers

Possible workaround/bugfix: ship a resolved.conf which clears the FallbackDNS parameter.

[Solution]
In ubuntu, we disable fallback DNS at build time, via build system configuration flags.

This issue has been discussed in the Debian BTS (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=761658). My interpretation of the Debian package maintainer's position is that a user concerned with the privacy implications shouldn't let systemd get into a state where it uses the fallback DNS servers (quoting Marco d'Itri: "Short summary: have a resolv.conf file or use DHCP"). I would argue that it's safest not to have fallback DNS servers configured at all by default.

[Regression Potential]
Missconfigured networks, that do not have a DNS server would previously magically work due to having Google DNS preconfigured regardless. With this change, such network configurations will fail to work, and one will have to properly fix network config to point at the right/existing name server.

CVE References

Martin Pitt (pitti)
Changed in systemd (Ubuntu):
status: New → Triaged
importance: Undecided → Low
Changed in systemd:
status: Unknown → Fix Released
Revision history for this message
Anders Kaseorg (andersk) wrote :

The 8.8.8.8 fallback is not only used on misconfigured systems! It’s also used for a short period while initially connecting or reconnecting to totally healthy networks with DHCP. So the excuse that privacy-conscious users should just use DHCP holds no water.

https://github.com/systemd/systemd/issues/4175#issuecomment-252571482

Martin Pitt (pitti)
tags: added: resolved
Revision history for this message
Anders Kaseorg (andersk) wrote :

In fact, having recently disabled FallbackDNS for myself, I find that I get no DNS at all maybe a quarter of the time I reboot my system. This suggests that systemd-resolved might be silently relying on the 8.8.8.8 fallback much more often than even I suggested above.

Can we try disabling FallbackDNS for this development cycle? (Then, once it becomes clear exactly how broken the DNS situation in Ubuntu has become, can we please get rid of this systemd-resolved nonsense for good?)

Revision history for this message
ed20900 (ed20900) wrote :

Yes please. I am struggling with systemd-resolved. It sucks. I am getting these messages:

Jun 15 15:34:19 n53sn systemd-resolved[1503]: Switching to fallback DNS server 8.8.8.8.
Jun 15 15:34:19 n53sn systemd-resolved[1503]: Using degraded feature set (UDP+EDNS0+DO) for DNS server 8.8.8.8.
Jun 15 15:34:20 n53sn systemd-resolved[1503]: Switching to fallback DNS server 8.8.4.4.
Jun 15 15:34:24 n53sn systemd-resolved[1503]: Switching to fallback DNS server 2001:4860:4860::8888.
Jun 15 15:34:24 n53sn systemd-resolved[1503]: Switching to fallback DNS server 2001:4860:4860::8844.
Jun 15 15:34:24 n53sn systemd-resolved[1503]: Switching to fallback DNS server 8.8.8.8.
Jun 15 15:34:24 n53sn systemd-resolved[1503]: Switching to fallback DNS server 8.8.4.4.
Jun 15 15:34:25 n53sn systemd-resolved[1503]: Switching to fallback DNS server 2001:4860:4860::8888.
Jun 15 15:34:25 n53sn systemd-resolved[1503]: Switching to fallback DNS server 2001:4860:4860::8844.
Jun 15 15:34:25 n53sn systemd-resolved[1503]: Switching to fallback DNS server 8.8.8.8.
Jun 15 15:34:29 n53sn systemd-resolved[1503]: Switching to fallback DNS server 8.8.4.4.
Jun 15 15:34:29 n53sn systemd-resolved[1503]: Using degraded feature set (UDP+EDNS0+DO) for DNS server 8.8.4.4.
Jun 15 15:34:30 n53sn systemd-resolved[1503]: Switching to fallback DNS server 2001:4860:4860::8888.
Jun 15 15:34:30 n53sn systemd-resolved[1503]: Switching to fallback DNS server 2001:4860:4860::8844.
Jun 15 15:34:30 n53sn systemd-resolved[1503]: Switching to fallback DNS server 8.8.8.8.
Jun 15 15:34:31 n53sn systemd-resolved[1503]: Switching to fallback DNS server 8.8.4.4.
Jun 15 15:34:34 n53sn systemd-resolved[1503]: Switching to fallback DNS server 2001:4860:4860::8888.
Jun 15 15:34:34 n53sn systemd-resolved[1503]: Switching to fallback DNS server 2001:4860:4860::8844.

tags: added: rls-aa-incoming
Changed in systemd (Ubuntu Artful):
assignee: nobody → Dimitri John Ledkov (xnox)
milestone: none → ubuntu-17.06
Changed in systemd (Ubuntu Artful):
status: Triaged → Fix Committed
Anders Kaseorg (andersk)
Changed in systemd:
importance: Unknown → Undecided
status: Fix Released → New
tags: removed: rls-aa-incoming
Changed in systemd (Debian):
status: Unknown → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 233-8ubuntu2

---------------
systemd (233-8ubuntu2) artful; urgency=medium

  * Disable fallback DNS servers.
    This causes resolved to call-home to google, attempt to access network when
    none is available, and spams logs. (LP: #1449001, #1698734)
  * SECURITY UPDATE: Out-of-bounds write in systemd-resolved.
    CVE-2017-9445 (LP: #1695546)

 -- Dimitri John Ledkov <email address hidden> Wed, 28 Jun 2017 13:27:28 +0100

Changed in systemd (Ubuntu Artful):
status: Fix Committed → Fix Released
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

removing xenial task, as resolved was not used by default in xenial as far as I can tell, thus this privacy issue is not as critical in xenial as it is in yakkety / zesty.

no longer affects: systemd (Ubuntu Xenial)
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in systemd (Ubuntu Yakkety):
status: New → Confirmed
Changed in systemd (Ubuntu Zesty):
status: New → Confirmed
no longer affects: systemd (Ubuntu Yakkety)
Revision history for this message
Andre (softtest) wrote :

Just upgraded to 17.04 from 16.04 and found DNS resolution stopped randomly working.

As it turns out systemd-resolved decided for some obscure reason to switch to google DNS which it can't reach compared to the locally provided recursive resolver which continues to work just fine.

In summary in my case it isn't just a privacy concern but actually breaks DNS resolution.

Aug 05 11:29:07 dtank0 systemd-resolved[8051]: Switching to system DNS server 10.124.196.1.
Aug 05 11:29:07 dtank0 systemd[1]: Started Network Name Resolution.
Aug 05 11:33:58 dtank0 systemd-resolved[8051]: Switching to fallback DNS server 8.8.8.8.

After the switch to 8.8.8.8 DNS resolution on the host stopped working because 8.8.8.8 is not reachable from the host.

Interestingly stopping and disabling systemd-resolved followed by an "resolvconf -u" did not revert the config back to a working configuration.
It required removing /run/resolvconf/interface/systemd-resolved by hand (starting systemd-resolved will add that file but not remove on stop).

Changed in systemd (Ubuntu Zesty):
status: Confirmed → In Progress
description: updated
description: updated
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Malcolm, or anyone else affected,

Accepted systemd into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/232-21ubuntu6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in systemd (Ubuntu Zesty):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-zesty
Mathew Hodson (mhodson)
Changed in systemd (Ubuntu Zesty):
importance: Undecided → Low
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Checked that zesty container has google DNS in the resolved.conf and checked that when no interfaces are configured systemd-resolve --status lists google DNS servers.

Upgraded to 232-21ubuntu6 and check that resolved.conf no longer lists google dns by default and when no links are configured there are no nameservers listed in systemd-resolve --status.

tags: added: verification-done verification-done-zesty
removed: verification-needed verification-needed-zesty
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Malcolm, or anyone else affected,

Accepted systemd into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/232-21ubuntu7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-needed verification-needed-zesty
removed: verification-done verification-done-zesty
Revision history for this message
Anders Kaseorg (andersk) wrote :

Verified using ubuntu-17.04-desktop-amd64.iso.

tags: added: verification-done verification-done-zesty
removed: verification-needed verification-needed-zesty
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 232-21ubuntu7

---------------
systemd (232-21ubuntu7) zesty; urgency=medium

  * networkd: accept `:' in ifnames in systemd/networkd. (LP: #1714933)
  * networkd: add support for ActiveSlave and PrimarySlave netdev options.
    (LP: #1709135)
  * Cherrypick upstream fix for a race between .mount and .automount units,
    which currently may result in automounts hanging. (LP: #1709649)
  * systemd.postinst: Fix-up version number check in the previous sru.
    The version check in the postinst was too tight, thus the SRU fix failed
    validation. (LP: #1710410)

systemd (232-21ubuntu6) zesty; urgency=medium

  * link: Fix offload features initialization.
    This fixes a regression introduced in v232 which caused TCP
    segmentation offloads being disabled by default, resulting in
    significant performance issues under certain conditions. (Closes: #864073)
    (LP: #1703393)
  * loginctl: Fix loginctl ignoring user given session IDs at command-line
    (LP: #1682154)
  * Disable fallback DNS servers.
    This causes resolved to call-home to google, attempt to access network when
    none is available, and spams logs. (LP: #1449001)
  * initramfs-tools: trigger udevadm add actions with subsystems first.
    This updates the initramfs-tools init-top udev script to trigger udevadm
    actions with type specified. This mimicks the
    systemd-udev-trigger.service. Without type specified only devices are
    triggered, but triggering subsystems may also be required and should happen
    before triggering the devices. This is the case for example on s390x with zdev
    generated udev rules. (LP: #1713536)
  * Enable systemd-resolved by default. (LP: #1710410)
  * core: fix systemd failing to serialize tasks correctly on daemon-reload.
    (LP: #1702823)

 -- Dimitri John Ledkov <email address hidden> Wed, 04 Oct 2017 14:22:02 +0100

Changed in systemd (Ubuntu Zesty):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.