DNS domain search paths not updated when VPN started

network-manager has resolved integration and it does push search domains to resolved and they are updated in the /etc/resolv.conf.

This should continue to work in 17.10. Hence marking this bug affect network-manager and network-manager-openvpn

To be clear, they are NOT updated in /etc/resolv.conf. They ARE updated in /run/NetworkManager/resolv.conf, but that has no effect since we're not using that for resolution: instead we're using systemd-resolve.

And as I noted above, it does appear that systemd-resolve is notified about the search domains, because "systemd-resolve --status" does show them associated with the openvpn tun0 device.

However, those search domains are never manifested in the /run/systemd/resolve/stub-resolv.conf which is what /etc/resolv.conf is pointing to, so they never take effect.

Thank you for the update. I will check the code, and will try to resolve this as soon as possible.

Does everything work, if you route all traffic via VPN?

Right, so it seems like NM checks the never-default setting, and doesn't push them as resolve domains =/ I will test things out to be sure.

Specifically:

$ nmcli c show YOUR_VNP_CONNECTION_NAME | grep never-default
ipv4.never-default: yes
ipv6.never-default: yes

And if it is never-default, it doesn't push search domains out, only routing domains.

I believe this might be actually correctly intended configuration of the openvpn and network-manager and resolved. Since all traffic is not routed to the VPN connection, its domain names should not be used for resolution. E.g. resolving "foo.company.example.net" should work and should end up being resolved via nameserver on the vpn, and not leak DNS resolution to public DNS server, and one should be able to connect to that host. But resovling "foo" should be failing.

I understand this is a change of behaviour, thus does seem like a regression. I will consult on what is correct behaviour and how to best fix it.

I'm not sure I understand what you mean. In a typical configuration you never send "foo" to the nameservice, there's always a search domain and those lookups are always tried first (because the default value for the ndots is 1). This is handled by the libc resolver linked into every program, it's not handled by a central service.

I don't have any idea how systemd-resolve works. But, my understanding of how it used to work with dnsmasq using split tunneling was that the nameservice mapped domains to DNS servers, and the resolver would only forward requests to the DNS server that matched the domain for the host being requested.

If you have a VPN interface that adds "mycorp.com" to the search domain that appears in /etc/resolv.conf search, so it contains "mycorp.com localdomain" for example. Then when someone tries to resolve "myhost", the libc resolver sees that there are no dots here and so it starts appending search paths to the hostname, in order, and sending them to the DNS service to look up. So first it will send "myhost.mycorp.com" to the resolver. The resolver sees that the hostname ends in "mycorp.com" and it knows that the VPN DNS servers "own" that domain, so it forwards that request to those servers for lookup.

If that doesn't match, then the libc resolver will try to look up "myhost.localdomain". That does NOT match the VPN domain, so it will not try to forward that to the DNS servers for the VPN connection and instead use a different resolver.

I don't think there's any information leakage here.

It would probably be a good idea to forward it to n-m upstream as well on bugzilla.gnome.org

Hi Sebastien, thanks for your interest. This issue is causing me extreme pain. I believe I'm going to have to reconfigure N-M to use the old-style dnsmasq setup and throw out systemd-resolve pretty soon.

It's not clear to me whether this is an N-M bug vs. a systemd bug, but I'm happy to file more bugs. Did you want me to file a bug with N-M?

Paul Smith, what you describe is information leakage and shouldn't IMHO work as you say by default.

Consider that I'm connected to a corporate network and have an (untrusted) VPN active which I only want to use to access resources on its network (never-default: yes). Then by having the resolver adding the domain of the VPN network to short name lookups could leak those local names to the remote VPN (depending on the order the lookups are performed in) and potentially allow the untrusted network to take over internal services that are accessed using short names. This could happen by mistake also (such as setting "mail" as your smtp server if the remote network uses the same name).

I don't think the order of the lookups can be controlled to prevent this, for example what should determine the order when you have to VPN active?

to->two

BTW, the long timeout you see on short name lookup failures is most likely due to LLMNR being on by default. I think this is insane and I always switch it off in /etc/systemd/resolved.conf .

Andreas: unfortunately disallowing short name lookups is not acceptable: many environments use short names in embedded URLs all over the place, and without domain search paths the entire environment is rendered completely unusable (e.g., URLs are simply https://tools/foo or https://wiki/foo or whatever; this means no cross-links work).

Connecting to an untrusted VPN is a tiny sliver of a minority of the usage of VPN; 99.99% of the time when someone connects to a VPN they're trying to create a secure link to another trusted network (e.g., working remotely).

If you want to provide support for both modes of handling short names with some kind of checkbox to select between them that's fine with me, but the current behavior is a serious loss of functionality from previous solutions.

This change in behavior is deliberate. There are two mutually incompatible interpretations of DNS search lists provided via a VPN connection. One is for split DNS, to say "this is the list of domains for which you should send lookups to the accompanying DNS server". The other is to use it as a search list for resolv.conf. Unfortunately, interpreting wrongly in either direction breaks client configs. But whereas there are other ways that one can configure the behavior of resolv.conf to add search domains, the only reasonable way to configure split DNS is to do so by providing this information directly from network-manager-openvpn to systemd-resolved.

It may be that network-manager-openvpn needs an additional configuration option, to allow the user to declare which of these two ways (or both, or neither) they want to use the VPN server-provided DNS search list.

Status changed to 'Confirmed' because the bug affects multiple users.

I hope something can be worked out before this lands in the next LTS-release. This breaks one of the most common uses of a VPN (remote work).

Right now a workaround is using this script after connecting to a VPN-server:

https://github.com/jonathanio/update-systemd-resolved

It calls systemd-resolved via DBus to add the proper settings after the OpenVPN connection is established.

Of course this means abandoning NetworkManager for use with OpenVPN, because there is also no way to specify --up and --down parameters.

I'm not sure why it's being asserted that these two uses are mutually incompatible, especially since I've been using them in a coordinated way forever and can still do so (at the expense of junking systemd-resolve and going back to using dnsmasq and a custom configuration, which is obviously a big pain). I've even, in the past, used MULTIPLE VPNs, even at the same time, and it still just works.

If the resolver library (e.g., gethostbyname etc.) gets an unqualified hostname, it uses the search path just as it always has including ndots and all that stuff, to generate FQ hostnames. No change there.

When the local resolver caching service (dnsmasq, systemd-resolv) gets a FQ hostname it looks through the extensions provided by the VPN DHCP information and if the hostname matches that extension it forwards the lookup to the DNS server for that VPN. If it doesn't match, it doesn't forward the request. If it doesn't match any of the VPN search paths, it forwards the request to the default DNS servers.

I honestly don't understand why we're considering these uses incompatible. They seem to me to be exactly compatible and exactly what you want to do, at least the vast majority of the time.

I would also like to chime in, that it is very useful to have the ability to append a DNS-suffix when doing short-name DNS lookups on a split-tunnel VPN connection.

It would be great if the devs could find a way to include this functionality in the upcoming LTS! :)

We won't be able to add the option to select between the two options this cycle.

So, is anyone going to fix this? It is in LTS now and causing havoc. Fwiw, I support that it should allow split resolution just like it used to do.

This change was a major breaking one that shouldn't have been forced on people. Security opinions don't belong in implementation, only configuration.

What does that rls-bb-notfixing tag mean? This is just a bug that won't be fixed? Split tunnel DNS with openconnect is still completely broken in 19.04 and it would be very nice to have this working. The last couple years of Ubuntu releases have required me to ditch systemd-resolved in order to access my company's internal services while remote.

My guess is that rls-bb-notfixing means this is not getting fixed in Bionic Beaver 18.04. Really hoping a fix can make it into the 20.04 LTS release though.

I agree with Paul - systemd-resolve is not doing what it promises to do. From systemd.network(5):

    Both "search" and "routing-only" domains are used for routing of DNS queries:
    look-ups for host names ending in those domains (hence also single label names, if
    any "search domains" are listed), are routed to the DNS servers configured for
    this interface. The domain routing logic is particularly useful on multi-homed
    hosts with DNS servers serving particular private DNS zones on each interface.

    The "routing-only" domain "~." (the tilde indicating definition of a routing
    domain, the dot referring to the DNS root domain which is the implied suffix of
    all valid DNS names) has special effect. It causes all DNS traffic which does not
    match another configured domain routing entry to be routed to DNS servers
    specified for this interface. This setting is useful to prefer a certain set of
    DNS servers if a link on which they are connected is available.

    This setting is read by systemd-resolved.service(8). "Search domains" correspond
    to the domain and search entries in resolv.conf(5). Domain name routing has no
    equivalent in the traditional glibc API, which has no concept of domain name
    servers limited to a specific link.

One partial workaround is to remove `[!UNAVAIL=return]` after `resolve` in /etc/nsswitch.conf:

    hosts: files mdns4_minimal [NOTFOUND=return] resolve dns myhostname

That way a traditional DNS resolution is done after systemd-resolve fails. This works for everything I've tested so far _except_ Google Chrome.

Still an issue in 20.04 LTS

Still an issue in 20.10