Installing resolvconf causes loss of manually entered nameserver information in resolv.conf

This has always been a shortcoming in the resolvconf package: it does not do a fully automatic installation. The case Stefan Bader describes in this report is the most important lacuna: the package does not add information to /etc/network/interfaces.

Until now we have dealt with this by adding warnings to the README and to the package description

    This package may require some manual configuration.
    Please read the README file for detailed instructions.

and by implementing the feature described in the following debconf template.

=== BEGIN ===
Template: resolvconf/link-tail-to-original
Type: boolean
Default: false
_Description: Append original file to dynamic file?
 If the original static resolver configuration file (/etc/resolv.conf)
 contains name server addresses, those addresses should be listed on
 "dns-nameservers" lines in /etc/network/interfaces. For more
 information, please consult the resolvconf(8) man page and the
 README file.
 .
 Until /etc/network/interfaces has been edited and the affected
 interfaces brought down and up again, the name server addresses will
 not be included in the dynamically generated resolver configuration
 file.
 .
 If you choose this option, a temporary workaround will be put in
 place: a symbolic link will be created (if it does not already exist)
 from /etc/resolvconf/resolv.conf.d/tail to
 /etc/resolvconf/resolv.conf.d/original. This will cause the whole of
 that original resolver configuration file to be appended to the
 dynamically generated file.
 .
 After the required "dns-nameservers" lines have been added to
 /etc/network/interfaces, the /etc/resolvconf/resolv.conf.d/tail link
 should be replaced by one to /dev/null.
=== END ===

The link-tail-to-original feature is very unsatisfactory. It was only implemented because implementing anything better is difficult... and it's almost impossible to implement a fully automated installation which will never make a mistake.

The problem is: when the resolvconf package is installed it isn't easy to tell where the information from /etc/resolv.conf belongs in the resolvconf scheme of things. Figuring this out takes some work. Confronted with an existing /etc/resolv.conf with "nameserver 1.2.3.4", the maintainer script has to figure out what interface this is associated with, what tool configured that interface, (in the case of ifupdown:) what logical interface was used, etc.

We should at least be able to provide the admin with an interactive program that assists her in moving info from /etc/resolv.conf to /etc/network/interfaces.

Some choice at least would draw the attention. Though it is hard to say whether this would be seen as nuisance. For a fresh install I would assume the installer sets things up. At least after that there is enough information in the created file to alert anyone touching it. Must admit I don't know whether there always was one which I ignored out of habit. For myself it is not that bad anyway as I only have a limited number of personal use machines. Just trying to anticipate what happens on a larger scale role out.

Would it be an option to do nothing (not touch resolv.conf) if there is neither an active interface with dhcp nor dns-server lines in /etc/network/interfaces? Though that surely has the drawback of not really giving a hint about the newer way of defining things.

The real bug here seems to be why netcfg didn't put the dns- lines in /etc/network/interfaces because it has code to do it and definitely should have done it.

This happened because this is based on a d-i install back on Oneiric. That did make a dhcp setup which was manually changed into one with a static IP.

Right, so after talking with Stefan on IRC, the exact failing case is:
1) Installed system through debian-installer using DHCP configuration (choosing static there would have added the dns- fields)
2) Post-install, turned the system into static network configuration
3) Upgraded from Oneiric to Precise which added resolvconf

I'm guessing this might be a pretty likely scenario for servers indeed and we should be doing something about it.
Now I'm not sure whether we want to try and add the existing DNS servers (link-tail-to-original) or if we should instead show a debconf message telling the user about the change in the way we do DNS resolving and asking them to move /etc/resolvconf/resolv.conf.d/original to /etc/network/interfaces.

An option would be to use link-tail-to-original but only if we detect that we have interfaces in /etc/network/interfaces that are marked as auto but don't have dns- fields (easy to check with ifquery) and that we don't have Network Manager on the system.

I'd be against a link-tail-to-original by default as it'd mostly be confusing on systems where the DNS configuration is dynamic,

Thinking about the debconf message: I think this would be ok for me as sort of occasionally affected. And maybe it would be ok as it would warn any admin to action before a upgrade on a server farm. Or they go mental (and if we are unlucky on us).

So anything automatic might be better. Writing down my thinking (which I admit is quite hard to formulate as a nice script without too many dependencies). If I understand correctly things are bad if

1. there is no active/auto interface in interfaces that uses dhcp and
2. there is no active/auto interface in interfaces with dns-nameservers and
3. there are nameservers defined in resolv.conf

In that case the servers and search need to go somewhere into interfaces and the somewhere is the problem. I wonder whether it would be ok to just pick the interfacce related to the default route...

Stefan Bader wrote in #2:
> Would it be an option to do nothing (not touch resolv.conf)
> if there is neither an active interface with dhcp nor
> dns-server lines in /etc/network/interfaces?

The aforementioned "link-tail-to-original" approximates this.

Stéphane Graber wrote in #5:
> the exact failing case is:
> 1) Installed system [...] using DHCP configuration [...]
> 2) Post-install, turned the system into static [...]
> 3) Upgraded [...] to Precise which added resolvconf

If this is the only case then we might be able to get away with (1) detecting this case and then (2) turning on link-tail-to-original, (3) assuming that the admin who manually switches from DHCP to static is competent to edit configuration files, (4) notifying the admin that information needs to be moved from /etc/resolvconf/resolv.conf.d/original to /etc/network/interfaces.

Stéphane Graber continued:
> I'd be against a link-tail-to-original by default as it'd mostly be confusing on systems where the DNS configuration is dynamic

Agreed. It's something to do only in the case described. It was designed for exactly that case.

Stéphane Graber also wrote:
> An option would be to use link-tail-to-original but only if we
> detect that we have interfaces in /etc/network/interfaces
> that are marked as auto but don't have dns- fields
> (easy to check with ifquery) and that we don't have
> Network Manager on the system.

and Stefan Bader wrote in #6:
> If I understand correctly things are bad if
> 1. there is no active/auto [logical] interface [...] that uses dhcp, and
> 2. there is no active/auto [logical] interface [...] with dns-nameservers, and
> 3. there are nameservers defined in resolv.conf

Worry: Physical interfaces are marked "auto" but the "dns-" fields are in logical interface stanzas. So we have to assume that only logical interface stanzas are used that are named like the corresponding physical interface.

Worry: Configuration can be split across multiple files by means of the "source" feature.

Can we boil the proposal down to this?:

If there is nameserver information in the original /etc/resolv.conf and there is no physical interface marked "auto" in /e/n/i for which there is a logical interface definition in /e/n/i that could supply nameserver information dynamically (i.e., either a logical interface with the "dhcp" method or a logical interface with the "static" method and "dns-" lines) then adjust link-tail-to-original's default value to "true" and its priority to "critical" ("will probably break the system without user intervention").

I'm not sure whether this is related to this or not. I was about to raise a bug for it, but thought I'd comment here in case it is related to something someone did recently in regards to this bug. Here's my scenario:

I'm using oneiric-mini.iso to install from a local ubuntu mirror (more like a cache) that is constructed using reprepro. In essence it's a mirror of only the important bits of oneiric to set up an ubuntu-standard server + resolvconf from precise since the one in oneiric is totally broken.

Until very recently (1.63ubuntu3) this installed just fine (using DHCP for the single interface and getting the DNS info from that). Now I've just updated my "mirror" to the latest packages which pulled in resolvconf 1.63ubuntu6 from precise and now this no longer installs properly. In particular /etc/resolv.conf is still a file and not a sym-link after a clean expert install.

If I subsequently try to fix thinks by running dpkg-reconfigure resolvconf, it goes through a couple questions (Yes, I want to prepare /etc/resolv.conf and no I don't want to append the original contents) it fails saying (in effect) that /run/resolvconf does not exist. If I create /run/resolvconf and /run/resolvconf/interface manually and then run through dpkg-reconfigure it appears to start coming right.

I don't have the means to try this on a plain precise install at the moment.

Anybody any comments?

Seb

> it fails saying (in effect) that /run/resolvconf does not exist

Please compare https://bugs.launchpad.net/ubuntu/+source/resolvconf/+bug/922491

Thomas Hood wrote:
> Worry: Physical interfaces are marked "auto" but the "dns-" fields are in logical interface
> stanzas. So we have to assume that only logical interface stanzas are used that are named
> like the corresponding physical interface.

Not sure I understand the difference here. In my understanding the name used in the auto and iface lines, both refer to a logical interface (iow after they have been renamed by udev/kernel based).

> Worry: Configuration can be split across multiple files by means of the "source" feature.

Agreed that this makes it complicated. Though I must admit, that I am not sure since when such a thing existed in previous releases. If the man page is correct (in Oneriric) options are quite limited.

> Can we boil the proposal down to this?:

Mostly agree. The only thing that worries me is that I do not understand what exactly an outcome of merging things will be and what effects that "priority critical" will have. The link-tail-to original only happens when there is nothing in e/n/i that would create any content. So the dynamic part is empty (unless the configuration is changed). IMO this results in a dynamic file that should be the same as before. That could change when the admin only adds dhcp interfaces or dns- lines without removing the file used to append. Would that break the system? But in general I think this sounds like it should work.

Hate to spoil this bug and I probably should open a new one. Doing orchestra install tests I got the following from a fresh precise (seed modified to install ubuntu-desktop) install yesterday:

ii resolvconf 1.63ubuntu6 name server information handler

$ ls -la /etc/resolv.conf
-rw-r--r-- 1 root root 40 Jan 30 19:38 /etc/resolv.conf

Seems network manager is in charge (according to the comment in e/n/i). Is it expected to have resolv.conf not as a link to /run?

> Not sure I understand the difference here.

interfaces(5):

    Lines beginning with the word "auto" are used to identify
    the physical interfaces to be brought up when ifup is run
    with the -a option.

[...]

    Stanzas defining logical interfaces start with a line
    consisting of the word "iface" followed by the name of
    the logical interface.

Read the passages on mapping.

> [...]
> Would that break the system?

Appending the original file to the end of resolv.conf can't have terribly deleterious effects. (<Knock wood>) The worst that can happen is that the original file contains obsolete "domain" or "search" lines which override a new valid "search" line. (Only the last one of these counts.) The original file may also list obsolete nameserver addresses but so long as new valid dynamic ones are listed earlier this doesn't make any difference. (Only the first functional address counts.) And listing an obsolete nameserver address only has the effect of delaying the name service failure condition by a few seconds. These issues are minor compared with the problem of resolv.conf *lacking* nameserver addresses that *are* in the original file. That's what will bring the pitchfork-armed admins to our doors.
--
Thomas

Stefan Bader wrote in #11:
> Seems network manager is in charge (according to
> the comment in e/n/i). Is it expected to have resolv.conf
> not as a link to /run?

See https://bugs.launchpad.net/ubuntu/+source/resolvconf/+bug/922677 especially comments 2 and 4.

Thomas Hood wrote:
> interfaces(5):
>
> Read the passages on mapping.

Ok, point taken. It is much much more complicated than I believed. Guess I need to anticipate beyond Linux.

>> [...]
>> Would that break the system?

> And listing an obsolete nameserver address only has the effect of delaying the
> name service failure condition by a few seconds. These issues are minor
> compared with the problem of resolv.conf *lacking* nameserver addresses that
> *are* in the original file. That's what will bring the pitchfork-armed admins to our
> doors.

And since we talk about a specific case of upgrading really, either the addresses are already obsolete or not. So that should not make things really worse. I guess the conditional link-to-tail solution sounds like the way forward then.
Thanks for the reference to the other bug. That is handled over there then.

On Mon, Jan 30, 2012 at 07:53:00PM -0000, Thomas Hood wrote:
> Can we boil the proposal down to this?:

> If there is nameserver information in the original /etc/resolv.conf and
> there is no physical interface marked "auto" in /e/n/i for which there
> is a logical interface definition in /e/n/i that could supply nameserver
> information dynamically (i.e., either a logical interface with the
> "dhcp" method or a logical interface with the "static" method and "dns-"
> lines) then adjust link-tail-to-original's default value to "true" and
> its priority to "critical" ("will probably break the system without
> user intervention").

Ubuntu policy prohibits debconf prompting on the desktop as part of the
standard upgrade. We need to figure out how to make this not break the
system without having to prompt the user about it. (AFAICS, the proposal
above would result in prompts for any desktop system that is using NM
exclusively for network configuration - which would mean there are no auto
interfaces in /e/n/i.)

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

Steve wrote:
> Ubuntu policy prohibits debconf prompting [...]

1. OK, no prompting. :)

If we don't debconf-prompt then what's the best way of letting the admin know that she should edit /e/n/i?

> (AFAICS, the proposal above would result in prompts
> for any desktop system that is using NM exclusively
> for network configuration - which would mean there
> are no auto interfaces in /e/n/i.)

2. You are right, I forgot to mention NM in the conditions.

Proposal version 2:

If there is nameserver information in the original /etc/resolv.conf which wasn't written by NetworkManager and there is nothing set up to supply dynamic nameserver information then adjust link-tail-to-original's default value to "true". By "nothing is set up to supply dynamic nameserver information" we mean: NetworkManager is not active and there is no physical interface marked "auto" in /e/n/i for which there is a logical interface definition in /e/n/i that is either "dhcp"-methodic a "static"-methodic with "dns-" lines.

My own familiarity with networking setups is not exceptionally broad, so it would be good if someone who does have broad familiarity would look at this proposal and think about consequences for, e.g., servers with multiple network interfaces, laptops with VPN interfaces, etc.
--
Thomas

Status changed to 'Confirmed' because the bug affects multiple users.

setting to critical. After an upgrade of a working system from Oneiric to Precise, it is unable to do DNS resolution.

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/924734

Hmm, Jean-Baptiste, could you give a bit more details on how that system was initially installed?

My understanding is that upgrade works fine if it's a desktop install with Network Manager or if it was installed with debian-installer and had a static config.

If you installed the machine using DHCP in d-i, then manually altered the configuration in /etc/network/interfaces to be static but without adding the dns-nameservers and dns-search options that d-i would have added, it's indeed a bug but it's far from being Critical (as it's not something that'd break with a default install).

This system was initially running Natty and installed with d-i. The file /etc/network/interfaces was changed to be static. Everything worked. The machine was then upgraded to Oneiric and still working. Then upgraded to Precise and hit by this defect.

It is critical because the configuration of the system was changed, without notification, in a way that severely affects the whole system and make applications relying on DNS resolution unusable.

After a bit of discussion with Steve on IRC, we decided that we don't want to prompt the user, even in the weirdest scenario.
Instead, I developed the following list of criteria to determine whether or not to link the current resolv.conf to tail:

 1) If "Generated by NetworkManager" in /etc/resolv.conf => exit (it's already dynamic, managed by NM, so no need to be appended)
 2) If empty /etc/resolv.conf (no nameserver entry) => exit (it's empty, so no need to be appended)
 3) Get list of configured network interfaces "ifquery --allow auto --list" => if only loopback in the list => append (at this point we have no interface configuration but a resolv.conf containing something, not sure if it's dynamic or static but better keep it)
 4) Grep /etc/network/interfaces for "inet dhcp" (non-commented) => if any => exit (it's already dynamic, managed by dhclient, so no need to be appended)
 5) Grep /etc/network/interfaces for dns- entries (non-commented) => if any => exit (resolvconf will use them, so no need to be appended)
 6) At this point, only configured interfaces in /etc/network/interfaces without dns-* fields and not using DHCP should remain => append (usual case of system installed with DHCP and manually altered afterwards, we need to keep these)

Is there any case that's not covered by the above checks or something you think should be handled differently?

Re: comment #22

Ik looks as if you can omit step 3 in your algorithm.

Either way, your algorithm seems to be equivalent to my "proposal #2" from comment #16. Good sign that we came up with the same set of conditions.

I'll comment further once I get back to my laptop (now on my way to the FOSDEM beer event...).

indeed, 3) won't match either 4) or 5) and so will be caught by 6) giving the same result

We should grep for "dns-nameservers" and not just for "dns-" in step 5 in the algorithm.

The algorithm will still fail in a case like this:

=== /e/n/i ===
auto eth0

iface eth0 inet static
    address 192.168.0.2
    netmask 255.255.255.0

iface foo inet static
    dns-nameservers 1.2.3.4
================

In this case the algorithm will fail to link tail to original (and thus, fail to cause /sbin/resolvconf to include the original as the tail of resolv.conf) because it will see a dns-nameservers line in a logical interface even though that logical interface isn't used. Perhaps we should only refrain from linking tail to original due to the presence of a dns-nameservers line if the default mapping is used and the dns-nameservers line found is in a logical interface definition whose name is listed on an "auto" line. That would be safer. (Remember, the consequences of wrongly omitting the link are worse than the consequences of wrongly creating it.)

We can refine the algorithm even further but we will never be able to make it work perfectly under all possible circumstances. I've mentioned the "mapping" and "source" features but there are other ways in which the networking configuration may have been customized by the admin. There's no limit to what people may have put onto "up" lines.

Because we can end up with tail linked to original in cases where it's not appropriate, it is important to minimize the negative consequences of this.

I see one small way of reducing the possible negative impact of an unneeded tail->original. Consider that /sbin/resolvconf composes a resolv.conf file from:

    head
    base + dynamic content, ordered according to /etc/resolvconf/interface-order
    tail

For nameserver addresses it's fine to include them in the tail file because any dynamically added addresses will precede them and thus override them.

For "domain" and "search" lines it's not fine since, according to resolv.conf(5), the last instance wins. Thus a "domain" or "search" line in the original file will always override the dynamic "search" line.

It would thus be better to include "domain" and "search" lines from the original file in the head file and not in the tail file.

What I would propose is that instead of creating a symlink, the installation program splits up the original file into "head-from-original" (containing domain and/or search lines) and "tail-from-original" (including nameserver lines and anything else) and that /sbin/resolvconf builds resolv.conf from

    head
    head-from-original
    base + dynamic content
    tail-from-original
    tail

--
Thomas

Here's the next iteration of the proposal.

1) /etc/resolv.conf contains no nameserver information => exit
2) "Generated by NetworkManager" in /etc/resolv.conf => exit
3) /etc/network/interfaces contains "mapping" or "source" lines => append
4) /etc/network/interfaces contains a logical interface definition with name N and configuration method name "dhcp" and the file also contains an "auto N" line => exit
5) /etc/network/interfaces contains a logical interface definition with name N and a "dns-nameservers" option and the file also contains an "auto N" line => exit
6) Append

Time is short so I'd say we should go ahead and do link-tail-to-original rather than something fancier as described in #25.

This algorithm seems rather complicated. With the split from #25, is there any need for trying to figure out whether or not to do all these checks?

Why not do the split from #25 unconditionally. I can only see negative impact from not including old information. With #25, I can see no negative impact from including the old information when it is in fact not needed.

Re comment #27:

It's an interesting idea, but I think we should, so far as possible, avoid including original nameserver information statically, for the simple reason that the correct nameserver information can change. For example, if a current nameserver address has been obtained from NetworkManager then the address could easily be different next time. Note that invalid search domains or nameserver addresses cause delays.

I'm not entirely sure that this bug also applies to DHCP setups, so I've opened bug #933723 as it is affecting the entire certification lab setup

Regarding comment #29: Report #933723 was traced to a bug in bind9 -- different issue from the one here (#923685).

>> It looks as if you can omit step 3 in your algorithm.
> indeed, 3) won't match either 4) or 5) and so will be caught by 6) giving the same result

Except in the case that there are inet dhcp entries in /e/n/i that are *not* marked auto. Thinking about whether this should actually be handled differently.

yeah, in that case 4) is a reasonable option anyway, so we don't need to treat 3) specially.

On Sat, Feb 11, 2012 at 08:16:57PM -0000, Thomas Hood wrote:
> Here's the next iteration of the proposal.

> 1) /etc/resolv.conf contains no nameserver information => exit
> 2) "Generated by NetworkManager" in /etc/resolv.conf => exit
> 3) /etc/network/interfaces contains "mapping" or "source" lines => append

A 'mapping' stanza should have no effect at all on whether we append.

If a 'source' stanza exists at all, it most likely points to other interface
definitions that *do* provide resolver info, so why would we want to append
in that case except as a last resort? I.e., I think this also collapses
into the fallback case 6).

> 4) /etc/network/interfaces contains a logical interface definition with
> name N and configuration method name "dhcp" and the file also contains an
> "auto N" line => exit
> 5) /etc/network/interfaces contains a logical interface definition with
> name N and a "dns-nameservers" option and the file also contains an "auto
> N" line => exit

The plan I discussed with Stéphane explicitly does *not* require the auto
line for either of these. The reasoning is that there is already an
interface definition which we know will overwrite the resolver info in some
cases, so /etc/resolv.conf is *not* static. If the admin assumed it was
static, then they've made a judgement error; resolvconf is merely exposing
this error, not causing a bug in its own right.

> 6) Append

So I think that, modulo the dropping of 3) ifquery, the earlier proposal
from Stéphane is still what we want to go with.

I think we agree on the main principle, which is: append, unless we are confident that dynamic nameserver information will be supplied. In the case of ifupdown the condition means: there is a logical interface definition that is either "dhcp" or contains "dns-nameservers".

#26 differs from #33 only in applying the additional condition that there must be evidence that the logical interface definition is automatically activated, i.e., for logical interface L there is a line "auto L" and the default mapping is in effect. (Any use of "source" or "mapping" calls into question whether the default mapping is in effect.)

I follow your reasoning in #33, though; and not applying the additional condition will simplify the code considerably.

This bug was fixed in the package resolvconf - 1.63ubuntu8

---------------
resolvconf (1.63ubuntu8) precise; urgency=low

  * Use a post-stop script for disabling resolvconf, not a pre-stop script,
    since upstart seems to be silently ignoring pre-stop when there's no
    main process. LP: #933566.
  * debian/postinst: mkdir -p /run/resolvconf/interface again, just in case
    there's been a reboot between the preinst and postinst which would wipe
    out /run. May or may not address LP 933035.
  * debian/config, debian/templates, debian/postinst: if we don't know that
    /etc/resolv.conf was being dynamically managed before install (in at
    least some cases), link the original contents of /etc/resolv.conf to
    /etc/resolvconf/resolv.conf.d/tail so that any statically configured
    nameservers aren't lost. LP: #923685.
  * when called with --enable-updates, ignore failures from the hooks.
    LP: #933723.
 -- Steve Langasek <email address hidden> Mon, 20 Feb 2012 19:17:32 +0000

I just upgraded to Precise and was bitten by this bug! Boo!

I had a statically defined resolv.conf and it was rudely removed by resolvconf.

Mark,

You'll notice that this bug is marked as having been fixed two months ago. If your /etc/resolv.conf contents were incorrect after upgrade, please file a new bug report and provide more details.