NetworkManager causes orphaned inodes

Status changed to 'Confirmed' because the bug affects multiple users.

Thanks for the info Steven. I looked into this a bit more.

It would seem this bug introduced the omitting: bug #869635

The resolution makes, I think, a false assumption that when network-manager is killed that it will take dhclient with it. This does not seem to be the case. Or, more likely, the processes are not waited on properly after being sent the kill signal. I looked through the code of network manager quite a bit, and could not find any code that explicitly stops the dhcp manager when SIGTERM is received, though I am no glib expert, so its possible when the main loop is exitted there is some kind of process that fires the watches which lead to the stop methods being called.

I do see that the dhclients die when I manually stop the network-manager job. This job is guaranteed to stop before umountrootfs because it stops on stopping dbus, which stops on deconfiguring-networking. deconfiguring-networking is always waited on during the shutdown.

One possiblity is that network-manager takes longer than 5 seconds to stop, at which point, upstart would send it a SIGKILL, and move on. To test that hypothesis we can just raise the kill timeout in the job to a very high level with

kill timeout 100

and then try to reproduce the problem.

Clint, I don't see the dhclient process die when network-manager is stopped. But I think I was guilty of some fuzzy thinking in saying that the problem is due to the pid file being held open; the problem is that the dhclient pid file isn't deleted from /var/run/sendsigs.omit.d (so /etc/init.d/sendsigs doesn't kill dhclient at shutdown time) *and* dhclient has a dhcp lease file open for write. The following should (I hope) show what I'm talking about:

$ ps auxw | grep NetworkManager | grep -v grep # NetworkManager and dhclient are both running
$ ls /var/run/sendsigs.omit.d # there are pid files for both dhclient and dnsmasq
$ sudo stop network-manager
$ ps auxw | grep NetworkManager | grep -v grep # now only dhclient is still running
$ ls /var/run/sendsigs.omit.d # and the dhclient pid file is still there
$ sudo lsof -c dhclient | grep lease # dhclient still has a lease file open for write

Because sendsigs won't kill dhclient and the lease file is open for write, /etc/init.d/umountroot fails and the filesystem must be recovered. And dhclient still has libc .so files open as well, which is what leads to orphaned inodes if those files have been overwritten during a libc upgrade.

If the dhclient pid file was removed from /var/run/sendsigs.omit.d when network-manager is stopped, then sendsigs would kill dhclient at shutdown. Or, if dhclient could be terminated when network-manager shuts down, that should work too. I don't understand enough to guess why the pid files are written to sendsigs.omit.d in the first place...

One last data point: I'm seeing the problem only with wired connections when "Available to all users" has been checked for that connection (right-click on the nm-applet icon, select "Edit Connections...", select the connection and click "Edit..." then check/uncheck "Available to all users" in the lower left).

To /etc/init.d/umountroot, just before the 'mount -n -o remount,ro /' sequence, I added:

/bin/ps > /DEBUG
/usr/bin/lsof >> DEBUG

/DEBUG shows that when (and only when) "Available to all users" has been selected for a wired connection, when umountroot runs dhclient is still running and has a dhcp lease file open for write; consequently umountroot fails. I'm attaching a copy of this DEBUG file. For wired connections with "Available to all users" unchecked, and for wireless connections, dhclient has gone away by the time umountroot runs and so there is no problem in those cases.

Can confirm:

after unchecking "Available to all users" on my wired card:

- on shutdown/reboot is now no "mount: / is bussy" at the end
- on boot is now no "ext4-fs : fsck required" (which BTW in my case had end with no orphans in 9/10 cases)

I'm unable to reproduce this. On all my systems NetworkManager shuts down cleanly and indeed doesn't stop dhclient, but that's "expected" behavior, since when you're turning NM off you may wish to retain wired connections, which happens to indeed be the case for users with mounted network shares.

Is there anything else that's special with your installation which could explain this being an issue?

I am using ext2 as a /boot. Also I have a tmpfs mounted as:

tmpfs /tmp tmpfs nodev,nosuid 0 0

Other than that, everthing seems stock. I have installed/deinstalled alot of packages. But debsums seems to find no error. I just wish fsck would learn how to chill.

The underlying problem is that, at system shutdown time, dhclient isn't
given the opportunity to gracefully exit, closing any files it has open.
This causes /etc/init.d/umountroot to fail. When umountroot fails, the
filesystem must be recovered.

There is, to the best of my knowledge, nothing particularly unusual with
the various installs of 12.04 that I've been testing, and the problem is
evident with all of them. One difference might be that I have the
splash screen turned off in grub and so can see the messages. Maybe
that explains why you're not seeing them.

At some point, possibly in oneiric, changes were made which put the
dhclient pid in /var/run/sendsigs.omit.d/ and I believe that's the basis
of the problem. And it IS a problem - the root filesystem should not
need recovery on every reboot. The "orphaned inodes" messages are just
icing on the cake. Do "sudo apt-get install --reinstall libc6" then
reboot, and run "dmesg | grep EXT" and you should see the messages.

These changes were made on purpose in NetworkManager to avoid init killing dhclient before NetworkManager is shut down, which solves slowness at shutdown and also solves a long-time issue with mounted network shares at shutdown. NetworkManager also doesn't stop dhclient by itself on shutdown on purpose, since you may want to remain connected to a wired network despite stopping NetworkManager.

Yes, yes, I understand all of that, there is no need to continue
repeating it. Your motivations don't change the fact that, in trying to
fix one problem, you've created another. Your changes are causing
umountroot to FAIL at shutdown time.

After installing a clean copy of ubuntu-12.04-beta2-dvd-amd64.iso--and applying updates--this bug is still there. I install ed in vmware. I used the same command as Steven Farmer to see the errors in dmseg.

On my test systems I fixed this bug by editing /etc/init.d/umountroot and adding...

/sbin/killall5 -15
sleep 2

...just before the part where it remounts the root fs RO. That tells the system to
ask all non-kernel tasks to clean up and exit. No idea if this is a "proper" fix
but on my installs it results in a clean shutdown and no recovery on restart.
Also not a NetworkManager file... but rather a generic way of dealing with the issue
of tasks still running when taking down the system. Would be better if it only
terminated tasks that actually have open files on the root FS.

I found a solution. But it's bad idea.
I added the following line to /etc/rc0.d/S20sendsigs
 nmcli nm sleep true
Because nmcli needs Dbus, nmcli must call before killall.
This is to resolve this bug. But another problem occurs.
There is umountnfs after this.

I believe the right answer here is that NM should be reaping all of its dhclient helpers on shutdown, including those for wired interfaces. This is the only robust way to handle the system shutdown case, and it makes sense to me that this would be the expected behavior when shutting down NM any other time as well.

Oops, why is this still Incomplete? I've been working on a patch and it's pretty much ready now; so setting to In Progress until an upload to -proposed.

it seems that this issue reported here also affect me : https://bugs.launchpad.net/ubuntu/+source/linux/+bug/962113

Hello Steven, or anyone else affected,

Accepted network-manager into precise-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

After installing network-manager 0.9.4.0-0ubuntu4 from precise-proposed,
things are looking good. The root filesystem is clean on reboot even
after a reinstall of libc6. Many thanks to Mathieu!

This bug was fixed in the package network-manager - 0.9.4.0-0ubuntu4

---------------
network-manager (0.9.4.0-0ubuntu4) precise-proposed; urgency=low

  * debian/patches/quit_dhclient_on_exit.patch: make sure we kill dhclient when
    we are stopping. (LP: #963106)
  * debian/patches/lp949743_ensure_remoteconnection_disconnects_494f0a2.patch:
    make sure NMRemoteConnection objects have their signals disconnected.
    (LP: #949743)
  * debian/patches/lp990011_use_tempaddr_sysctl_default.patch: *really* fix the
    use_tempaddr sysctl to properly be set to 2 on the wired interface.
    (LP: #990011)
 -- Mathieu Trudel-Lapierre <email address hidden> Fri, 27 Apr 2012 18:13:56 -0400

I fear I cannot verify this fix :(

I'm still suffering from this problem, using:
network-manager/precise uptodate 0.9.4.0-0ubuntu4

All the networkmanager stuff wants to terminate AFTER the umountroot script. I used "halt" to shutdown the sytem without poweroff and made a screenshot to show you (see attached umountroot_fail.jpg).

Please help, as I really fear for my file system... :-S

Additionally I let umountroot make a ps -ef and a lsof -nP just before trying to remount readonly. The output is attached, too (see safe.root.debug.txt)

Daniel,

I notice "/etc/rc6.d/S03umountroot stop" in your ps output. On my 12.04
installs I have S60umountroot in /etc/rc0.d and /etc/rc6.d. S03 sounds
awfully early to me.

Try "sudo mv /etc/rc6.d/S03umountroot /etc/rc6.d/S60umountroot" and then
do "sudo shutdown -H now" and see what happens.

Indeed, umountroot is supposed to be at 60; based on how we ship these scripts on a default install.

umountroot must be at least lower than S35networking; which will emit "deconfiguring-networking": that's the stop condition for dbus, which is the stopping condition (stopping dbus) for network-manager. Only once NM is stopped can you safely run umountroot; otherwise the dhclient binaries would still be running.

Thank you so much Mathieu, that was a stunning catch. Something obviously shuffled my rc links around O.o

I once (months ago) had the problem that vmware player was changing the position of "halt" but seemingly it changed a lot more :(

Anyway: Obviously my problem is not this bug and I'm now sure this bug really is solved. I will now fix my script links... If anybody has an idea how I can easily reset them to their defaults, I would be very pleased. Using "update-rc.d -f <scriptname> remove" and then "... defaults" did not work, as it sets everything to "20" only.

Daniel, try:

$ sudo update-rc.d -f umountroot remove
$ sudo update-rc.d umountroot start 60 0 6 .

Note the period at the end of the 2nd command. And yes, my previous
suggestion of using mv won't work, you have to use update-rc.d, sorry.

When I used update-rc.d to put umountroot at 03 as you have, I see the
same symptoms you do. Using the above commands to put it back to 60
fixes the problem. Hope this helps, please let us know.

Thank you very much. Sadly it turned out that most of the sequence numbers are totally wrong. It's a wonder that still everything worked flawlessly except this unmounting...

I searched for a way to set all sequence numbers to their distribution default, but I failed to find such a way.

I now have a file list from another correct precise installation and I'm fixing all the links manually using update-rc.d just as you described. After I have fixed the vast majority, I will reboot and report back, if this fixed the initial problem, but this really will take some time...

After fixing all my init script symlinks, this problem is gone. Thank you all so much for your help.

Some background information for others who might have my problem:

Cause:
Installing Vmware Player broke all rc script links. I guess that happened already when still running ubuntu 11.10. See here: http://communities.vmware.com/message/1846324

Solution:
For every service with broken symlink I:
- deleted it using: update-rc.d -f <servicename> remove
- searched for the related package using: dpkg -S /etc/init.d/<servicename>
- reconfigured this package using: dpkg-reconfigure <packagename>

reconfiguration ran the postinstall script and recreated the now missing symlinks with correct sequence numbers

The perl script /usr/share/sysv-rc/saveconfig prints out the current
configuration as a series of update-rc.d commands. If you like, I can
send you a copy from a more-or-less vanilla 12.04 install. It can't be
completely vanilla without my doing a fresh install in a VM, and I won't
volunteer for that (I'm running non-pae hardware, and they've made 12.04
installs *very* aggravating). After logging in to launchpad, you should
be able to find my email address.

Oops, I sent my previous message before I saw yours saying you've fixed things. Congratulations, and thanks for sharing the method - I'll have to remember that.

I think I have a problem with this message also on shut down:

"mount: / is busy."

I don't remember when it started to appear. I just realized this messages yesterday. So, I google over looking for answer. I hope here is the right place to ask.

Several ways I have tried:
1. I have uninstalled network manager and and reinstalled libc6. (just like what Steven described on https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/952315 ). Still, no changes, the message still appears.
2. I have also tried:
    $ sudo update-rc.d -f umountroot remove
    $ sudo update-rc.d umountroot start 60 0 6 .

Also, no changes.

3. I have now network-manager_0.9.4.0-0ubuntu4.1_amd64.deb installed on my system, and the message still appears.

I don't really have idea about this, but here I attach my ps -f and losf -nP output. Hope there will any help.
I'm using Ubuntu 12.04 64bit.

Thanks.

Tuhu, your ps output shows quite a lot still running. Did you execute
the command from a terminal in the desktop environment? To be helpful,
the ps and lsof commands should be run from the /etc/init.d/umountroot
script. See message #5 above.

Before adding debug commands to umountroot, you might want to try
un-checking "Available to all users" for your active network connection.
If the "mount: / is busy" message is no longer present when you shut the
system down, then your problem might be related to this bug.

Either way, you'll probably want to proceed with getting some debug
print from umountroot. There was a typo in message #5, although I'm not
sure that it makes a difference in practice. The lsof command should
read:

/usr/bin/lsof >> /DEBUG

Be sure to make a backup copy of umountroot before editing it:

sudo cp -p /etc/init.d/umountroot ~

Thanks for the respond Steven,

Yes, I did execute it from a terminal in the desktop environment.

I have un-checked "Available to all users" for my active network connection, but, the "mount: / is busy" message still appears. So, I think my problem is not with the network manager.

So, as you suggested, I added:
/bin/ps > /DEBUG
/usr/bin/lsof >> /DEBUG
to /etc/init.d/umountroot before the 'mount -n -o remount,ro /' sequence. And you can see the out put in the attachment. This time, I input the 'ps -ef and losf -nP' command not from terminal but from tty1. Was that what you meant? Or how is it to reproduce the output you meant?

Tuhu, no, I wasn't asking that you run the commands from tty1.

Now that you've added the ps and lsof commands to the umountroot script,
it should create the text file '/DEBUG' when the system is shut down.

Please attach the file /DEBUG.

The error message "mount: / is busy" is being printed by the mount
command in the umountroot script. The /DEBUG file should show us what
files were still open just before that mount command is executed. It's
likely that a process still has a file open for write.

Also, have you ever installed VMware Player (see previous messages in
this bug report) or any non-ubuntu software which may have altered the
order in which your init scripts are run?

Here is the DEBUG file.

I have no VMware player installed on my system, the non-Ubuntu applications that are installed in my system are Oracle Virtualbox, remastersys, and nvidia proprietary driver. But, I remember, last time, I installed a dialer and driver for Huawei ec156 USB modem (provided in the modem). But, it was not work, the make process of the driver was failed. Thus, I deleted the installed directory of the dialer manually.

And, here is something. The message "mount: / is busy" does not appear each time I power off the system from tty1 while the lightdm is off.

The /DEBUG file shows that modem-manager is still running, and at line
114 you can see that the init process has /var/log/upstart/modemmanager
open for write. That explains the "mount: / is busy" message.

I would think that modem-manager should have exited before umountroot is
run. Looking at /etc/init.d/sendsigs, it appears that processes
associated with upstart jobs (like modemmanager) aren't killed, and I
see no facility there for waiting on them to exit.

I'm not an ubuntu developer, but if you care to try an experiment, you
could try adding a sync command to line 64 of /etc/init.d/sendsigs .
This is just before a comment that starts 'Upstart jobs have their own'
and a sequence of code that collects the pids of upstart jobs into a
list of pids to ignore (i.e. pids not to kill). If modem-manager or
init are waiting on i/o to complete, a sync here might give
modem-manager time to exit before it gets put into the OMITPIDS list.

If you decide to change sendsigs, be sure to make a backup copy first:

sudo cp -p /etc/init.d/sendsigs ~

Other than that, I don't have any new ideas. Let's hope that Mathieu or
another one of the powers-that-be will now jump in with some suggestions
as to how to proceed.

I have put a sync command on the line 64 of /etc/init.d/sendsigs, Steven. But, the message "mount: / is busy" still appears. Hope there will be some suggestion from the others then. Thank you for your suggestions so far.

Tuhu, if you type the following three commands into a terminal window
what is printed after each command?

sudo stop network-manager
initctl status modemmanager
ps p $(pgrep modem-manager) 2>/dev/null

The third command should print something only if modem-manager is still
running. It shouldn't be, but your /DEBUG file showed that it was and
that's part of the problem. You'll want to restart network-manager by
typing:

sudo start network-manager

What network devices are active on your computer whenever the problem
occurs? Any mobile broadband devices at all? Have you looked at the
log file /var/log/upstart/modemmanager.log to see if there are any error
messages or warnings there?

Bug #974187 reports modem-manager taking a very long time to shut
down. That bug is unresolved and due to expire soon due to lack of
activity. You might want to subscribe to that bug and add a "me too".

I could not open the file of /dev/null, where the output of 'ps p $(pgrep modem-manager) 2>/dev/null' went, the error on gedit: "Could not open the file /dev/null. /dev/null is not a regular file". But, when I piped it to Desktop, like: "ps p $(pgrep modem-manager) 2>/root/Desktop" , the output showed an error: "ERROR: List of process IDs must follow p. ********* simple selection ********* ********* selection by list ********* ... and.. so-on...".

The message "mount: / is busy" appears no matter my Huawei ec156 mobile broadband is attached or not. Or, when wireless network is active or not.

There is no error in /var/log/upstart/modemmanager.log, as you can see in the atachment.

You didn't answer my question in #42 above.

Type "man null" to learn how /dev/null works.

The ps command produced an error message. Assuming that it wasn't
mis-typed, that means that the modem-manager process wasn't running,
which is the correct result. That's also consistent with last line in
the modemmanager.log that you attached.

I suspect that your problem could be that modemmanager simply hasn't had
time to complete the shutdown process before umountroot is run. Since
modemmanager is an upstart job, sendsigs doesn't kill it.

I've seen no evidence that your problem is a network-manager bug. If I
were you, I would open a new bug report using ubuntu-bug, probably
against upstart.

I have a mobile broadband device here. I have also removed "modemmanager " but the message still appears.
I'm thinking to open a new bug-report. Thanks.

For some reason, the main issue of this report now hit me. Root cannot be unmounted, because dhclient and dnsmasq is still running at this point (verified by running lsof in umountroot).

The order of my init scripts is still fine now, I have S35networking, S40umountfs and S60umountroot, so this is not the cause anymore.

Also when stopping networking manually, dhclient and dnsmasq are both still running.

I took the time to update to 12.10 (quantal), but still no change to this issue for me. My workaround is a "killall dnsmasq dhclient; sleep 2" in umountroot, but obviously, this doesn't even count as dirty hack, this is even more ugly.

I can confirm this bug. I have two systems running 12.10 and both experience this issue.

Killing dnsmasq and dhclient in the umountroot script results in a clean shutdown on both systems.

I have this bug in 12.10 and Network manager version 0.9.6.0-0ubuntu7
/sbin/dhclient and /usr/sbin/dnsmasq are not stopped and / cannot be unmounted

This bug is still affecting quantal as reported by Sebastien. The workaround #5 works for me.

I fixed it for me in upstart, modifying /etc/init/network-manager.conf. Patch attached.

This bug also affects Quantal, but I don't know how to mark the bug for this.

umount-fix.patch:

Best if it is
cat /var/run/sendsigs.omit.d/network-manager.dhclient-*

As if you have multiple ETH, this will not kill it.

I installed the Raring network-manager package (0.9.7.995+git201301311547.17123fc-0ubuntu1) in my Quantal install, and the bug seems gone.

Now I just have this displayed at shutdown :

nm-dispatcher.action: Caught signal 15, shutting down...
Rather than invoking init scripts through /etc/init.d, use the service utility ..........
... blabla...

But it's just a warning.

Bug still present in kubuntu 13.04, network manager 0.9.8.0

I upgraded to network manager 0.9.8.0, still staying in Quantal.

Bug still not present on my system.

@Sebastien : you should check the output of lsof (see comment #5).

I can confirm this problem still exists in 13.04 (raring). I'm using Daniel Eckl's patch in comment #49 to work around it, but I modified it to only kill dhclient. It appears that dnsmasq is being terminated properly in 13.04.

I still have this bug too. Can't we get rid of NetworkManager if no one wants to fix it?

This bug affects 14.04 as well and somehow tracked here https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1169614

2 years bug…