Ubuntu
linux package

[quantal] overlayfs over r/o NFS mount fails to overwrite existing files

Bug #1039402 reported by Reinhard Tartler on 2012-08-21

This bug affects 8 people

Affects		Status	Importance	Assigned to	Milestone
	linux (Ubuntu)	In Progress	Medium	Andy Whitcroft

Bug Description

This is a followup to bug #1038075

Testcase: (Kernel taken from http://people.canonical.com/~apw/lp1038075-quantal/)

$ uname -a
Linux faui49i 3.5.0-11-generic #11lp1038075v201208181342 SMP Sat Aug 18 12:56:20 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
$ modprobe overlayfs
$ mkdir /live/tmpfs /tmp/root
$ mount -t nfs -o ro 192.168.42.40:/srv/fai/nfsroot.quantal64 /mnt
$ stat /mnt/etc/resolv.conf
  File: `/mnt/etc/resolv.conf'
  Size: 206 Blocks: 8 IO Block: 1048576 regular file
Device: 1ch/28d Inode: 36044830 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2012-08-20 10:53:27.177775447 +0200
Modify: 2012-08-08 12:50:21.000000000 +0200
Change: 2012-08-16 04:06:12.317826670 +0200
Birth: -
$ mount -t overlayfs -o noatime,lowerdir=/mnt,upperdir=/live/overlayfs overlayfs /live/root
$ stat /live/root/etc/resolv.conf
  File: `/live/root/etc/resolv.conf'
  Size: 206 Blocks: 8 IO Block: 1048576 regular file
Device: 1ch/28d Inode: 36044830 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2012-08-20 10:53:27.177775447 +0200
Modify: 2012-08-08 12:50:21.000000000 +0200
Change: 2012-08-16 04:06:12.317826670 +0200
Birth: -
$ echo foo > /live/root/etc/resolv.conf
bash: /live/root/etc/resolv.conf: Permission denied

This permission error does not happen with a local filesystem but only with NFS.

However this seems to work:

$ rm /live/root/etc/resolv.conf
$ echo foo > /live/root/etc/resolv.conf
$ cat /live/root/etc/resolv.conf
foo
---
ApportVersion: 2.4-0ubuntu8
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/controlC0', '/dev/snd/hwC0D2', '/dev/snd/hwC0D3', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/pcmC0D3p', '/dev/snd/pcmC0D2c', '/dev/snd/pcmC0D7p', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
CurrentDmesg:
Error: command ['sh', '-c', 'dmesg | comm -13 --nocheck-order /var/log/dmesg -'] failed with exit code 1: comm: /var/log/dmesg: Permission denied
dmesg: write failed: Broken pipe
DistroRelease: Ubuntu 12.10
IwConfig:
eth0 no wireless extensions.

lo no wireless extensions.
MachineType: FUJITSU ESPRIMO P7935
Package: linux (not installed)
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: root=UUID=871b8166-1de3-42f9-845b-d3353e32beec ro quiet splash
ProcVersionSignature: Ubuntu 3.5.0-11.11lp1038075v201208181342-generic 3.5.2
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RfKill:

Tags: quantal
Uname: Linux 3.5.0-11-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: netgrp netgrp sbuild
WifiSyslog:

dmi.bios.date: 01/12/2010
dmi.bios.vendor: FUJITSU // Phoenix Technologies Ltd.
dmi.bios.version: 6.00 R1.20.2812.A2
dmi.board.name: D2812-A2
dmi.board.vendor: FUJITSU
dmi.board.version: S26361-D2812-A2
dmi.chassis.type: 6
dmi.chassis.vendor: FUJITSU
dmi.modalias: dmi:bvnFUJITSU//PhoenixTechnologiesLtd.:bvr6.00R1.20.2812.A2:bd01/12/2010:svnFUJITSU:pnESPRIMOP7935:pvr:rvnFUJITSU:rnD2812-A2:rvrS26361-D2812-A2:cvnFUJITSU:ct6:cvr:
dmi.product.name: ESPRIMO P7935
dmi.sys.vendor: FUJITSU

See original description

Tags:

Revision history for this message

Reinhard Tartler (siretart) wrote on 2012-08-21: AcpiTables.txt

AcpiTables.txt Edit (105.1 KiB, text/plain)

apport information

tags:	added: apport-collected quantal
description:	updated

Revision history for this message

Reinhard Tartler (siretart) wrote on 2012-08-21: AlsaInfo.txt

AlsaInfo.txt Edit (23.8 KiB, text/plain)

apport information

Revision history for this message

Reinhard Tartler (siretart) wrote on 2012-08-21: Lspci.txt

Lspci.txt Edit (14.0 KiB, text/plain)

apport information

Revision history for this message

Reinhard Tartler (siretart) wrote on 2012-08-21: Lsusb.txt

Lsusb.txt Edit (743 bytes, text/plain)

apport information

Revision history for this message

Reinhard Tartler (siretart) wrote on 2012-08-21: ProcCpuinfo.txt

ProcCpuinfo.txt Edit (3.1 KiB, text/plain)

apport information

Revision history for this message

Reinhard Tartler (siretart) wrote on 2012-08-21: ProcEnviron.txt

ProcEnviron.txt Edit (143 bytes, text/plain)

apport information

Revision history for this message

Reinhard Tartler (siretart) wrote on 2012-08-21: ProcInterrupts.txt

ProcInterrupts.txt Edit (2.8 KiB, text/plain)

apport information

Revision history for this message

Reinhard Tartler (siretart) wrote on 2012-08-21: ProcModules.txt

ProcModules.txt Edit (3.5 KiB, text/plain)

apport information

Revision history for this message

Reinhard Tartler (siretart) wrote on 2012-08-21: UdevDb.txt

UdevDb.txt Edit (135.7 KiB, text/plain)

apport information

Revision history for this message

Reinhard Tartler (siretart) wrote on 2012-08-21: UdevLog.txt

#10

UdevLog.txt Edit (316.2 KiB, text/plain)

apport information

description:

updated

Revision history for this message

Brad Figg (brad-figg) wrote on 2012-08-21: Missing required logs.

#11

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1039402

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status:	New → Incomplete

Revision history for this message

Reinhard Tartler (siretart) wrote on 2012-08-21:

#12

Brad, that's exactly what I did half an hour ago. I assume that your message (#11) was automated and did not really notice the information that I have posted. Therefore, I'm setting the status back to 'confirmed'. If there is any information still missing, please do tell me!

Changed in linux (Ubuntu):
status:	Incomplete → Confirmed

Joseph Salisbury (jsalisbury) on 2012-08-21

Changed in linux (Ubuntu):
importance:	Undecided → Medium
tags:	added: bot-stop-nagging

Revision history for this message

Andy Whitcroft (apw) wrote on 2012-08-21:

#13

This seems to be related to some of the inode information being invalid in the overlayfs inodes:

    $ ls -l
    ls: cannot access yyy: No such file or directory
    total 4
    -rw-rw-r-- 1 mru mru 6 Aug 17 16:56 xxx
    -????????? ? ? ? ? ? yyy

Investigating.

Changed in linux (Ubuntu):
status:	Confirmed → In Progress
assignee:	nobody → Andy Whitcroft (apw)

Revision history for this message

Andy Whitcroft (apw) wrote on 2012-08-23:

#14

Ok think I have found the underlying issue here. It seems its related to a lack of xattr support in the lower filesystem throwing out the copyup operations. I have hopefully fixed this up, could you test the kernels below (pick the newer dated one) and report back here:

http://people.canonical.com/~apw/lp1038075-quantal/

Thanks!

Revision history for this message

Reinhard Tartler (siretart) wrote on 2012-08-23:

#16

screenshot-bug1038075.png Edit (381.5 KiB, image/png)

Unfortunately, the new kernel doesn't solve the symptoms I'm experiencing. See the attached screenshot.

Revision history for this message

Reinhard Tartler (siretart) wrote on 2012-08-23:

#17

Summary from irc.

The proposed patch does fix the symptoms if the NFS mount is mounted with locking enabled (option 'lock', which is enabled by default and always with nfsv4).

However, live-boot mounts the nfs with "-o nolock" (and on our site with nfsv3, which can be forced with the mount option "-o vers=3") to not require a locking daemon. In this scenario, the symptom persists.

Revision history for this message

Jason Gerard DeRose (jderose) wrote on 2013-03-19:

#18

I can confirm the same behavior with NFSv3 and overlayfs on Raring.

Revision history for this message

Michał Sawicz (saviq) wrote on 2013-04-07:

#19

Can we please get an update on this? @apw, is this actually in progress?

Revision history for this message

Ryan Tandy (rtandy) wrote on 2013-07-03:

#20

This still affects linux 3.10.0-2.9 in saucy. In September apw posted a pair of patches related to this bug [1] but they don't seem to have made it past the RFC stage and neither one has been integrated (AFAICT) in the current overlayfs tree [2]. I couldn't find any more recent conversations about this.

[1] https://lkml.org/lkml/2012/9/6/378
[2] https://git.kernel.org/cgit/linux/kernel/git/mszeredi/vfs.git

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntulinux package

[quantal] overlayfs over r/o NFS mount fails to overwrite existing files

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntu
linux package