overlayfs mounts as R/O over idmapped mount

Bug #2009065 reported by Aleksandr Mikhalitsyn
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Medium
Unassigned
Jammy
Invalid
Undecided
Unassigned
Kinetic
Fix Released
Medium
Unassigned
linux-hwe-5.19 (Ubuntu)
Jammy
Fix Released
Medium
Unassigned
Kinetic
Invalid
Undecided
Unassigned

Bug Description

Reproducer:

create unprivileged LXC container
$ lxc launch ubuntu:22.04 idmap-test
$ lxc exec idmap-test bash

check that root is idmapped by:
$ cat /proc/self/mountinfo | grep idmap

780 675 8:1 /var/snap/lxd/common/lxd/storage-pools/default/containers/idmap-test/rootfs / rw,relatime,idmapped shared:323 master:319 - ext4 /dev/sda1 rw,discard,errors=remount-ro

$ mkdir {work,upper,lower,ovl}
$ mount -t overlay overlay -o lowerdir=lower,upperdir=upper,workdir=work ovl
$ touch ovl/test
touch: cannot touch 'ovl/test': Read-only file system

The problem is in __vfs_removexattr_noperm() function that called from ovl_workdir_create().

In dmesg I can see an error:
overlayfs: failed to create directory work/work (errno: 1); mounting read-only

Reproducible on:
# uname -a
Linux ubuntu 5.19.0-32-generic #33~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Jan 30 17:03:34 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.1 LTS"

Suspicious commit:
https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/commit/fs/overlayfs/overlayfs.h?h=Ubuntu-hwe-5.19-5.19.0-32.33_22.04.1&id=3418435738af5730918fafbdfe2905a98ce2ef05

Chunk:
+ inode_lock(inode);
+ err = __vfs_setxattr_noperm(&init_user_ns, dentry, name, value, size, flags);
+ inode_unlock(inode);

should be:
+ inode_lock(inode);
+ err = __vfs_setxattr_noperm(ovl_upper_mnt_userns(ofs), dentry, name, value, size, flags);
+ inode_unlock(inode);

I'll send a patch soon.

affects: linux (Ubuntu) → linux-hwe-5.19 (Ubuntu)
Changed in linux-hwe-5.19 (Ubuntu):
assignee: nobody → Aleksandr Mikhalitsyn (mihalicyn)
affects: linux-hwe-5.19 (Ubuntu) → linux (Ubuntu)
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 2009065

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Aleksandr Mikhalitsyn (mihalicyn) wrote :
Revision history for this message
Aleksandr Mikhalitsyn (mihalicyn) wrote :
tags: added: patch
Revision history for this message
Aleksandr Mikhalitsyn (mihalicyn) wrote :

It makes no sense to run apport-collect 2009065 in this case :)

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
affects: linux (Ubuntu) → linux-hwe-5.19 (Ubuntu)
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 2009065

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Stefan Bader (smb)
Changed in linux-hwe-5.19 (Ubuntu Kinetic):
status: New → Invalid
Changed in linux (Ubuntu Jammy):
status: New → Invalid
Changed in linux (Ubuntu Kinetic):
importance: Undecided → Medium
status: New → In Progress
Changed in linux-hwe-5.19 (Ubuntu Jammy):
importance: Undecided → Medium
Revision history for this message
Stefan Bader (smb) wrote :

This might go a little bumpy. Normally the hwe-5.19 kernel is updated when the kinetic one is. But states here likely do not follow automatically.

Changed in linux-hwe-5.19 (Ubuntu Jammy):
status: New → In Progress
Changed in linux-hwe-5.19 (Ubuntu):
status: Confirmed → Invalid
Changed in linux (Ubuntu):
status: Incomplete → Fix Committed
importance: Undecided → Medium
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-oem-6.1/6.1.0-1008.8 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-jammy-linux-oem-6.1 verification-needed-jammy
Changed in linux (Ubuntu Kinetic):
status: In Progress → Fix Committed
no longer affects: linux-hwe-5.19 (Ubuntu)
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 6.2.0-18.18

---------------
linux (6.2.0-18.18) lunar; urgency=medium

  * lunar/linux: 6.2.0-18.18 -proposed tracker (LP: #2011750)

  * lunar/linux 6.2 fails to boot on arm64 (LP: #2011748)
    - SAUCE: Revert "efi: random: fix NULL-deref when refreshing seed"
    - SAUCE: Revert "efi: random: refresh non-volatile random seed when RNG is
      initialized"

 -- Andrea Righi <email address hidden> Wed, 15 Mar 2023 23:54:18 +0100

Changed in linux (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/5.19.0-40.41 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-kinetic' to 'verification-done-kinetic'. If the problem still exists, change the tag 'verification-needed-kinetic' to 'verification-failed-kinetic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-kinetic-linux verification-needed-kinetic
Revision history for this message
Gilles Schintgen (shigi) wrote :

Hi, I do think I'm affected by the same issue. Is it still helpful to test the 5.19.0-40.41 kernel as proposed by the kernel bot in the previous message? I don't want to mess up my server, but if it helps getting this fixed faster for 22.04, I'm willing to install the kernel from -proposed.

Currently I'm on 22.04 with the following kernel:
linux-hwe-5.19-headers-5.19.0-38/jammy-security,jammy-updates,now 5.19.0-38.39~22.04.1 all [installed,automatic]

The issue I'm having is with LXD on btrfs when trying to use docker with overlay2 in an unprivileged container.

In the container:
# docker run -it --rm busybox
docker: Error response from daemon: mkdir /var/lib/docker/overlay2/551bd2f3c71ebb9a3d672ac5993d72aea9d58035c0283708e503fc695a131c42-init/merged/proc: read-only file system.
See 'docker run --help'.

Host dmesg:
[ 6344.916322] overlayfs: failed to create directory /var/lib/docker/overlay2/551bd2f3c71ebb9a3d672ac5993d72aea9d58035c0283708e503fc695a131c42-init/work/work (errno: 1); mounting read-only
[ 6344.916340] overlayfs: fs on '/var/lib/docker/overlay2/l/QCM5T4AYJGCEMUPJJKGRKLPDGO' does not support file handles, falling back to xino=off.
[ 6344.916356] overlayfs: POSIX ACLs are not yet supported with idmapped layers, mounting without ACL support.

It seems to be one long sentence, split up in a misleading way. At first I read it as "mounting read-only", but later noticed that it actually states "mounting read-only fs on ..." as if the filesystem contained in the image were read-only by itself, which I'm not sure of.

Revision history for this message
Aleksandr Mikhalitsyn (mihalicyn) wrote :

You are right. That's it. Please try -proposed kernel and write about your testing results.
Feel free to reach me on the forum/IRC if you have any questions.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (58.2 KiB)

This bug was fixed in the package linux-hwe-5.19 - 5.19.0-40.41~22.04.1

---------------
linux-hwe-5.19 (5.19.0-40.41~22.04.1) jammy; urgency=medium

  * jammy/linux-hwe-5.19: 5.19.0-40.41~22.04.1 -proposed tracker (LP: #2011875)

  [ Ubuntu: 5.19.0-40.41 ]

  * kinetic/linux: 5.19.0-40.41 -proposed tracker (LP: #2012668)
  * CVE-2023-0468
    - io_uring: fix tw losing poll events
    - io_uring: make poll refs more robust
  * Regression in ext4 during online resize (LP: #2003816)
    - ext4: fix bad checksum after online resize
    - ext4: fix corruption when online resizing a 1K bigalloc fs
    - ext4: fix corrupt backup group descriptors after online resize

  [ Ubuntu: 5.19.0-39.40 ]

  * kinetic/linux: 5.19.0-39.40 -proposed tracker (LP: #2011876)
  * CVE-2023-26545
    - net: mpls: fix stale pointer if allocation fails during device rename
  * CVE-2023-1032
    - net: avoid double iput when sock_alloc_file fails
  * CVE-2022-41218 // CVE-2022-41218 is assigned to those bugs above.
    - media: dvb-core: Fix UAF due to refcount races at releasing
  * CVE-2023-1281
    - net/sched: tcindex: update imperfect hash filters respecting rcu
  * linux: CONFIG_SERIAL_8250_MID=y (LP: #2009283)
    - [Config] enable CONFIG_SERIAL_8250_MID=y
  * apply the latest TDX attestation driver from Intel (LP: #2009437)
    - Revert "UBUNTU: SAUCE: selftests: tdx: Test GetReport TDX attestation
      feature"
    - Revert "UBUNTU: SAUCE: x86/tdx: Add TDX Guest attestation interface driver"
    - x86/tdx: Add a wrapper to get TDREPORT0 from the TDX Module
    - virt: Add TDX guest driver
    - selftests/tdx: Test TDX attestation GetReport support
    - [Config] enable TDX attestation driver as module by default
  * cpufreq: intel_pstate: Update Balance performance EPP for Sapphire Rapids
    (LP: #2008519)
    - cpufreq: intel_pstate: Adjust balance_performance EPP for Sapphire Rapids
  * Fail to output sound to external monitor which connects via docking station
    (LP: #2009024)
    - [Config] Enable CONFIG_SND_HDA_INTEL_HDMI_SILENT_STREAM
  * Fix mediatek wifi driver crash when loading wrong SAR table (LP: #2009118)
    - wifi: mt76: mt7921: fix error code of return in mt7921_acpi_read
  * CVE-2022-47929
    - net: sched: disallow noqueue for qdisc classes
  * Intel graphic driver is not probing[8086:468b] (LP: #1990242)
    - drm/i915: Add new ADL-S pci id
  * Some QHD panels fail to refresh when PSR2 enabled (LP: #2009014)
    - drm/i915/psr: Use full update In case of area calculation fails
    - drm/i915: Ensure damage clip area is within pipe area
    - SAUCE: drm/i915/psr: Use calculated io and fast wake lines
  * overlayfs mounts as R/O over idmapped mount (LP: #2009065)
    - SAUCE: overlayfs: handle idmapped mounts in ovl_do_(set|remove)xattr
  * Kinetic update: upstream stable patchset 2023-02-28 (LP: #2008832)
    - usb: musb: remove extra check in musb_gadget_vbus_draw
    - arm64: dts: qcom: ipq6018-cp01-c1: use BLSPI1 pins
    - arm64: dts: qcom: sm8250-sony-xperia-edo: fix touchscreen bias-disable
    - arm64: dts: qcom: msm8996: Add MSM8996 Pro support
    - arm64: dts: qcom: msm8996: fix supported-hw in cpufreq OPP tables
    - arm64: dts:...

Changed in linux-hwe-5.19 (Ubuntu Jammy):
status: In Progress → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (58.1 KiB)

This bug was fixed in the package linux - 5.19.0-40.41

---------------
linux (5.19.0-40.41) kinetic; urgency=medium

  * kinetic/linux: 5.19.0-40.41 -proposed tracker (LP: #2012668)

  * CVE-2023-0468
    - io_uring: fix tw losing poll events
    - io_uring: make poll refs more robust

  * Regression in ext4 during online resize (LP: #2003816)
    - ext4: fix bad checksum after online resize
    - ext4: fix corruption when online resizing a 1K bigalloc fs
    - ext4: fix corrupt backup group descriptors after online resize

linux (5.19.0-39.40) kinetic; urgency=medium

  * kinetic/linux: 5.19.0-39.40 -proposed tracker (LP: #2011876)

  * CVE-2023-26545
    - net: mpls: fix stale pointer if allocation fails during device rename

  * CVE-2023-1032
    - net: avoid double iput when sock_alloc_file fails

  * CVE-2022-41218 // CVE-2022-41218 is assigned to those bugs above.
    - media: dvb-core: Fix UAF due to refcount races at releasing

  * CVE-2023-1281
    - net/sched: tcindex: update imperfect hash filters respecting rcu

  * linux: CONFIG_SERIAL_8250_MID=y (LP: #2009283)
    - [Config] enable CONFIG_SERIAL_8250_MID=y

  * apply the latest TDX attestation driver from Intel (LP: #2009437)
    - Revert "UBUNTU: SAUCE: selftests: tdx: Test GetReport TDX attestation
      feature"
    - Revert "UBUNTU: SAUCE: x86/tdx: Add TDX Guest attestation interface driver"
    - x86/tdx: Add a wrapper to get TDREPORT0 from the TDX Module
    - virt: Add TDX guest driver
    - selftests/tdx: Test TDX attestation GetReport support
    - [Config] enable TDX attestation driver as module by default

  * cpufreq: intel_pstate: Update Balance performance EPP for Sapphire Rapids
    (LP: #2008519)
    - cpufreq: intel_pstate: Adjust balance_performance EPP for Sapphire Rapids

  * Fail to output sound to external monitor which connects via docking station
    (LP: #2009024)
    - [Config] Enable CONFIG_SND_HDA_INTEL_HDMI_SILENT_STREAM

  * Fix mediatek wifi driver crash when loading wrong SAR table (LP: #2009118)
    - wifi: mt76: mt7921: fix error code of return in mt7921_acpi_read

  * CVE-2022-47929
    - net: sched: disallow noqueue for qdisc classes

  * Intel graphic driver is not probing[8086:468b] (LP: #1990242)
    - drm/i915: Add new ADL-S pci id

  * Some QHD panels fail to refresh when PSR2 enabled (LP: #2009014)
    - drm/i915/psr: Use full update In case of area calculation fails
    - drm/i915: Ensure damage clip area is within pipe area
    - SAUCE: drm/i915/psr: Use calculated io and fast wake lines

  * overlayfs mounts as R/O over idmapped mount (LP: #2009065)
    - SAUCE: overlayfs: handle idmapped mounts in ovl_do_(set|remove)xattr

  * Kinetic update: upstream stable patchset 2023-02-28 (LP: #2008832)
    - usb: musb: remove extra check in musb_gadget_vbus_draw
    - arm64: dts: qcom: ipq6018-cp01-c1: use BLSPI1 pins
    - arm64: dts: qcom: sm8250-sony-xperia-edo: fix touchscreen bias-disable
    - arm64: dts: qcom: msm8996: Add MSM8996 Pro support
    - arm64: dts: qcom: msm8996: fix supported-hw in cpufreq OPP tables
    - arm64: dts: qcom: msm8996: fix GPU OPP table
    - ARM: dts: qcom: apq8064: fix coresight compatible
    - arm64: dts:...

Changed in linux (Ubuntu Kinetic):
status: Fix Committed → Fix Released
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-ibm/5.19.0-1021.23 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-kinetic' to 'verification-done-kinetic'. If the problem still exists, change the tag 'verification-needed-kinetic' to 'verification-failed-kinetic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-kinetic-linux-ibm
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-aws/5.19.0-1025.26 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-kinetic' to 'verification-done-kinetic'. If the problem still exists, change the tag 'verification-needed-kinetic' to 'verification-failed-kinetic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-kinetic-linux-aws
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-azure/5.19.0-1026.29 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-kinetic' to 'verification-done-kinetic'. If the problem still exists, change the tag 'verification-needed-kinetic' to 'verification-failed-kinetic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-kinetic-linux-azure
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-nvidia-5.19/5.19.0-1010.10 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-jammy-linux-nvidia-5.19
tags: added: fixed-linux-oem-6.5
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.