btrfs: file write crashes with false ENOSPC during snapshot creation since kernel 4.4 - fix available

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1584052

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

There is no matching report in /var/crash. The bug is already diagnosed and fixed upstream, and may be reproduced with the information provided in https://patchwork.kernel.org/patch/7967161/

The commit in mainline is:
4da2e26 btrfs: Continue write in case of can_not_nocow

I built a Xenial test kernel with a pick of 4da2e26. The test kernel can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1584052/

Can you test this kernel and see if it resolves this bug?

I have now installed the kernel image on a server which previously experienced MySQL crashes every 1 to 2 hours. The server uses MySQL for an Icinga2 monitoring database (so it's frequently writing) and is configured to create btrfs snapshots every 5 minutes.

# cat /proc/version
Linux version 4.4.0-22-generic (root@gomeisa) (gcc version 5.3.1 20160413 (Ubuntu 5.3.1-14ubuntu2) ) #40~lp1584052 SMP Fri May 20 14:34:33 UTC 2016

I'll let it run for 24 hours and report back.

The Xenial test kernel with a pick of 4da2e26 resolves the bug, no negative side effects were encountered.

With the Xenial release kernel, there were 18 MySQL server crashes in a 24 hour period due to false ENOSPC indications. With the test kernel, not a single crash occured in a 24 hour period.

Additionally, I've checked with a script derived from https://patchwork.kernel.org/patch/7967161/ (see attachment), which creates a temporary btrfs test image in a plain file. The script provoked ENOSPC errors with the release kernel on most runs (sometimes it missed, presumably due to timing conditions). The script could not provoke a single ENOSPC condition with the test kernel.

Another 24 error-free hours on the server with the test kernel running the MySQL/Icinga2 combination and creating btrfs snapshots every 5 minutes.

I'd like to add that this server, which previously experienced frequent false ENOSPC conditions with the Xenial release kernel, used a fairly plain btrfs configuration at all times:
- The '-o nodatacow' mount option was *not* used (in contrast to the test script).
- Disabling copy-on-write (CoW) on a per-file basis (using chattr +C) was *not* used.

# mount | grep /volumes/01
/dev/sda2 on /volumes/01 type btrfs (rw,noatime,nodiratime,ssd,space_cache,subvolid=5,subvol=/)

# lsattr /var/lib/mysql/ibdata1
---------------- /var/lib/mysql/ibdata1

That is great news. I'll submit an SRU request for Xenial.

Hope the SRU will be accepted quickly. Should be the case, since this issue causes loss of user data as applications will have no reason to retry files writes under out of space conditions.

Thanks for the quick and professional cooperation!

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Verified with the test script of #5. Kernel in -proposed solves the problem.

Steps used:

1. Verified that release kernel 4.4.0-24-generic still had the problem.

2. Verified that kernel 4.4.0.25 from -proposed contains the resolution:

# apt-get install -s -t xenial-proposed 'linux-headers-4.4.0.25$' 'linux-headers-4.4.0.25-generic$' 'linux-image-4.4.0.25-generic$' 'linux-image-extra-4.4.0.25-generic$'

# reboot

# cat /proc/version
Linux version 4.4.0-25-generic (buildd@lgw01-05) (gcc version 5.3.1 20160413 (Ubuntu 5.3.1-14ubuntu2.1) ) #44-Ubuntu SMP Fri Jun 10 18:19:48 UTC 2016

# ./test-btrfs-write-while-snapshotting.sh
[problem did not show up on 10 consecutive runs]

Problem solved, changing tag to 'verification-done-xenial'.

This bug was fixed in the package linux - 4.4.0-25.44

---------------
linux (4.4.0-25.44) xenial; urgency=low

  [ Kamal Mostafa ]

  * Release Tracking Bug
    - LP: #1591289

  * Xenial update to v4.4.13 stable release (LP: #1590455)
    - MIPS64: R6: R2 emulation bugfix
    - MIPS: math-emu: Fix jalr emulation when rd == $0
    - MIPS: MSA: Fix a link error on `_init_msa_upper' with older GCC
    - MIPS: Don't unwind to user mode with EVA
    - MIPS: Avoid using unwind_stack() with usermode
    - MIPS: Fix siginfo.h to use strict posix types
    - MIPS: Fix uapi include in exported asm/siginfo.h
    - MIPS: Fix watchpoint restoration
    - MIPS: Flush highmem pages in __flush_dcache_page
    - MIPS: Handle highmem pages in __update_cache
    - MIPS: Sync icache & dcache in set_pte_at
    - MIPS: ath79: make bootconsole wait for both THRE and TEMT
    - MIPS: Reserve nosave data for hibernation
    - MIPS: Loongson-3: Reserve 32MB for RS780E integrated GPU
    - MIPS: Use copy_s.fmt rather than copy_u.fmt
    - MIPS: Fix MSA ld_*/st_* asm macros to use PTR_ADDU
    - MIPS: Prevent "restoration" of MSA context in non-MSA kernels
    - MIPS: Disable preemption during prctl(PR_SET_FP_MODE, ...)
    - MIPS: ptrace: Fix FP context restoration FCSR regression
    - MIPS: ptrace: Prevent writes to read-only FCSR bits
    - MIPS: Fix sigreturn via VDSO on microMIPS kernel
    - MIPS: Build microMIPS VDSO for microMIPS kernels
    - MIPS: lib: Mark intrinsics notrace
    - MIPS: VDSO: Build with `-fno-strict-aliasing'
    - affs: fix remount failure when there are no options changed
    - ASoC: ak4642: Enable cache usage to fix crashes on resume
    - Input: uinput - handle compat ioctl for UI_SET_PHYS
    - ARM: mvebu: fix GPIO config on the Linksys boards
    - ARM: dts: at91: fix typo in sama5d2 PIN_PD24 description
    - ARM: dts: exynos: Add interrupt line to MAX8997 PMIC on exynos4210-trats
    - ARM: dts: imx35: restore existing used clock enumeration
    - ath9k: Add a module parameter to invert LED polarity.
    - ath9k: Fix LED polarity for some Mini PCI AR9220 MB92 cards.
    - ath10k: fix debugfs pktlog_filter write
    - ath10k: fix firmware assert in monitor mode
    - ath10k: fix rx_channel during hw reconfigure
    - ath10k: fix kernel panic, move arvifs list head init before htt init
    - ath5k: Change led pin configuration for compaq c700 laptop
    - hwrng: exynos - Fix unbalanced PM runtime put on timeout error path
    - rtlwifi: rtl8723be: Add antenna select module parameter
    - rtlwifi: btcoexist: Implement antenna selection
    - rtlwifi: Fix logic error in enter/exit power-save mode
    - rtlwifi: pci: use dev_kfree_skb_irq instead of kfree_skb in
      rtl_pci_reset_trx_ring
    - aacraid: Relinquish CPU during timeout wait
    - aacraid: Fix for aac_command_thread hang
    - aacraid: Fix for KDUMP driver hang
    - hwmon: (ads7828) Enable internal reference
    - mfd: intel-lpss: Save register context on suspend
    - mfd: intel_soc_pmic_core: Terminate panel control GPIO lookup table
      correctly
    - PM / Runtime: Fix error path in pm_runtime_force_resume()
    - cpuidle: Indicate when a device has been unregiste...

This bug was fixed in the package linux - 4.4.0-28.47

---------------
linux (4.4.0-28.47) xenial; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1595874

  * Linux netfilter local privilege escalation issues (LP: #1595350)
    - netfilter: x_tables: don't move to non-existent next rule
    - netfilter: x_tables: validate targets of jumps
    - netfilter: x_tables: add and use xt_check_entry_offsets
    - netfilter: x_tables: kill check_entry helper
    - netfilter: x_tables: assert minimum target size
    - netfilter: x_tables: add compat version of xt_check_entry_offsets
    - netfilter: x_tables: check standard target size too
    - netfilter: x_tables: check for bogus target offset
    - netfilter: x_tables: validate all offsets and sizes in a rule
    - netfilter: x_tables: don't reject valid target size on some architectures
    - netfilter: arp_tables: simplify translate_compat_table args
    - netfilter: ip_tables: simplify translate_compat_table args
    - netfilter: ip6_tables: simplify translate_compat_table args
    - netfilter: x_tables: xt_compat_match_from_user doesn't need a retval
    - netfilter: x_tables: do compat validation via translate_table
    - netfilter: x_tables: introduce and use xt_copy_counters_from_user

  * Linux netfilter IPT_SO_SET_REPLACE memory corruption (LP: #1555338)
    - netfilter: x_tables: validate e->target_offset early
    - netfilter: x_tables: make sure e->next_offset covers remaining blob size
    - netfilter: x_tables: fix unconditional helper

linux (4.4.0-27.46) xenial; urgency=low

  [ Kamal Mostafa ]

  * Release Tracking Bug
    - LP: #1594906

  * Support Edge Gateway's Bluetooth LED (LP: #1512999)
    - Revert "UBUNTU: SAUCE: Bluetooth: Support for LED on Marvell modules"

linux (4.4.0-26.45) xenial; urgency=low

  [ Kamal Mostafa ]

  * Release Tracking Bug
    - LP: #1594442

  * linux: Implement secure boot state variables (LP: #1593075)
    - SAUCE: UEFI: Add secure boot and MOK SB State disabled sysctl

  * failures building userspace packages that include ethtool.h (LP: #1592930)
    - ethtool.h: define INT_MAX for userland

linux (4.4.0-25.44) xenial; urgency=low

  [ Kamal Mostafa ]

  * Release Tracking Bug
    - LP: #1591289

  * Xenial update to v4.4.13 stable release (LP: #1590455)
    - MIPS64: R6: R2 emulation bugfix
    - MIPS: math-emu: Fix jalr emulation when rd == $0
    - MIPS: MSA: Fix a link error on `_init_msa_upper' with older GCC
    - MIPS: Don't unwind to user mode with EVA
    - MIPS: Avoid using unwind_stack() with usermode
    - MIPS: Fix siginfo.h to use strict posix types
    - MIPS: Fix uapi include in exported asm/siginfo.h
    - MIPS: Fix watchpoint restoration
    - MIPS: Flush highmem pages in __flush_dcache_page
    - MIPS: Handle highmem pages in __update_cache
    - MIPS: Sync icache & dcache in set_pte_at
    - MIPS: ath79: make bootconsole wait for both THRE and TEMT
    - MIPS: Reserve nosave data for hibernation
    - MIPS: Loongson-3: Reserve 32MB for RS780E integrated GPU
    - MIPS: Use copy_s.fmt rather than copy_u.fmt
    - MIPS: Fix MSA ld_*/st_* asm macros to use PTR_ADDU
    - MIPS: Prevent "restoration" of MSA c...