Data corruption during parallel file copying with interruptions

Bug #1543633 reported by Martin Steghöfer
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
eCryptfs
Fix Released
High
Unassigned

Bug Description

When performing a lot of I/O operations in an ecryptfs file system and interrupting those operations, the involved files can get corrupted, even if they are only supposed to be *read* (not written) during the operation.

During a parallel build ("make -j8" or "make -j4") of a big C++ project making use of precompiled headers, I realized that by interrupting the build (CTRL+C) I was able to corrupt the PCH files pretty reliably. Since both GCC and GNU Make have safety mechanisms in place to deal with interruptions (they both independently delete partially written output files when interrupted), I investigated the issue and got suspicious about ecryptfs and tried to reproduce the problem in a simple environment.

Find attached an example and 2 test runs of a Makefile that creates a big file and copies it several times. When executed in parallel ("make -j*") and interrupted at the right moments, files can get corrupted, as verified by the "md5sum" command. Note that the corrupted files are not output files that have been written only partially (GNU Make deletes them reliably), but instead *input* files of copy operations. They weren't supposed to be opened for writing at all (just for reading).

I'm not entirely sure, if ecryptfs is responsible, but none of the effects presented here could be reproduced without ecryptfs.

The parallel execution is important to increase the probability of a corruption. It also seems to be important to have slow I/O (HDD instead of SSD; small or deactivated caches) for the corruptions to occur. Even so, several test runs might be necessary to reproduce the problem.

Revision history for this message
Martin Steghöfer (martin-steghoefer) wrote :
Revision history for this message
Martin Steghöfer (martin-steghoefer) wrote :
Revision history for this message
Martin Steghöfer (martin-steghoefer) wrote :
Revision history for this message
Mikko Rantalainen (mira) wrote : Re: [Bug 1543633] Re: Data corruption during parallel file copying with interruptions

> ** Attachment added: "Test run of the Makefile. The files "file06.xd",
"file07.xd", "file09.xd", "file10.xd" and "file11.xd" get corrupted in a
phase where they are only used as *input* for copy operations."

Can you reproduce the problem using some another computer to rule RAM or
other hardware issues out?

Revision history for this message
Tyler Hicks (tyhicks) wrote :

Thanks for the clear bug report and reproducer. That goes a long way towards identifying a fix.

What kernel are you seeing this with?

As Mikko suggested, ruling out RAM errors is always a good idea with bugs such as this. I'll also try to reproduce the issue locally.

Revision history for this message
Tyler Hicks (tyhicks) wrote :

I'm unable to reproduce what you're seeing with Linus' current git tree HEAD (4.5.0-rc3+) on top of an ext4 lower filesystem in an amd64 KVM guest.

I'll need to know your CPU architecture, kernel version, your eCryptfs mount options (you can filter out the key sigs), and the lower filesystem type before I can look further into this. Thanks!

Changed in ecryptfs:
status: New → Incomplete
Revision history for this message
Martin Steghöfer (martin-steghoefer) wrote :

I haven't yet had the chance to test it on a different machine. Maybe tonight, maybe tomorrow... The RAM of the current machine seems fine, at least Memtest didn't report any problems (that was one of the first things I checked after seeing data corruptions). Of course there can be other hardware defects. However, I would expect data corruptions outside ecryptfs file systems and unexpected crashes, if there was a hardware problem. Anyway, I will try on a different machine, when I get the chance.

Sorry for not providing any version numbers earlier. Here they come. Find attached the detailed information. Here a short summary:
* CPU Architecture: amd64
* Kernel version: 4.2.0-27.32 (Ubuntu)
* Underlying file system: ext4

Revision history for this message
Martin Steghöfer (martin-steghoefer) wrote :

Find attached an example of a data corruption. Typically, one or more blocks of small integer multiples of 4KB in size are corrupted. Some corrupted blocks contain seemingly random data, others contain patterns. Neither the file size nor any meta information (permissions, ownership etc.) ever gets corrupted, just part of the content.

Revision history for this message
Tyler Hicks (tyhicks) wrote :

I have verified this bug in an up-to-date Ubuntu 15.10 amd64 KVM guest.

Changed in ecryptfs:
importance: Undecided → High
status: Incomplete → Confirmed
Revision history for this message
Tyler Hicks (tyhicks) wrote :

Here's an slightly improved Makefile to reproduce the issue. I can reliably reproduce the bug like so by interrupting the processes after I see at least one line printed *after* the "cp big.dat file99.xd" line scrolls by:

$ make -j100
...
cp big.dat file97.xd
cp big.dat file98.xd
cp big.dat file99.xd
cp file04.xd file04.xc
cp file22.xd file22.xc
cp file09.xd file09.xc
^C
...

Now to see if corruption is found:

$ make check
./file26.xd: FAILED
./file25.xd: FAILED
md5sum: WARNING: 2 computed checksums did NOT match
Makefile:24: recipe for target 'check' failed
make: *** [check] Error 1

This corruption only occurs in the eCryptfs page cache. What's being written to disk is valid data. This can be demonstrated by running `make check` after either executing `echo 1 | sudo tee /proc/sys/vm/drop_caches` or unmounting and then remounting the eCryptfs mount.

Revision history for this message
Tyler Hicks (tyhicks) wrote :

Hi Martin - I've had a few changes to look into this bug recently but I'm back to not being able to reproduce it. Can you still reliably reproduce the corruption? If so, would you be willing to boot into some test kernels to tell me if the issue is fixed?

Revision history for this message
Martin Steghöfer (martin-steghoefer) wrote :

I can still reproduce it under the original Kernel (Ubuntu Kernel 4.2.0-27.32), but I can no longer reproduce it in the two subsequent Ubuntu Kernel versions that are available in the Ubuntu repositories (4.2.0-30.36 and 4.2.0-34.39).

I can't tell, if the issue has been fixed between 4.2.0-27 and 4.2.0-30 or if it is just more difficult to reproduce the problem because the problem was somehow masked by another change.

Sure, I can boot into some more test kernels and try to reproduce the issue with them, if that helps. Just tell me, which ones. Do I need to compile them from source or can I use packages?

Revision history for this message
Mikko Rantalainen (mira) wrote :

On 30 Mar 2016 01:20, "Tyler Hicks" <email address hidden> wrote:
>
> Hi Martin - I've had a few changes to look into this bug recently but
> I'm back to not being able to reproduce it. Can you still reliably
> reproduce the corruption? If so, would you be willing to boot into some
> test kernels to tell me if the issue is fixed?

The bug appears to be caused by some kind of race condition. Perhaps it's
easier to reproduce it with Ubuntu lowlatency kernels or even with RT
kernels. If I remember correctly, Ubuntu doesn't build/distribute *-rt
kernels anymore, though.

Revision history for this message
Tyler Hicks (tyhicks) wrote :

Thanks for pointing out that a change between -27 and -30 has affected the behavior that you originally reported. I have verified that, as well. Since there were no eCryptfs changes between those two kernel versions, it makes me uncomfortable to simply mark this bug as fix-released.

I'm currently bisecting the changes between the two kernel versions to identify the specific patch that changed the behavior. That will hopefully shore up our confidence that the issue is truly fixed.

Changed in ecryptfs:
status: Confirmed → In Progress
Revision history for this message
Tyler Hicks (tyhicks) wrote :

Here's the fix identified, through bisection, as the first commit that fixes the corruption issues:

https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/wily/commit/?id=69aa9cd48005233cc5739c8a7b427e636b0f5e19

commit 69aa9cd48005233cc5739c8a7b427e636b0f5e19
Author: Peter Zijlstra <email address hidden>
Date: Tue Dec 1 14:04:04 2015 +0100

    sched/wait: Fix signal handling in bit wait helpers

    BugLink: http://bugs.launchpad.net/bugs/1536370

    commit 68985633bccb6066bf1803e316fbc6c1f5b796d6 upstream.

    Vladimir reported getting RCU stall warnings and bisected it back to
    commit:

      743162013d40 ("sched: Remove proliferation of wait_on_bit() action functions")

    That commit inadvertently reversed the calls to schedule() and signal_pending(),
    thereby not handling the case where the signal receives while we sleep.

    ...

Revision history for this message
Tyler Hicks (tyhicks) wrote :

I've not been able to pinpoint exactly how that change fixes this bug but it does not surprise me since the wait_on_bit() functions are used in some areas of the fs/ and mm/ kernel code.

I'm going to mark this bug as fix released for now unless anyone strongly feels like the commit mentioned above is only papering over a larger bug in eCryptfs.

Changed in ecryptfs:
status: In Progress → Fix Released
To post a comment you must log in.