kernel panic when running Python test suite on ecryptfs

Note that the -j 50 is crucial too. Without that, no panic.

This bug is missing log files that will aid in dianosing the problem. From a terminal window please run:

apport-collect 813146

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

Confirmed on 3.0+

Thank you for taking the time to file a bug report on this issue.

However, given the number of bugs that the Kernel Team receives during any development cycle it is impossible for us to review them all. Therefore, we occasionally resort to using automated bots to request further testing. This is such a request.

We have noted that there is a newer version of the development kernel currently in the release pocket than the one you tested when this issue was found. Please test again with the newer kernel and indicate in the bug if this issue still exists or not.

If the bug still exists, change the bug status from Incomplete to Confirmed. If the bug no longer exists, change the bug status from Incomplete to Fix Released.

Thank you for your help.

Just got this
My oops trace is at https://bugs.launchpad.net/ubuntu/+source/linux/+bug/850253
linux-image-3.0.0-11-generic 3.0.0-11.18

System was performing a large update through update manager, 300mb or so of deb files. This would follow the symptoms of the Python testing where heavy hard drive use could be the cause.

@Barry Warsaw
Is this on a SSD or HDD?

On Sep 14, 2011, at 06:49 PM, Guy Taylor wrote:

>@Barry Warsaw
>Is this on a SSD or HDD?

HDD

After looking into it more I have moved my bug report away and collected that set around https://bugs.launchpad.net/ubuntu/+source/linux/+bug/836513
Could someone more familiar with ecryptfs please check to see if this and 836513 are related.

I am seeing the same thing in different circumstances: while developing Launchpad, running "utilities/start-dev-soyuz.sh" and then "make run_all", I've seen the similar OOPS like in comment #23 twice now (i.e. exception in decrypt_scatterlist, vs. the OOPS from bug 836513 which happens in encrypt_scatterlist).

The worst thing is that system seems to remain usable (except that gnome-screensaver locks you out of the system until you kill it from a virtual console or ssh), but after you reboot, your changes/new files created after the panic are lost.

FWIW, my dmesg has following lines (more of them, but just at different times) on boot, but those could be the old "empty files start appearing" bug in ecryptfs or something different:

[ 68.145110] Valid eCryptfs headers not found in file header region or xattr region
[ 68.145113] Either the lower file is not in a valid eCryptfs format, or the key could not be retrieved. Plaintext passthrough mode is not enabled; returning -EIO

Thank you for taking the time to file a bug report on this issue.

However, given the number of bugs that the Kernel Team receives during any development cycle it is impossible for us to review them all. Therefore, we occasionally resort to using automated bots to request further testing. This is such a request.

We have noted that there is a newer version of the development kernel than the one you last tested when this issue was found. Please test again with the newer kernel and indicate in the bug if this issue still exists or not.

If the bug still exists, change the bug status from Incomplete to Confirmed. If the bug no longer exists, change the bug status from Incomplete to Fix Released.

Thank you for your help, we really do appreciate it.

I confirmed exactly because I am seeing it in the latest kernel version "Package: linux-image-3.0.0-12-generic 3.0.0-12.20" as indicated in the duplicate bug report I filed. I am re-confirming hoping the lpltk bot doesn't mark it as incomplete again.

Added tasks for Oneiric and Precise. This seems relatively straightforward to reproduce and as such should probably be triaged, fixed, and SRU'd to 11.10.

@danilo

Would it be possible for you to test the latest upstream kernel? It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the release candidate kernel versus the daily build. Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text. Please let us know your results.

Thanks in advance.

@Barry

Would it be possible to get a screen shot of your panic? This may be a duplicate of bug 836513

@Barry

It would also be helpful if you could test the mainline kernel as well. That will tell us if this is already fixed upstream.

Was this an issue in prior releases, or is this an Oneiric regression?

Here's a screen shot (crappy old iphone :) of the kernel panic. Yes, it's an Oneiric regression and I'll try to run it again on the mainline kernel.

Nice too that you can still ssh in and do a clean reboot. Thanks for that hint.

$ uname -a
Linux limelight 3.0.4-030004-generic #201108301138 SMP Tue Aug 30 11:42:30 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

Confirmed on Oneiric *not* to kernel panic. Full Python test suite completed successfully (well, with an unrelated upstream unit test failure :).

I just got it with amd64 kernel from http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.1-rc9-oneiric. I tried to take a screenshot with my phone, but was too late (and Ctrl-Alt-F1 won't bring me back to the virtual console) and I only have a crappy overexposed photo. However, dmesg output from http://pastebin.ubuntu.com/711741/ matches exactly what was on the screen for 10-15 seconds before it got me back to the UI.

$ uname -a
Linux bljubuntu 3.1.0-0301rc9-generic #201110050905 SMP Wed Oct 5 09:15:03 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

FWIW, bug 836513 does have a similar traceback (as I noted above in comment #28), but it's in encrypt_scatterlist, while our tracebacks are in decrypt_scatterlist.

Also, since there are plenty more duplicates now, details of my system and how I am triggering the bug are in bug 873199. Only my /home is encrypted. Off to reboot again.

This is the fix I've written against the 3.1-rc10 kernel. I plan on testing the error paths a little more and sending it to the eCryptfs and linux-fsdevel lists for peer review before committing it upstream.

It doesn't apply cleanly to the oneiric kernel sources (due to bf6c7f6c7bd0ea779757d35b5fdc9f9157f056b3), but it is a trivial cleanup when that time comes.

The attachment "[PATCH] eCryptfs: Prevent file create race condition" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-sponsors please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

I'd be happy to test the above patch out, but I lack the time to compile my own kernel. If there's a deb package, I'd happily stress test it to see if I can hit the problem again (or uncover any other problems).

3.0.0-13-generic-pae affected too:
Oct 26 22:28:10 fqc-book kernel: [326923.537027] ------------[ cut here ]------------
Oct 26 22:28:10 fqc-book kernel: [326923.537078] kernel BUG at /build/buildd/linux-3.0.0/fs/ecryptfs/crypto.c:348!
Oct 26 22:28:10 fqc-book kernel: [326923.537139] invalid opcode: 0000 [#1] SMP
Oct 26 22:28:10 fqc-book kernel: [326923.537176] Modules linked in: nls_utf8 isofs snd_seq_dummy snd_usb_audio snd_usbmidi_lib nls_iso8859_1 nls_cp437
vfat fat hidp bnep rfcomm pci_stub vboxpci vboxnetadp vboxnetflt vboxdrv parport_pc ppdev binfmt_misc dm_crypt snd_hda_codec_hdmi snd_hda_codec_idt hp_
wmi sparse_keymap snd_hda_intel snd_hda_codec snd_hwdep snd_pcm btusb snd_seq_midi dm_multipath uvcvideo videodev snd_rawmidi bluetooth joydev snd_seq_
midi_event snd_seq arc4 snd_timer rt2800pci snd_seq_device rt2800lib crc_ccitt rt2x00pci rt2x00lib psmouse snd serio_raw mac80211 cfg80211 soundcore sn
d_page_alloc intel_ips eeprom_93cx6 mei(C) hp_accel lis3lv02d input_polldev lp parport ums_realtek dm_raid45 xor dm_mirror dm_region_hash dm_log btrfs
zlib_deflate libcrc32c usb_storage uas usbhid hid ahci libahci i915 wmi drm_kms_helper drm r8169 i2c_algo_bit video
Oct 26 22:28:10 fqc-book kernel: [326923.537887]
Oct 26 22:28:10 fqc-book kernel: [326923.537901] Pid: 5475, comm: dconf-service Tainted: G WC 3.0.0-13-generic-pae #21-Ubuntu Hewlett-Packard H
P ProBook 4320s/1423
Oct 26 22:28:10 fqc-book kernel: [326923.537993] EIP: 0060:[<c12066c6>] EFLAGS: 00010246 CPU: 0
Oct 26 22:28:10 fqc-book kernel: [326923.538040] EIP is at encrypt_scatterlist+0xe6/0x120
Oct 26 22:28:10 fqc-book kernel: [326923.538077] EAX: 00000000 EBX: 00001000 ECX: cd423d74 EDX: cd423da4
Oct 26 22:28:10 fqc-book kernel: [326923.538126] ESI: cd76297c EDI: cd76297c EBP: cd423d48 ESP: cd423d1c
Oct 26 22:28:10 fqc-book kernel: [326923.538181] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Oct 26 22:28:10 fqc-book kernel: [326923.538229] Process dconf-service (pid: 5475, ti=cd422000 task=cfa2d940 task.ti=cd422000)
Oct 26 22:28:10 fqc-book kernel: [326923.538299] Stack:
Oct 26 22:28:10 fqc-book kernel: [326923.538318] 00000000 00000000 00000000 cd423d74 cd423d8c 00000000 cd423da4 00000200
Oct 26 22:28:10 fqc-book kernel: [326923.538389] 00001000 00000000 cd76297c cd423dc4 c1206ef4 00001000 cd423da4 cd762800
Oct 26 22:28:10 fqc-book kernel: [326923.538465] 00000000 cd423da4 f2702220 f50ed600 00000000 00000000 f50ed602 00000000
Oct 26 22:28:10 fqc-book kernel: [326923.538542] Call Trace:
Oct 26 22:28:10 fqc-book kernel: [326923.538570] [<c1206ef4>] ecryptfs_encrypt_extent+0xe4/0x230
Oct 26 22:28:10 fqc-book kernel: [326923.538620] [<c1207474>] ecryptfs_encrypt_page+0x94/0x1c0
Oct 26 22:28:10 fqc-book kernel: [326923.538667] [<c1205139>] ecryptfs_writepage+0x29/0x80
Oct 26 22:28:10 fqc-book kernel: [326923.538714] [<c10ef630>] __writepage+0x10/0x40
Oct 26 22:28:10 fqc-book kernel: [326923.538751] [<c10efaf3>] write_cache_pages+0x193/0x3f0
Oct 26 22:28:10 fqc-book kernel: [326923.538796] [<c10ef620>] ? set_page_dirty_lock+0x50/0x50
Oct 26 22:28:10 fqc-book kernel: [326923.538846] [<c10e9af9>] ? generic_file_aio_write+0x89/0xd0
Oct 26 22:28:10 fqc-book ke...

I am getting this crash occasionally since my initial report (dup Bug #876891) and usually on shutdown.

Not doing anything special or fancy, just basic stuff like Firefox and Update Manager.

Really annoying, but at least I am not losing any work (so far).

This is the version of the patch that I intend to upstream. It is against kernel version 3.2-rc1.

I've built an Oneiric test kernel, based upon 3.0.0-12.20, which contains this fix and a fix for bug #870326. Here are the links:
http://people.canonical.com/~tyhicks/ecryptfs/linux-headers-3.0.0-12-generic_3.0.0-12.20+ecryptfs1_amd64.deb
http://people.canonical.com/~tyhicks/ecryptfs/linux-image-3.0.0-12-generic_3.0.0-12.20+ecryptfs1_amd64.deb

This bug should be considered a security issue since one could implement a denial of service on a server using encrypted home.

Released in 3.2-rc3:

http://git.kernel.org/linus/b59db43ad4434519feb338eacb01d77eb50825c5

Submitted to Ubuntu kernel-team list:

https://lists.ubuntu.com/archives/kernel-team/2011-November/017950.html

Had this bug never before, but then twice today. Unfortunately I can't remember what updates I received today. I've doing nothing special or HD intensive. But then I was kicked out of Unity to a terminal showing me

kernel BUG at /build/buildd/linux-3.0.0/fs/ecryptfs/crypto.c:348!

By the way, I'm using a SSD. First I was afraid of cells dying away.

This bug is awaiting verification that the kernel for Oneiric in -proposed solves the problem (3.0.0-15.24). Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-oneiric' to 'verification-done-oneiric'.

If verification is not done by one week from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

I tested the original crash reproduction recipe on both the current Python 2.7 and 3.3 mercurial branches. No kernel panic in either case. I suppose it's possible that something else changed in Python or 11.10 to avoid the crash, but I doubt it for Python 2.7. So I think the oneiric-proposed kernel does fix the original bug report. Thanks!

This bug was fixed in the package linux - 3.0.0-15.25

---------------
linux (3.0.0-15.25) oneiric-proposed; urgency=low

  [Brad Figg]

  * Release Tracking Bug
    - LP: #910894

  [ Upstream Kernel Changes ]

  * Revert "clockevents: Set noop handler in clockevents_exchange_device()"
    - LP: #904569

linux (3.0.0-15.24) oneiric-proposed; urgency=low

  [Herton R. Krzesinski]

  * Release Tracking Bug
    - LP: #903188

  [ Alex Bligh ]

  * (config) Change Xen paravirt drivers to be built-in
    - LP: #886521

  [ Chase Douglas ]

  * Revert "SAUCE: HID: hid-ntrig: add support for 1b96:0006 model"
    - LP: #724831
  * Revert "SAUCE: hid: ntrig: Remove unused device ids"
    - LP: #724831

  [ Seth Forshee ]

  * SAUCE: dell-wmi: Demote unknown WMI event message to pr_debug
    - LP: #581312

  [ Upstream Kernel Changes ]

  * Revert "leds: save the delay values after a successful call to
    blink_set()"
    - LP: #893741
  * xfs: Fix possible memory corruption in xfs_readlink, CVE-2011-4077
    - LP: #887298
    - CVE-2011-4077
  * drm/i915: fix IVB cursor support
    - LP: #893222
  * drm/i915: always set FDI composite sync bit
    - LP: #893222
  * jbd/jbd2: validate sb->s_first in journal_get_superblock()
    - LP: #893148
    - CVE-2011-4132
  * ALSA: hda - Don't add elements of other codecs to vmaster slave
    - LP: #893741
  * virtio-pci: fix use after free
    - LP: #893741
  * ASoC: Don't use wm8994->control_data in wm8994_readable_register()
    - LP: #893741
  * sh: Fix cached/uncaced address calculation in 29bit mode
    - LP: #893741
  * drm/i915: Fix object refcount leak on mmappable size limit error path.
    - LP: #893741
  * drm/nouveau: initialize chan->fence.lock before use
    - LP: #893741
  * drm/radeon/kms: make an aux failure debug only
    - LP: #893741
  * ALSA: usb-audio - Check the dB-range validity in the later read, too
    - LP: #893741
  * ALSA: usb-audio - Fix the missing volume quirks at delayed init
    - LP: #893741
  * KEYS: Fix a NULL pointer deref in the user-defined key type
    - LP: #893741
  * hfs: add sanity check for file name length
    - LP: #893741
  * drm/radeon: add some missing FireMV pci ids
    - LP: #893741
  * sfi: table irq 0xFF means 'no interrupt'
    - LP: #893741
  * x86, mrst: use a temporary variable for SFI irq
    - LP: #893741
  * b43: refuse to load unsupported firmware
    - LP: #893741
  * md/raid5: abort any pending parity operations when array fails.
    - LP: #893741
  * mfd: Fix twl4030 dependencies for audio codec
    - LP: #893741
  * xen:pvhvm: enable PVHVM VCPU placement when using more than 32 CPUs.
    - LP: #893741
  * xen-gntalloc: integer overflow in gntalloc_ioctl_alloc()
    - LP: #893741
  * xen-gntalloc: signedness bug in add_grefs()
    - LP: #893741
  * powerpc/ps3: Fix lost SMP IPIs
    - LP: #893741
  * powerpc: Copy down exception vectors after feature fixups
    - LP: #893741
  * backing-dev: ensure wakeup_timer is deleted
    - LP: #893741
  * block: Always check length of all iov entries in blk_rq_map_user_iov()
    - LP: #893741
  * Linux 3.0.10
    - LP: #893741
  * drm/i915: add multi-threaded forcewake support
    - LP: #891270
  * (pre-sta...