oops in gss_validate

Bug #459265 reported by Brian J. Murrell
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Linux
Fix Released
Medium
linux (Ubuntu)
Fix Released
Medium
Unassigned
Karmic
Fix Released
Medium
Unassigned

Bug Description

SRU Justification:

Impact: Unbalanced references caused use after free which result in an oops.

Fix: Patch coming in with 2.6.31.2

---

As reported upstream, there is a regression in the 2.6.31 kernel with regard to GSS authenticated NFS mounts.

I have run into this on the Karmic 2.6.31-14-generic #48-Ubuntu kernel.

I can't see why this cannot and should not block the release of Karmic.

For the benefit of this bug, the oops looks like:

[253207.745918] BUG: unable to handle kernel NULL pointer dereference at 00000010
[253207.749013] IP: [<fb27d24b>] gss_validate+0x7b/0x1d0 [auth_rpcgss]
[253207.753994] *pde = 94fb8067
[253207.753994] Oops: 0000 [#1] SMP
[253207.753994] last sysfs file: /sys/devices/pci0000:00/0000:00:0b.1/usb1/1-3/1-3:1.0/host6/target6:0:0/6:0:0:0/block/sde/sde1/stat
[253207.753994] Modules linked in: xt_multiport binfmt_misc bridge stp bnep vboxnetflt vboxdrv tun des_generic cbc autofs4 video output rpcsec_gss_krb5 nfsd exportfs nfs lockd nfs_acl auth_rpcgss sunrpc nf_conntrack_ipv6 xt_hl ipt_LOG xt_limit ipt_REJECT xt_tcpudp x
[253207.842462]
[253207.842462] Pid: 4036, comm: rpciod/1 Tainted: P (2.6.31-14-generic #48-Ubuntu) System Product Name
[253207.842462] EIP: 0060:[<fb27d24b>] EFLAGS: 00010296 CPU: 1
[253207.842462] EIP is at gss_validate+0x7b/0x1d0 [auth_rpcgss]
[253207.842462] EAX: 00000004 EBX: 00000000 ECX: f6abde80 EDX: f28128e4
[253207.842462] ESI: 00000025 EDI: ec7b6fc4 EBP: f6abdea4 ESP: f6abde40
[253207.842462] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[253207.842462] Process rpciod/1 (pid: 4036, ti=f6abc000 task=f6a33ed0 task.ti=f6abc000)
[253207.842462] Stack:
[253207.842462] f6abde58 c049ca59 00000001 00000001 f28128e4 e43210c0 f6abde94 00000004
[253207.842462] <0> 00000000 00000000 f6abde8c c0121270 00000000 02020202 00000004 00000004
[253207.842462] <0> 00000025 f28128e4 f6abde94 00000004 00000100 85030000 ec7b6fc4 e43210c0
[253207.842462] Call Trace:
[253207.842462] [<c049ca59>] ? net_tx_action+0x59/0x130
[253207.842462] [<c0121270>] ? ack_apic_level+0x60/0x230
[253207.842462] [<fb242bf2>] ? rpcauth_checkverf+0x22/0x60 [sunrpc]
[253207.842462] [<c014b60f>] ? irq_exit+0x2f/0x70
[253207.842462] [<c0104f10>] ? do_IRQ+0x50/0xc0
[253207.842462] [<fb23b2df>] ? rpc_verify_header+0x1af/0x5c0 [sunrpc]
[253207.842462] [<c01039b0>] ? common_interrupt+0x30/0x40
[253207.842462] [<fb23b807>] ? call_decode+0x117/0x220 [sunrpc]
[253207.842462] [<fb33dfd0>] ? nfs4_xdr_dec_read+0x0/0x60 [nfs]
[253207.842462] [<fb242022>] ? __rpc_execute+0x92/0x1f0 [sunrpc]
[253207.842462] [<fb2421ab>] ? rpc_async_schedule+0xb/0x10 [sunrpc]
[253207.842462] [<c0157a7e>] ? run_workqueue+0x6e/0x140
[253207.842462] [<fb2421a0>] ? rpc_async_schedule+0x0/0x10 [sunrpc]
[253207.842462] [<c0157bd8>] ? worker_thread+0x88/0xe0
[253207.842462] [<c015c280>] ? autoremove_wake_function+0x0/0x40
[253207.842462] [<c0157b50>] ? worker_thread+0x0/0xe0
[253207.842462] [<c015bf8c>] ? kthread+0x7c/0x90
[253207.842462] [<c015bf10>] ? kthread+0x0/0x90
[253207.842462] [<c0104007>] ? kernel_thread_helper+0x7/0x10
[253207.842462] Code: 55 b4 8b 40 64 0f c8 89 45 f0 8d 45 f0 89 45 e4 8d 45 e4 c7 45 e8 04 00 00 00 e8 31 cf fc ff 8b 55 ac 8d 4d dc 89 75 dc 89 55 e0 <8b> 43 10 8d 55 b4 e8 2a 11 00 00 3d 00 00 0c 00 74 6b 85 c0 75
[253207.842462] EIP: [<fb27d24b>] gss_validate+0x7b/0x1d0 [auth_rpcgss] SS:ESP 0068:f6abde40
[253207.842462] CR2: 0000000000000010
[253207.845072] ---[ end trace ad285e035a384c5f ]---
[253208.107509] BUG: unable to handle kernel NULL pointer dereference at 00000010
[253208.107518] IP: [<fb27d24b>] gss_validate+0x7b/0x1d0 [auth_rpcgss]
[253208.107534] *pde = aee17067
[253208.107537] Oops: 0000 [#2] SMP
[253208.107540] last sysfs file: /sys/devices/pci0000:00/0000:00:0b.1/usb1/1-3/1-3:1.0/host6/target6:0:0/6:0:0:0/block/sde/sde1/stat
[253208.107544] Modules linked in: xt_multiport binfmt_misc bridge stp bnep vboxnetflt vboxdrv tun des_generic cbc autofs4 video output rpcsec_gss_krb5 nfsd exportfs nfs lockd nfs_acl auth_rpcgss sunrpc nf_conntrack_ipv6 xt_hl ipt_LOG xt_limit ipt_REJECT xt_tcpudp x
[253208.107607]
[253208.107611] Pid: 4033, comm: rpciod/0 Tainted: P D (2.6.31-14-generic #48-Ubuntu) System Product Name
[253208.107614] EIP: 0060:[<fb27d24b>] EFLAGS: 00010296 CPU: 0
[253208.107620] EIP is at gss_validate+0x7b/0x1d0 [auth_rpcgss]
[253208.107622] EAX: 00000004 EBX: 00000000 ECX: f64f7e80 EDX: d80a68e4
[253208.107625] ESI: 00000025 EDI: eb716c44 EBP: f64f7ea4 ESP: f64f7e40
[253208.107627] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[253208.107630] Process rpciod/0 (pid: 4033, ti=f64f6000 task=f6a34b60 task.ti=f64f6000)
[253208.107632] Stack:
[253208.107633] c0127c38 f64f7e58 c05707da f7070000 d80a68e4 e43210c0 f64f7e94 00000004
[253208.107637] <0> 00000000 00000000 00000292 ecb2d204 00000000 c0150c2b 00000004 00000004
[253208.107641] <0> 00000025 d80a68e4 f64f7e94 00000004 2942dfc4 88030000 eb716c44 e43210c0
[253208.107646] Call Trace:
[253208.107655] [<c0127c38>] ? default_spin_lock_flags+0x8/0x10
[253208.107660] [<c05707da>] ? _spin_lock_irqsave+0x2a/0x40
[253208.107664] [<c0150c2b>] ? mod_timer+0xcb/0x140
[253208.107695] [<fb242bf2>] ? rpcauth_checkverf+0x22/0x60 [sunrpc]
[253208.107709] [<fb23b2df>] ? rpc_verify_header+0x1af/0x5c0 [sunrpc]
[253208.107723] [<fb23b807>] ? call_decode+0x117/0x220 [sunrpc]
[253208.107756] [<fb33dfd0>] ? nfs4_xdr_dec_read+0x0/0x60 [nfs]
[253208.107772] [<fb242022>] ? __rpc_execute+0x92/0x1f0 [sunrpc]
[253208.107806] [<fb2421ab>] ? rpc_async_schedule+0xb/0x10 [sunrpc]
[253208.107811] [<c0157a7e>] ? run_workqueue+0x6e/0x140
[253208.107836] [<fb2421a0>] ? rpc_async_schedule+0x0/0x10 [sunrpc]
[253208.107849] [<c0157bd8>] ? worker_thread+0x88/0xe0
[253208.107858] [<c015c280>] ? autoremove_wake_function+0x0/0x40
[253208.107867] [<c0157b50>] ? worker_thread+0x0/0xe0
[253208.107870] [<c015bf8c>] ? kthread+0x7c/0x90
[253208.107873] [<c015bf10>] ? kthread+0x0/0x90
[253208.107877] [<c0104007>] ? kernel_thread_helper+0x7/0x10
[253208.107878] Code: 55 b4 8b 40 64 0f c8 89 45 f0 8d 45 f0 89 45 e4 8d 45 e4 c7 45 e8 04 00 00 00 e8 31 cf fc ff 8b 55 ac 8d 4d dc 89 75 dc 89 55 e0 <8b> 43 10 8d 55 b4 e8 2a 11 00 00 3d 00 00 0c 00 74 6b 85 c0 75
[253208.107898] EIP: [<fb27d24b>] gss_validate+0x7b/0x1d0 [auth_rpcgss] SS:ESP 0068:f64f7e40
[253208.107906] CR2: 0000000000000010
[253208.107909] ---[ end trace ad285e035a384c60 ]---

Changed in linux:
status: Unknown → Confirmed
Revision history for this message
Brian J. Murrell (brian-interlinx) wrote :

There is a patch upstream which the kernel team is looking for tests of. I wonder if Ubuntu want to add their to their "sauce".

Changed in linux:
status: Confirmed → In Progress
Revision history for this message
Wolfgang Granzer (wolfgang-granzer) wrote :

I have the same problem. We have about 30 clients that are running on Ubuntu and where the user homes are mounted via NFS4 (security enabled). Currently, we are not able to upgrade to Ubuntu 9.10 due to the problems with GSS authenticated NFS mounts. The old kernel 2.6.28 from Ubuntu 9.04 does also not work since it hangs during boot.

Is it planned to integrate the patch from the kernel team?

Revision history for this message
Leann Ogasawara (leannogasawara) wrote :

Seems this should be resolved with the following upstream patch which is included in the upstream 2.6.31.6 stable patchset which the Ubuntu Kernel will pull in as a Stable Release Update. I'll try to build a test kernel in the meantime. Please stay tuned.

ogasawara@emiko:~/linux-2.6$ git show 141aeb9f26f9f12f1584c128ce8697cdffb046e7
commit 141aeb9f26f9f12f1584c128ce8697cdffb046e7
Author: Trond Myklebust <email address hidden>
Date: Mon Oct 26 08:09:46 2009 -0400

    NFSv4: Fix two unbalanced put_rpccred() issues.

Changed in linux (Ubuntu):
importance: Undecided → Medium
status: New → Triaged
tags: added: 2.6.31.6
Revision history for this message
Wolfgang Granzer (wolfgang-granzer) wrote : Re: [Bug 459265] Re: oops in gss_validate

I have already build a Ubuntu x86 kernel including the patch from
linux-kernel-bugs #14249. I have tested the kernel for about one week.
Everything works fine. The "gss_validate" problem has been disappeared.

Changed in linux:
status: In Progress → Fix Released
Revision history for this message
Stefan Bader (smb) wrote :

Should be in Lucid.

Changed in linux (Ubuntu):
status: Triaged → Fix Released
Changed in linux (Ubuntu Karmic):
importance: Undecided → Medium
status: New → Fix Committed
Stefan Bader (smb)
description: updated
Revision history for this message
Brian J. Murrell (brian-interlinx) wrote :

Does comment #3 mean we should see a fix for this in Karmic?

Revision history for this message
Wolfgang Granzer (wolfgang-granzer) wrote :

It seems that the fix has not been integrated yet in Karmic. I have updated to 2.6.31-15. However, the "gss_validate" problem remains.

Is there any reason why the fix has not been submitted for Karmic? This bug is really serious since it makes Karmic completely unusable if NFS4 mounts are used.

Revision history for this message
Brian J. Murrell (brian-interlinx) wrote :

On Thu, 2009-11-26 at 07:47 +0000, Wolfgang Granzer wrote:
> It seems that the fix has not been integrated yet in Karmic.

Right.

> I have
> updated to 2.6.31-15. However, the "gss_validate" problem remains.

Yup. IIRC that is based on 2.6.31.4 and the fix went into .6 (these are
upstream release versions, not Ubuntu package release versions).

> Is there any reason why the fix has not been submitted for Karmic?

Good question.

> This
> bug is really serious since it makes Karmic completely unusable if NFS4
> mounts are used.

Indeed.

What really puzzles me though is how does this kind of crap always seem
to slip through the QA process? Are they not trying to exercise at
least the most common functionality before calling a release "good"?

Surely NFS4 is not all that uncommon.

Revision history for this message
Steve Langasek (vorlon) wrote :

On Thu, Nov 26, 2009 at 01:11:29PM -0000, Brian J. Murrell wrote:
> Yup. IIRC that is based on 2.6.31.4 and the fix went into .6 (these are
> upstream release versions, not Ubuntu package release versions).

And the bug has been marked 'fix committed', implying that it will be pulled
into the next kernel SRU via 2.6.31.6.

> Surely NFS4 is not all that uncommon.

Yes, it is.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

Revision history for this message
Brian J. Murrell (brian-interlinx) wrote :

On Fri, 2009-11-27 at 04:50 +0000, Steve Langasek wrote:
>
> And the bug has been marked 'fix committed', implying that it will be pulled
> into the next kernel SRU via 2.6.31.6.

Great. Can't wait to get off of this 2.6.30 kernel.

> Yes, it is.

I guess my point was that it should not be. Using/exercising NFSv4
during a QA cycle should not be difficult -- quite simple in fact. It's
the sort of thing that should be included into standard QA testing given
how easily it can be done. No special tests even need be created for
it. Just using it instead of (or rather in conjunction with) NFSv3,
CIFS and local filesystems should be enough.

Revision history for this message
Martin Pitt (pitti) wrote : Please test proposed package

Accepted linux into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

tags: added: verification-needed
Revision history for this message
Brian J. Murrell (brian-interlinx) wrote :

I see the new kernel (2.6.31-17.54) in karmic-proposed, but I don't see anything in the changelog that indicates that it fixes this problem. Neither does this bug number show up as one of the fixed issues in the changelog nor does the changelog show that this is an upstream 2.6.31.6 kernel.

Revision history for this message
Alex (d-f0rce) wrote :

@Brian Murrell
"aptitude changelog linux-image-2.6.31-17-generic" says:

----
* NFSv4: Fix two unbalanced put_rpccred() issues.
  - LP: #459265, #480144
----

That is the fix, isn't it?

@Martin Pitt
I have been running this kernel for a few days now. Didn't experience any crashes since.

Revision history for this message
Brian J. Murrell (brian-interlinx) wrote : Re: [Bug 459265] Re: oops in gss_validate

On Sun, 2009-12-13 at 12:23 +0000, Alex wrote:
> @Brian Murrell
> "aptitude changelog linux-image-2.6.31-17-generic" says:
>
> ----
> * NFSv4: Fix two unbalanced put_rpccred() issues.
> - LP: #459265, #480144

Indeed. I must have missed this the first time around.

Martin Pitt (pitti)
tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (11.9 KiB)

This bug was fixed in the package linux - 2.6.31-17.54

---------------
linux (2.6.31-17.54) karmic-proposed; urgency=low

  [ John Johansen ]

  * SAUCE: AppArmor: Fix oops after profile removal
    - LP: #475619
  * SAUCE: AppArmor: Fix Oops when in apparmor_bprm_set_creds
    - LP: #437258
  * SAUCE: AppArmor: Fix cap audit_caching preemption disabling
    - LP: #479102
  * SAUCE: AppArmor: Fix refcounting bug causing leak of creds
    - LP: #479115
  * SAUCE: AppArmor: Fix oops there is no tracer and doing unsafe
    transition.
    - LP: #480112

  [ Leann Ogasawara ]

  * Revert "[Upstream] (drop after 2.6.31) usb-storage: Workaround devices
    with bogus sense size"
    - LP: #461556
  * Revert "[Upstream] (drop after 2.6.31) Input: synaptics - add another
    Protege M300 to rate blacklist"
    - LP: #480144

  [ Tim Gardner ]

  * [Config] udeb: Add squashfs to fs-core-modules
    - LP: #352615

  [ Upstream Kernel Changes ]

  * Revert "e1000e: swap max hw supported frame size between 82574 and
    82583"
    - LP: #461556
  * Revert "drm/i915: Fix FDI M/N setting according with correct color
    depth"
    - LP: #480144
  * Revert "agp/intel: Add B43 chipset support"
    - LP: #480144
  * Revert "drm/i915: add B43 chipset support"
    - LP: #480144
  * Revert "ACPI: Attach the ACPI device to the ACPI handle as early as
    possible"
    - LP: #327499, #480144
  * SCSI: Retry ADD_TO_MLQUEUE return value for EH commands
    - LP: #461556
  * SCSI: Fix protection scsi_data_buffer leak
    - LP: #461556
  * SCSI: sg: Free data buffers after calling blk_rq_unmap_user
    - LP: #461556
  * ARM: pxa: workaround errata #37 by not using half turbo switching
    - LP: #461556
  * tracing/filters: Fix memory leak when setting a filter
    - LP: #461556
  * x86/paravirt: Use normal calling sequences for irq enable/disable
    - LP: #461556
  * USB: ftdi_sio: remove tty->low_latency
    - LP: #461556
  * USB: ftdi_sio: remove unused rx_byte counter
    - LP: #461556
  * USB: ftdi_sio: clean up read completion handler
    - LP: #461556
  * USB: ftdi_sio: re-implement read processing
    - LP: #461556
  * USB: pl2303: fix error characters not being reported to ldisc
    - LP: #461556
  * USB: digi_acceleport: Fix broken unthrottle.
    - LP: #461556
  * USB: serial: don't call release without attach
    - LP: #461556
  * USB: option: Toshiba G450 device id
    - LP: #461556
  * USB: ipaq: fix oops when device is plugged in
    - LP: #461556
  * USB: cp210x: Add support for the DW700 UART
    - LP: #461556
  * USB: Fix throttling in generic usbserial driver
    - LP: #461556
  * USB: storage: When a device returns no sense data, call it a Hardware
    Error
    - LP: #400652, #461556
  * arm, cris, mips, sparc, powerpc, um, xtensa: fix build with bash 4.0
    - LP: #461556
  * intel-iommu: Cope with broken HP DC7900 BIOS
    - LP: #461556
  * futex: Detect mismatched requeue targets
    - LP: #461556
  * futex: Fix wakeup race by setting TASK_INTERRUPTIBLE before queue_me()
    - LP: #461556
  * tpm-fixup-pcrs-sysfs-file-update
    - LP: #461556
  * TPM: fix pcrread
    - LP: #461556
  * Bluetooth: Disconnect HIDRAW devices on disconnect
    - LP...

Changed in linux (Ubuntu Karmic):
status: Fix Committed → Fix Released
Changed in linux:
importance: Unknown → Medium
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.