kernel thread hang on iscsi target disconnect

btw, I don't know why it is attached to pwernap, I choosed linux-image-3.2.0-30-generic as package.
I don't know how change this :-(

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1056746

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.6 kernel[0] (Not a kernel in the daily directory) and install both the linux-image and linux-image-extra .deb packages.

Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. Please only remove that one tag and leave the other tags. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text.

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.6-quantal/

Hello!

I have exactly the same problem.
The Server hangs on iscsiadm logouts.
My ISCSI Target ist a HP P4500

Here is my stack trace:

[ 2360.832232] sd 3:0:0:0: [sdb] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB)
[ 2360.833438] sd 3:0:0:0: [sdb] Write Protect is off
[ 2360.833445] sd 3:0:0:0: [sdb] Mode Sense: 77 00 00 08
[ 2360.834112] sd 3:0:0:0: [sdb] Write cache: disabled, read cache: disabled, doesn't support DPO or FUA
[ 2360.850500] sdb: sdb1 sdb2 < sdb5 >
[ 2360.852989] sd 3:0:0:0: [sdb] Attached SCSI disk
[ 2403.635452] BUG: soft lockup - CPU#6 stuck for 22s! [kworker/u:0:5]
[ 2403.635805] Modules linked in: ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi usbhid hid veth ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables openvswitch_mod(O) bridge stp kvm_intel kvm nbd vesafb bonding dm_multipath psmouse serio_raw sb_edac edac_core hpilo ioatdma dca acpi_power_meter mac_hid lp parport hpsa tg3 [last unloaded: scsi_transport_iscsi]
[ 2403.635841] CPU 6
[ 2403.635842] Modules linked in: ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi usbhid hid veth ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables openvswitch_mod(O) bridge stp kvm_intel kvm nbd vesafb bonding dm_multipath psmouse serio_raw sb_edac edac_core hpilo ioatdma dca acpi_power_meter mac_hid lp parport hpsa tg3 [last unloaded: scsi_transport_iscsi]
[ 2403.635864]
[ 2403.635866] Pid: 5, comm: kworker/u:0 Tainted: G W O 3.2.0-31-generic #50-Ubuntu HP ProLiant DL360p Gen8
[ 2403.635869] RIP: 0010:[<ffffffff8165a70f>] [<ffffffff8165a70f>] _raw_spin_unlock_irqrestore+0x1f/0x30
[ 2403.635878] RSP: 0018:ffff88080c0e5da0 EFLAGS: 00000286
[ 2403.635880] RAX: 0000000000000286 RBX: 0000000000000286 RCX: 0000000000000001
[ 2403.635881] RDX: ffff880fffbc3c08 RSI: 0000000000000286 RDI: 0000000000000286
[ 2403.635883] RBP: ffff88080c0e5da0 R08: ffffffff81cddaa0 R09: 0000000000000100
[ 2403.635884] R10: 0000000000000040 R11: 0000000000000000 R12: 0000000000000001
[ 2403.635885] R13: 0000000000000001 R14: ffffffff81cddaa0 R15: 0000000000000100
[ 2403.635887] FS: 0000000000000000(0000) GS:ffff88101f200000(0000) knlGS:0000000000000000
[ 2403.635889] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2403.635890] CR2: 00007f43c5c87cd0 CR3: 0000000001c05000 CR4: 00000000000426e0
[ 2403.635892] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2403.635894] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 2403.635896] Process kworker/u:0 (pid: 5, threadinfo ffff88080c0e4000, task ffff88080c0c5c00)
[ 2403.635897] Stack:
[ 2403.635992] ffff88080c0e5dd0 ffffffff8142f3b4 ffff881001bff130 ffff881001bff000
[ 2403.635996] ffff881001bff018 0000000000000000 ffff88080c0e5e00 ffffffffa0106fec
[ 2...

@Joe

I'm able to reproduce the issue systematically in a KVM vm so I ran the following tests :

Kernel
3.2.0-31-virtual HANG
3.2.0-29-virtual OK
3.2.28-030228-generic HANG (this is the mainline kernel corresponding to 3.2.0-31-virtual I think)
3.5.0-17-generic OK (Quantal latest)

So looks like the issue is not seen in Quantal but do happen with the mainline kernel.

One important thing is that running 3.2.0-29-virtual DOES WORKAROUND this issue !

One thing that is important to know is that it will not happen on a system that does not use multipath. On the latest kernel, one workaround is to flush the multipath tables before logging out the iscsi nodes :

 $ sudo invoke-rc.d multipath-tools stop
 $ sudo multipath -F
 $ sudo iscsiadm -m node --logout

This will work and avoid hanging the kernel.

I'm now proceeding to bisect between those two kernel versions

This has been our experience as well.

I can verify that flushing the paths using *sudo multipath -F* prior to
logging off the iSCSI nodes would avoid the hang.

However as under normal operations we have LVM volumes on the multipath
device it was necessary to perform a *sudo lvchange -an <path to logical
volume>* to mark the lv as unavailable, otherwise the the multipath -F
command fails sue to an active path.

Nevertheless, if we follow this procedure we can reliably logout of the
iscsi nodes without a hang.

Stephen

On 16 October 2012 15:20, Louis Bouchard <email address hidden>wrote:

> One thing that is important to know is that it will not happen on a
> system that does not use multipath. On the latest kernel, one
> workaround is to flush the multipath tables before logging out the iscsi
> nodes :
>
> $ sudo invoke-rc.d multipath-tools stop
> $ sudo multipath -F
> $ sudo iscsiadm -m node --logout
>
> This will work and avoid hanging the kernel.
>
> I'm now proceeding to bisect between those two kernel versions
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1056746
>
> Title:
> kernel panic on iscsi target disconnect
>
> Status in “linux” package in Ubuntu:
> Incomplete
>
> Bug description:
> Hello!
>
> Just rebooted two identical Ubuntu 12.04.1 LTS servers to set linux-
> image-3.2.0-31-generic
>
> And got the same panic on both. Don't think it is fixed in -31.
>
> Servers are connected to IBM3512 over iscsi with multipath rdac and
> ocfs2.
> Here is log from one of servers:
>
>
>
> Sep 26 13:13:17 inetgw1 kernel: [1292980.531921] o2dlm: Leaving domain
> 68166ECC9E2A48208239B14D8218F0DA
> Sep 26 13:13:18 inetgw1 kernel: [1292981.054231] o2net: No longer
> connected to node inetgw2 (num 1) at 192.168.19.254:7777
> Sep 26 13:13:18 inetgw1 kernel: [1292981.055378] ocfs2: Unmounting
> device (252,6) on (node 0)
> Sep 26 13:13:20 inetgw1 kernel: [1292982.774884] ocfs2: Unregistered
> cluster interface o2cb
> Sep 26 13:13:24 inetgw1 kernel: [1292987.277805] sd 5:0:0:0: rdac:
> Detached
> Sep 26 13:13:24 inetgw1 kernel: [1292987.277882] sd 6:0:0:0: rdac:
> Detached
> Sep 26 13:13:24 inetgw1 kernel: [1292987.279549] sd 5:0:0:0: [sdb]
> Synchronizing SCSI cache
> Sep 26 13:13:24 inetgw1 kernel: [1292987.279808] sd 6:0:0:0: [sdc]
> Synchronizing SCSI cache
> Sep 26 13:13:26 inetgw1 kernel: [1292988.817740] sd 5:0:0:0: [sdb]
> Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Sep 26 13:13:26 inetgw1 kernel: [1292988.817746] sd 5:0:0:0: [sdb]
> Sense Key : Illegal Request [current]
> Sep 26 13:13:26 inetgw1 kernel: [1292988.817750] sd 5:0:0:0: [sdb]
> <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
>
> Sep 26 13:13:54 inetgw1 kernel: [1293013.970176] Modules linked in:
> btrfs zlib_deflate ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs
> reiserfs ext2 ip_vs libcrc32c scsi_dh_rdac xt_conntrack nf_conntrack_ftp
> xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_c
> onntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT
> xt_tcpudp iptable_filter ip_tables x_tables ib_iser rdm...

I had no chance to test mailine yet, but -31 and -32 has the same issue:

ip_vs libcrc32c xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables scsi_transport_iscsi vesafb dm_round_robin lp ioatdma joydev dm_multipath parport psmouse serio_raw dca i7core_edac mac_hid edac_core 8021q garp stp usbhid hid megaraid_sas e1000e [last unloaded: configfs]
[ 463.624735] CPU 1
[ 463.624737] Modules linked in: scsi_dh_rdac ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi ip_vs libcrc32c xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables scsi_transport_iscsi vesafb dm_round_robin lp ioatdma joydev dm_multipath parport psmouse serio_raw dca i7core_edac mac_hid edac_core 8021q garp stp usbhid hid megaraid_sas e1000e [last unloaded: configfs]
[ 463.624775]
[ 463.624778] Pid: 83, comm: kworker/u:2 Not tainted 3.2.0-32-generic #51-Ubuntu DEPO Computers X8STi/X8STi
[ 463.624784] RIP: 0010:[<ffffffff8142f6b0>] [<ffffffff8142f6b0>] __scsi_remove_target+0x20/0xf0
[ 463.624792] RSP: 0018:ffff880307067d80 EFLAGS: 00000246
[ 463.624795] RAX: 0000000000000000 RBX: ffffffff81cddaa0 RCX: 0000000000000010
[ 463.624798] RDX: ffff8802cc1a9008 RSI: 0000000000000286 RDI: ffff88030a75d930
[ 463.624800] RBP: ffff880307067da0 R08: ffffffff81cddaa0 R09: 0000000000000100
[ 463.624803] R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000010
[ 463.624806] R13: 0000000000000000 R14: 0000000000000010 R15: ffff880307067d70
[ 463.624809] FS: 0000000000000000(0000) GS:ffff88031fc20000(0000) knlGS:0000000000000000
[ 463.624812] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 463.624815] CR2: 00007f94fc247000 CR3: 0000000001c05000 CR4: 00000000000006e0
[ 463.624818] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 463.624821] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 463.624824] Process kworker/u:2 (pid: 83, threadinfo ffff880307066000, task ffff880307032e00)
[ 463.624826] Stack:
[ 463.652982] BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u:3:86]
[ 463.652985] Modules linked in: scsi_dh_rdac ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi ip_vs libcrc32c xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables scsi_transport_iscsi vesafb dm_round_robin lp ioatdma joydev dm_multipath parport psmouse serio_raw dca i7core_edac mac_hid edac_core 8021q garp stp usbhid hid megaraid_sas e1000e [last unloaded: configfs]
[ 463.653018] CPU 2
[ 463.653019] Modules linked in: scsi_dh_rdac ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi ip_vs libcrc32c xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle ...

I must correct one affirmation I made earlier. The issue is also visible on latest Quantal kernel, so the table should read :

Kernel
3.2.0-31-virtual HANG
3.2.0-29-virtual OK
3.2.28-030228-generic HANG (this is the mainline kernel corresponding to 3.2.0-31-virtual I think)
3.5.0-17-generic HANG (Quantal latest)

Kernel bisection outlines the following commit as the first bad :

$ git bisect good [0/1917]
1675b80790e5e373535d87b5ffbfafe7e532b9b9 is the first bad commit
commit 1675b80790e5e373535d87b5ffbfafe7e532b9b9
Author: Dan Williams <email address hidden>
Date: Thu Jun 21 23:47:28 2012 -0700

    fix hot unplug vs async scan race

    BugLink: http://bugs.launchpad.net/bugs/1032640

    commit 3b661a92e869ebe2358de8f4b3230ad84f7fce51 upstream.

    The following crash results from cases where the end_device has been
    removed before scsi_sysfs_add_sdev has had a chance to run.

     BUG: unable to handle kernel NULL pointer dereference at 0000000000000098
     IP: [<ffffffff8115e100>] sysfs_create_dir+0x32/0xb6
     ...
     Call Trace:
      [<ffffffff8125e4a8>] kobject_add_internal+0x120/0x1e3
      [<ffffffff81075149>] ? trace_hardirqs_on+0xd/0xf
      [<ffffffff8125e641>] kobject_add_varg+0x41/0x50
      [<ffffffff8125e70b>] kobject_add+0x64/0x66
      [<ffffffff8131122b>] device_add+0x12d/0x63a
      [<ffffffff814b65ea>] ? _raw_spin_unlock_irqrestore+0x47/0x56
      [<ffffffff8107de15>] ? module_refcount+0x89/0xa0
      [<ffffffff8132f348>] scsi_sysfs_add_sdev+0x4e/0x28a
      [<ffffffff8132dcbb>] do_scan_async+0x9c/0x145

    ...teach scsi_sysfs_add_devices() to check for deleted devices() before
    trying to add them, and teach scsi_remove_target() how to remove targets
    that have not been added via device_add().

    Reported-by: Dariusz Majchrzak <email address hidden>
    Signed-off-by: Dan Williams <email address hidden>
    Signed-off-by: James Bottomley <email address hidden>
    Signed-off-by: Ben Hutchings <email address hidden>
    Signed-off-by: Herton Ronaldo Krzesinski <email address hidden>

:040000 040000 8d739d0b793a09849cc280e781cb3613391ed8f9 ff60255ac7ac2eced39510bec9caf066b431ce7a M drivers

OK, I added this

   #sort of workaround
        /usr/sbin/invoke-rc.d multipath-tools stop
        /sbin/multipath -F

to stop section in open-iscsi
and
  /sbin/multipath -F

to multipath-tools after daemon stop.

tested on one of two servers, reboot is OK :-)

Thanks for th update louis, I'm a bit busy until next Tuesday but I will
test this in our configuration.

Does that multipath-tools stop have the same effect as the lvchange -an
command I was using to ensure that there were no active paths preventing
multipath -F from completing successfully?

Stephen

On 18 October 2012 09:58, sles <email address hidden> wrote:

> OK, I added this
>
> #sort of workaround
> /usr/sbin/invoke-rc.d multipath-tools stop
> /sbin/multipath -F
>
> to stop section in open-iscsi
> and
> /sbin/multipath -F
>
> to multipath-tools after daemon stop.
>
> tested on one of two servers, reboot is OK :-)
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1056746
>
> Title:
> kernel panic on iscsi target disconnect
>
> Status in “linux” package in Ubuntu:
> Confirmed
>
> Bug description:
> Hello!
>
> Just rebooted two identical Ubuntu 12.04.1 LTS servers to set linux-
> image-3.2.0-31-generic
>
> And got the same panic on both. Don't think it is fixed in -31.
>
> Servers are connected to IBM3512 over iscsi with multipath rdac and
> ocfs2.
> Here is log from one of servers:
>
>
>
> Sep 26 13:13:17 inetgw1 kernel: [1292980.531921] o2dlm: Leaving domain
> 68166ECC9E2A48208239B14D8218F0DA
> Sep 26 13:13:18 inetgw1 kernel: [1292981.054231] o2net: No longer
> connected to node inetgw2 (num 1) at 192.168.19.254:7777
> Sep 26 13:13:18 inetgw1 kernel: [1292981.055378] ocfs2: Unmounting
> device (252,6) on (node 0)
> Sep 26 13:13:20 inetgw1 kernel: [1292982.774884] ocfs2: Unregistered
> cluster interface o2cb
> Sep 26 13:13:24 inetgw1 kernel: [1292987.277805] sd 5:0:0:0: rdac:
> Detached
> Sep 26 13:13:24 inetgw1 kernel: [1292987.277882] sd 6:0:0:0: rdac:
> Detached
> Sep 26 13:13:24 inetgw1 kernel: [1292987.279549] sd 5:0:0:0: [sdb]
> Synchronizing SCSI cache
> Sep 26 13:13:24 inetgw1 kernel: [1292987.279808] sd 6:0:0:0: [sdc]
> Synchronizing SCSI cache
> Sep 26 13:13:26 inetgw1 kernel: [1292988.817740] sd 5:0:0:0: [sdb]
> Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Sep 26 13:13:26 inetgw1 kernel: [1292988.817746] sd 5:0:0:0: [sdb]
> Sense Key : Illegal Request [current]
> Sep 26 13:13:26 inetgw1 kernel: [1292988.817750] sd 5:0:0:0: [sdb]
> <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
>
> Sep 26 13:13:54 inetgw1 kernel: [1293013.970176] Modules linked in:
> btrfs zlib_deflate ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs
> reiserfs ext2 ip_vs libcrc32c scsi_dh_rdac xt_conntrack nf_conntrack_ftp
> xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_c
> onntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT
> xt_tcpudp iptable_filter ip_tables x_tables ib_iser rdma_cm ib_cm iw_cm i
> b_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
> scsi_transport_iscsi vesafb dm_round_robin 8021q garp stp ioatdma psmouse
> dca seri
> o_raw dm_multipath joydev i7core_edac edac_core mac_hid lp parport
> usbhid hid megaraid_sas e1000e [last unloaded: configfs]
> Sep 26 13:13:54 inetgw1 kernel: [1293013.970234] CPU 0
> Sep 26 13:13:54 inetgw1 ke...

Hi!

The multipath -F workaround does not really work in my enviroment, beacause i use openstack nova which does not
use this command. so if i terminate a nova instance, nova tries to logout and the whole compute-node hangs after some time :(

Alex

FYI, turns out, this is fixed in upstream 3.7 :

commit bc3f02a795d3b4faa99d37390174be2a75d091bd
Author: Dan Williams <email address hidden>
Date: Tue Aug 28 22:12:10 2012 -0700

    [SCSI] scsi_remove_target: fix softlockup regression on hot remove

    John reports:
     BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u:8:2202]
     [..]
     Call Trace:
      [<ffffffff8141782a>] scsi_remove_target+0xda/0x1f0
      [<ffffffff81421de5>] sas_rphy_remove+0x55/0x60
      [<ffffffff81421e01>] sas_rphy_delete+0x11/0x20
      [<ffffffff81421e35>] sas_port_delete+0x25/0x160
      [<ffffffff814549a3>] mptsas_del_end_device+0x183/0x270

    ...introduced by commit 3b661a9 "[SCSI] fix hot unplug vs async scan race".

    Don't restart lookup of more stargets in the multi-target case, just
    arrange to traverse the list once, on the assumption that new targets
    are always added at the end. There is no guarantee that the target will
    change state in scsi_target_reap() so we can end up spinning if we
    restart.

Cherry-pick of the commit has been tested on the Quantal & Precise kernels & submitted to the Ubuntu kernel ML.

This is fixed in upstream so marking raring fix committed as we have a 3.7 snapshot in our tree.

This is still an issue in with linux-image-3.2.0-33-generic (3.2.0-33.52) in precise-proposed.

I have tested the latest PPA pre-proposed kernel that contains the fix. The tested version is :

 - linux - 3.2.0-33.52pre201210190400

So unless I'm mistaken, the fix is queued up for the next -proposed package.

@Louis

I've tested the ppa you pointed to on one affected host and it seems like it fixes the problem. Thanks.

... I guess it's not my day as it apparently masked another problem. Back to debugging.

Hello!

Is there any chance to get fix in 12.04 updates ? :-)

Thank you!

@sles

It will make it to the -updates pocket eventually. It's not in the latest 3.2.0-33.52 kernel currently in -updates, but since the commit is in the ubuntu-precise git tree and made it to the PPA, it will appear in -updates in the near future.

I'll put a note here once it gets there.

@sles

FWIW The pre-proposed package (linux-image-3.2.0-34-generic) listed earlier
is working well in our environment on 12.04.

On Thu, Nov 15, 2012 at 9:20 AM, Louis Bouchard <
<email address hidden>> wrote:

> @sles
>
> It will make it to the -updates pocket eventually. It's not in the
> latest 3.2.0-33.52 kernel currently in -updates, but since the commit is
> in the ubuntu-precise git tree and made it to the PPA, it will appear in
> -updates in the near future.
>
> I'll put a note here once it gets there.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1056746
>
> Title:
> kernel thread hang on iscsi target disconnect
>
> Status in “linux” package in Ubuntu:
> Fix Committed
> Status in “linux” source package in Precise:
> Fix Committed
> Status in “linux” source package in Quantal:
> Fix Released
>
> Bug description:
> Hello!
>
> Just rebooted two identical Ubuntu 12.04.1 LTS servers to set linux-
> image-3.2.0-31-generic
>
> And got the same panic on both. Don't think it is fixed in -31.
>
> Servers are connected to IBM3512 over iscsi with multipath rdac and
> ocfs2.
> Here is log from one of servers:
>
>
>
> Sep 26 13:13:17 inetgw1 kernel: [1292980.531921] o2dlm: Leaving domain
> 68166ECC9E2A48208239B14D8218F0DA
> Sep 26 13:13:18 inetgw1 kernel: [1292981.054231] o2net: No longer
> connected to node inetgw2 (num 1) at 192.168.19.254:7777
> Sep 26 13:13:18 inetgw1 kernel: [1292981.055378] ocfs2: Unmounting
> device (252,6) on (node 0)
> Sep 26 13:13:20 inetgw1 kernel: [1292982.774884] ocfs2: Unregistered
> cluster interface o2cb
> Sep 26 13:13:24 inetgw1 kernel: [1292987.277805] sd 5:0:0:0: rdac:
> Detached
> Sep 26 13:13:24 inetgw1 kernel: [1292987.277882] sd 6:0:0:0: rdac:
> Detached
> Sep 26 13:13:24 inetgw1 kernel: [1292987.279549] sd 5:0:0:0: [sdb]
> Synchronizing SCSI cache
> Sep 26 13:13:24 inetgw1 kernel: [1292987.279808] sd 6:0:0:0: [sdc]
> Synchronizing SCSI cache
> Sep 26 13:13:26 inetgw1 kernel: [1292988.817740] sd 5:0:0:0: [sdb]
> Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Sep 26 13:13:26 inetgw1 kernel: [1292988.817746] sd 5:0:0:0: [sdb]
> Sense Key : Illegal Request [current]
> Sep 26 13:13:26 inetgw1 kernel: [1292988.817750] sd 5:0:0:0: [sdb]
> <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
>
> Sep 26 13:13:54 inetgw1 kernel: [1293013.970176] Modules linked in:
> btrfs zlib_deflate ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs
> reiserfs ext2 ip_vs libcrc32c scsi_dh_rdac xt_conntrack nf_conntrack_ftp
> xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_c
> onntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT
> xt_tcpudp iptable_filter ip_tables x_tables ib_iser rdma_cm ib_cm iw_cm i
> b_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
> scsi_transport_iscsi vesafb dm_round_robin 8021q garp stp ioatdma psmouse
> dca seri
> o_raw dm_multipath joydev i7core_edac edac_core mac_hid lp parport
> usbhid hid megaraid_sas e1000e [last unloaded: configfs]
> Sep 26 13:13:54 inetgw1 kernel: [1293013.970234] CPU 0
> Sep 26 13:13:54 inetg...

The commit for this issue came in via a stable upstream release (linux 3.2.32). As such it is not subject to the standard bug verification process.

As a reference to everyone, the current kernel in the -proposed pocket does contain the fix for this issue. Its version is 3.2.0-34-generic #53-Ubuntu.

This should become available publicly in -updates in the upcoming week.

Hi

Just to add a twist. I am able to confirm that I could repeat this problem. The thing is, I found I didn't get the error until I added an entry for the iSCSI volume in fstab. And in fact, after removing the entry from fstab and restarting, I was able to again restart without an error.

Ok, so I had not mounted the volume before restarting. So the issue isn't fstab, it is when the volume is mounted.

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

This bug was fixed in the package linux - 3.2.0-34.53

---------------
linux (3.2.0-34.53) precise-proposed; urgency=low

  [Luis Henriques]

  * Release Tracking Bug
    - LP: #1078760

  [ Kamal Mostafa ]

  * SAUCE: input: Cypress PS/2 Trackpad list additional contributors

  [ Kyle Fazzari ]

  * SAUCE: input: Cypress PS/2 Trackpad fix multi-source, double-click
    - LP: #1055788
  * SAUCE: input: Cypress PS/2 Trackpad fix lost sync upon palm contact
    - LP: #1048258
  * SAUCE: input: Cypress PS/2 Trackpad fix taps turning into hardware
    clicks
    - LP: #1064086

  [ Tim Gardner ]

  * [Config] remove ndiswrapper from Provides:
    - LP: #1076395
  * [Config] CONFIG_HP_WATCHDOG=m for x86en
    - LP: #1076342

  [ Upstream Kernel Changes ]

  * Revert "SUNRPC: Ensure we close the socket on EPIPE errors too..."
    - LP: #1075355
  * Revert "ath9k_hw: Updated AR9003 tx gain table for 5GHz"
    - LP: #1075355
  * eCryptfs: check for eCryptfs cipher support at mount
    - LP: #338914
  * isci: fix isci_pci_probe() generates warning on efi failure path
    - LP: #1068162
  * mtd: nand: Use the mirror BBT descriptor when reading its version
    - LP: #1068162
  * drm/i915: prevent possible pin leak on error path
    - LP: #1068162
  * workqueue: add missing smp_wmb() in process_one_work()
    - LP: #1068162
  * TTY: ttyprintk, don't touch behind tty->write_buf
    - LP: #1068162
  * Remove BUG_ON from n_tty_read()
    - LP: #1068162
  * n_gsm.c: Implement 3GPP27.010 DLC start-up procedure in MUX
    - LP: #1068162
  * n_gsm: uplink SKBs accumulate on list
    - LP: #1068162
  * n_gsm : Flow control handling in Mux driver
    - LP: #1068162
  * char: n_gsm: remove message filtering for contipated DLCI
    - LP: #1068162
  * n_gsm: added interlocking for gsm_data_lock for certain code paths
    - LP: #1068162
  * n_gsm: avoid accessing freed memory during CMD_FCOFF condition
    - LP: #1068162
  * n_gsm: replace kfree_skb w/ appropriate dev_* versions
    - LP: #1068162
  * n_gsm: memory leak in uplink error path
    - LP: #1068162
  * UBI: fix autoresize handling in R/O mode
    - LP: #1068162
  * UBI: erase free PEB with bitflip in EC header
    - LP: #1068162
  * firmware: Add missing attributes to EFI variable attribute print out
    from sysfs
    - LP: #1068162
  * tools/hv: Fix exit() error code
    - LP: #1068162
  * slab: fix the DEADLOCK issue on l3 alien lock
    - LP: #1068162
  * gspca_pac7302: Add usb-id for 145f:013c
    - LP: #1068162
  * gspca_pac7302: add support for device 1ae7:2001 Speedlink Snappy
    Microphone SL-6825-SBK
    - LP: #1068162
  * xhci: Warn when hosts don't halt.
    - LP: #1068162
  * xHCI: add cmd_ring_state
    - LP: #1068162
  * xHCI: add aborting command ring function
    - LP: #1068162
  * xHCI: cancel command after command timeout
    - LP: #1068162
  * hpsa: Use LUN reset instead of target reset
    - LP: #1068162
  * rc: ite-cir: Initialise ite_dev::rdev earlier
    - LP: #1068162
  * staging: speakup_soft: Fix reading of init string
    - LP: #1068162
  * target: fix return code in target_core_init_configfs error path
    - LP: #1068162
  * powerpc/eeh: Lock module while handling EEH event
  ...

The same issue happens for me with the linux-image-server:

Dec 7 12:04:24 node1 multipathd: sdc: remove path (uevent)
Dec 7 12:04:24 node1 kernel: [65815.457105] sd 1:0:0:1: [sdc] Synchronizing SCSI cache
Dec 7 12:04:24 node1 kernel: [65815.458094] sd 1:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Dec 7 12:04:24 node1 kernel: [65815.458101] sd 1:0:0:1: [sdc] Sense Key : Illegal Request [current]
Dec 7 12:04:24 node1 kernel: [65815.458106] sd 1:0:0:1: [sdc] Add. Sense: Logical unit not supported
Dec 7 12:04:48 node1 kernel: [65839.483571] BUG: soft lockup - CPU#10 stuck for 22s! [kworker/u:1:8014]
Dec 7 12:04:48 node1 kernel: [65839.486057] Modules linked in: dm_snapshot ipt_LOG ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables kvm_intel kvm ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vesafb xfs joydev i7core_edac edac_core ioatdma bridge stp tpm_tis mac_hid lp parport dm_round_robin usbhid hid mptsas mptscsih mptbase igb scsi_transport_sas dca dm_multipath [last unloaded: scsi_wait_scan]
Dec 7 12:04:48 node1 kernel: [65839.486109] CPU 10
Dec 7 12:04:48 node1 kernel: [65839.486110] Modules linked in: dm_snapshot ipt_LOG ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables kvm_intel kvm ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vesafb xfs joydev i7core_edac edac_core ioatdma bridge stp tpm_tis mac_hid lp parport dm_round_robin usbhid hid mptsas mptscsih mptbase igb scsi_transport_sas dca dm_multipath [last unloaded: scsi_wait_scan]
Dec 7 12:04:48 node1 kernel: [65839.486151]
Dec 7 12:04:48 node1 kernel: [65839.486154] Pid: 8014, comm: kworker/u:1 Not tainted 3.2.0-32-generic #51-Ubuntu INTEL MFS5520VI Compute Module/MFS5520VI Compute Module
Dec 7 12:04:48 node1 kernel: [65839.486160] RIP: 0010:[<ffffffff8165b049>] [<ffffffff8165b049>] _raw_spin_unlock_irqrestore+0x19/0x30
Dec 7 12:04:48 node1 kernel: [65839.486170] RSP: 0018:ffff880c44a73d98 EFLAGS: 00000286
Dec 7 12:04:48 node1 kernel: [65839.486172] RAX: 0000000000000286 RBX: 0000000000000018 RCX: 0000000000000018
Dec 7 12:04:48 node1 kernel: [65839.486175] RDX: ffff8806433e0408 RSI: 0000000000000286 RDI: 0000000000000286
Dec 7 12:04:48 node1 kernel: [65839.486178] RBP: ffff880c44a73da0 R08: ffffffff81cddaa0 R09: 0000000000000100
Dec 7 12:04:48 node1 kernel: [65839.486181] R10: 0000000000000018 R11: 0000000000000001 R12: ffffffff81cddaa0
Dec 7 12:04:48 node1 kernel: [65839.486183] R13: 0000000000000100 R14: 0000000000000018 R15: 0000000000000001
Dec 7 12:04:48 node1 kernel: [65839.486187] FS: 0000000000000000(0000) GS:ffff880657ca0000(0000) knlGS:0000000000000000
Dec 7 12:04:48 node1 kernel: [65839.486190] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Dec 7 12:04:48 node1 kernel: [6...

> This kernel is running here:
>
> root@node1:~# dpkg -s linux-image-server
> ...
> Version: 3.2.0.34.37

Just to eliminate the obvious, does "uname -a" agree on the kernel version?

Status changed to 'Confirmed' because the bug affects multiple users.