Multipath core dumps when VMs booted from volume go to ERROR state

Bug #1567353 reported by Elisabeth
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
multipath-tools (Ubuntu)
Expired
Undecided
Unassigned

Bug Description

We are using
multipath-tools 0.4.9-3ubuntu7.6

We found a couple of multipath core dumps generated on some compute nodes in our DC:

/var/log/crash/cores:
total 664
drwxr-x--- 2 root root 68 Apr 4 10:11 ./
drwxr-x--- 4 root root 38 Mar 30 22:09 ../
-rw-r----- 1 root root 679936 Apr 4 10:11 core.compute-0-11.domain.tld.1459757511.multipath.33543

/var/log/crash/cores:
total 640
drwxr-x--- 2 root root 68 Apr 4 10:11 ./
drwxr-x--- 4 root root 38 Mar 30 22:09 ../
-rw-r----- 1 root root 655360 Apr 4 10:11 core.compute-0-12.domain.tld.1459757513.multipath.31922

/var/log/crash/cores:
total 796
drwxr-x--- 2 root root 68 Apr 4 10:12 ./
drwxr-x--- 4 root root 38 Mar 30 22:09 ../
-rw-r----- 1 root root 815104 Apr 4 10:12 core.compute-1-10.domain.tld.1459757567.multipath.38307

/var/log/crash/cores:
total 772
drwxr-x--- 2 root root 67 Apr 4 10:12 ./
drwxr-x--- 4 root root 38 Mar 30 22:09 ../
-rw-r----- 1 root root 790528 Apr 4 10:12 core.compute-2-5.domain.tld.1459757554.multipath.60237

These seem to be related to the issue when a volume was deleted (most probably these volumes was leftovers from when a VM booted from volume but the VM went into ERROR state due to expiry of block_device_allocate_retries).

Example from coredump related to compute-0-11:
cinder-volume.log:
2016-04-04 10:11:51.605 24563 INFO cinder.volume.manager [req-af0bea83-2b63-4118-8b3a-1c3aa1c57c8d 80c2726a44c54702a90265160a4adb5b e9e448cc9c59455f96b8325c694bd60c - - -] volume dd29523a-5a19-4075-ac77-84cddb75edc4: deleting
2016-04-04T10:11:51.605818+02:00 cic-1 cinder-volume 2016-04-04 10:11:51.605 24563 INFO cinder.volume.manager [req-af0bea83-2b63-4118-8b3a-1c3aa1c57c8d 80c2726a44c54702a90265160a4adb5b e9e448cc9c59455f96b8325c694bd60c - - -] volume dd29523a-5a19-4075-ac77-84cddb75edc4: deleting

kern.log:
2016-04-04T10:11:51.237422+02:00 compute-0-11 kernel: [397287.619468] device-mapper: multipath: Failing path 8:64.
2016-04-04T10:11:51.241272+02:00 compute-0-11 kernel: [397287.621247] scsi 2:0:0:244: alua: Detached
2016-04-04T10:11:51.245269+02:00 compute-0-11 kernel: [397287.623588] sd 1:0:0:244: alua: port group 01 state A non-preferred supports tolUsNA
2016-04-04T10:11:51.269243+02:00 compute-0-11 kernel: [397287.648665] bio: create slab <bio-1> at 1
2016-04-04T10:11:51.316343+02:00 compute-0-11 kernel: [397287.666870] device-mapper: multipath: Failing path 8:32.
2016-04-04T10:11:51.316352+02:00 compute-0-11 kernel: [397287.666881] end_request: I/O error, dev dm-3, sector 2872
2016-04-04T10:11:51.316354+02:00 compute-0-11 kernel: [397287.671502] bio: create slab <bio-1> at 1
2016-04-04T10:11:51.316358+02:00 compute-0-11 kernel: [397287.698405] Buffer I/O error on device dm-4, logical block 103
2016-04-04T10:11:51.405279+02:00 compute-0-11 kernel: [397287.731521] end_request: I/O error, dev dm-3, sector 2872
2016-04-04T10:11:51.405285+02:00 compute-0-11 kernel: [397287.733538] multipath[33543]: segfault at 0 ip 00007f308136d444 sp 00007ffcb0f3a5d8 error 4 in libc-2.19.so[7f308122b000+1bb000]
2016-04-04T10:11:51.405287+02:00 compute-0-11 kernel: [397287.745535] bio: create slab <bio-1> at 1
2016-04-04T10:11:51.405287+02:00 compute-0-11 kernel: [397287.762127] Buffer I/O error on device dm-4, logical block 103
2016-04-04T10:11:51.461579+02:00 compute-0-11 kernel: [397287.796824] device-mapper: table: 252:3: multipath: invalid initial priority group
2016-04-04T10:11:51.461588+02:00 compute-0-11 kernel: [397287.843587] device-mapper: ioctl: error adding target to table
2016-04-04T10:11:51.509422+02:00 compute-0-11 kernel: [397287.843660] device-mapper: table: 252:3: multipath: invalid initial priority group
2016-04-04T10:11:51.513240+02:00 compute-0-11 kernel: [397287.891469] device-mapper: ioctl: error adding target to table
2016-04-04T10:11:51.558139+02:00 compute-0-11 kernel: [397287.907562] end_request: I/O error, dev dm-3, sector 2560
2016-04-04T10:11:51.558147+02:00 compute-0-11 kernel: [397287.940151] Buffer I/O error on device dm-4, logical block 64
2016-04-04T10:11:51.626379+02:00 compute-0-11 kernel: [397287.976418] end_request: I/O error, dev dm-3, sector 2560
2016-04-04T10:11:51.626387+02:00 compute-0-11 kernel: [397287.977382] bio: create slab <bio-1> at 1
2016-04-04T10:11:51.626388+02:00 compute-0-11 kernel: [397288.008373] Buffer I/O error on device dm-4, logical block 64
2016-04-04T10:11:51.692068+02:00 compute-0-11 kernel: [397288.041954] end_request: I/O error, dev dm-3, sector 2080
2016-04-04T10:11:51.692075+02:00 compute-0-11 kernel: [397288.074050] Buffer I/O error on device dm-4, logical block 4
2016-04-04T10:11:51.757379+02:00 compute-0-11 kernel: [397288.107268] end_request: I/O error, dev dm-3, sector 2080
2016-04-04T10:11:51.757386+02:00 compute-0-11 kernel: [397288.139348] Buffer I/O error on device dm-4, logical block 4
2016-04-04T10:11:51.822990+02:00 compute-0-11 kernel: [397288.172705] end_request: I/O error, dev dm-3, sector 6144
2016-04-04T10:11:51.823012+02:00 compute-0-11 kernel: [397288.204953] Buffer I/O error on device dm-4, logical block 512
2016-04-04T10:11:51.901760+02:00 compute-0-11 kernel: [397288.249673] end_request: I/O error, dev dm-3, sector 41942912
2016-04-04T10:11:51.901767+02:00 compute-0-11 kernel: [397288.283682] Buffer I/O error on device dm-4, logical block 5242608
2016-04-04T10:11:51.972044+02:00 compute-0-11 kernel: [397288.319783] end_request: I/O error, dev dm-3, sector 41942912
2016-04-04T10:11:51.972050+02:00 compute-0-11 kernel: [397288.353952] Buffer I/O error on device dm-4, logical block 5242608
2016-04-04T10:11:52.042861+02:00 compute-0-11 kernel: [397288.390466] end_request: I/O error, dev dm-3, sector 41943024

Revision history for this message
Elisabeth (elisabeth-kemper) wrote :
Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :

Elisabeth,

Can you upgrade multipath-tools and multipath-tools-boot to the latest version (or at least 0.4.9-3ubuntu7.10) and rebuild the initramfs (update-initramfs -u) and reboot, and verify if the problem is fixed?

Joshua Powers (powersj)
Changed in multipath-tools (Ubuntu):
status: New → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for multipath-tools (Ubuntu) because there has been no activity for 60 days.]

Changed in multipath-tools (Ubuntu):
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.