Trusty multipath-tools suffering seg faults

Bug #1532789 reported by Rafael David Tinoco
18
This bug affects 1 person
Affects Status Importance Assigned to Milestone
multipath-tools (Ubuntu)
Invalid
High
Unassigned
Trusty
Fix Released
High
Louis Bouchard

Bug Description

[Impact]

 * multipath segfaults constantly on given circumstances
 * multipath might make other tools relying on it to misbehave (openstack)

[Test Case]

 * Core Dump given. No test case or reproducer.
 * Will ask user in charge for the dump to verify fix.

[Regression Potential]

 * Cherry-picked commit that EXACTLY fits seg fault and dump analysis.
 * Minor change.

[Other Info]

####

Original BUG Description:

It was brought to my knowledge that multipath-tools is facing segfaults:

----
Core was generated by `/sbin/multipath -v0 /dev/sdab'. Program terminated with signal 11, Segmentation fault. #0 0x00007f2b5f6c8e88 in scsi_sysfs_pathinfo (pp=0xe5c0e0) at discovery.c:441
----

All core dumps I received - several - have IP at the same place:

----
#0 0x00007f2b5f6c8e88 in scsi_sysfs_pathinfo (pp=0xe5c0e0) at discovery.c:441
#1 0x00007f2b5f6c9c26 in sysfs_pathinfo (pp=0xe5c0e0) at discovery.c:689
#2 0x00007f2b5f6ca48b in pathinfo (pp=0xe5c0e0, hwtable=0xe222f0, mask=63) at discovery.c:823
#3 0x00007f2b5f6c7858 in store_pathinfo (pathvec=0xe2d8d0, hwtable=0xe222f0, udevice=0xe5bc60, flag=31, pp_ptr=0x0) at discovery.c:57
#4 0x00007f2b5f6c7953 in path_discover (pathvec=0xe2d8d0, conf=0xe22110, udevice=0xe5bc60, flag=31) at discovery.c:91
#5 0x00007f2b5f6c7ad3 in path_discovery (pathvec=0xe2d8d0, conf=0xe22110, flag=31) at discovery.c:127
#6 0x00000000004025e6 in configure () at main.c:322
----

From libmultipath/discovery.c:

if (!strncmp(udev_device_get_subsystem(parent), "scsi", 4)) {

It is clear that strncmp is not taking care of cases when udev_device_get_subsystem returns NULL and that is - most likely - what is happening here. I have checked upstream development for this particular fix and found the following commit:

----
commit e546687e046b1eb1cbe5567e9d7850cbbd682dba
Author: Benjamin Marzinski <email address hidden>
Date: Sat Jan 12 00:04:54 2013 -0600

    multipath: check for NULL from udev_device_get_*

    The udev_device_get_* functions can return NULL, an occassionally do
    so in the multipathd code. multipath needs to check if the result
    is NULL before dereferencing it.

    Signed-off-by: Benjamin Marzinski <email address hidden>
----

Takes care of exactly this situation. This commit is found in the following versions:

----
inaddy@rafael(/home/inaddy/Codes/bugs/1532789/upstream/multipath-tools)$ git tag --contains e546687e046b1eb1cbe5567e9d7850cbbd682dba
0.5.0
----

And, from rmadison, this fix should be backported to Trusty & Vivid (not at 0.5.0 yet).

Changed in multipath-tools (Ubuntu):
status: New → In Progress
assignee: nobody → Rafael David Tinoco (inaddy)
description: updated
Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :

Precise being fixed together with:

https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1520192

SRU proposal.

Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :

Attaching the debdiff for the fix.

description: updated
tags: added: sts
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "trusty_multipath-tools_0.4.9-3ubuntu7.8.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Mathew Hodson (mhodson)
Changed in multipath-tools (Ubuntu):
importance: Undecided → High
Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :

Mathieu,

Please take a look at the following bug:

https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1535898

It looks like Trusty debian/patches are suffering a regression. This might have to be fixed together.

Mathew Hodson (mhodson)
Changed in multipath-tools (Ubuntu Trusty):
importance: Undecided → High
Mathew Hodson (mhodson)
Changed in multipath-tools (Ubuntu Trusty):
status: New → Triaged
Changed in multipath-tools (Ubuntu):
assignee: Rafael David Tinoco (inaddy) → nobody
Louis Bouchard (louis)
Changed in multipath-tools (Ubuntu Trusty):
assignee: nobody → Louis Bouchard (louis-bouchard)
Louis Bouchard (louis)
tags: added: sts-sru
Louis Bouchard (louis)
Changed in multipath-tools (Ubuntu):
status: In Progress → Invalid
Changed in multipath-tools (Ubuntu Trusty):
status: Triaged → In Progress
Revision history for this message
Louis Bouchard (louis) wrote :

Hello,

After verification onsite, the patch listed in comment #2 does fix the segfault issue so I am SRUing the fix.

Revision history for this message
Chris J Arges (arges) wrote :

Can you re-do the patch so quilt refresh makes the debdiff clearer?

Revision history for this message
Louis Bouchard (louis) wrote :

Hi,

Just re-uploaded a cleaner source pkg. I really didn't have my mind when I tried to force the patch to be numbered sequentially. Sorry for that.

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Rafael, or anyone else affected,

Accepted multipath-tools into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/multipath-tools/0.4.9-3ubuntu7.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in multipath-tools (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Louis Bouchard (louis) wrote :

Fix verified on platform that was reproducing the bug

tags: added: verification-done
removed: verification-needed
Louis Bouchard (louis)
tags: added: verification-needed
removed: verification-done
Revision history for this message
Martin Pitt (pitti) wrote :

Louis, you said you verified, but then you reverted the tag back to v-needed; was that on purpose?

Revision history for this message
Gábor Mészáros (gabor.meszaros) wrote :

Martin,

The revert was intentional, due to a minor misunderstanding, which we clarified.

Revision history for this message
Louis Bouchard (louis) wrote :

Verified on the platform having the issue

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package multipath-tools - 0.4.9-3ubuntu7.12

---------------
multipath-tools (0.4.9-3ubuntu7.12) trusty; urgency=medium

  * 0044-multipath-check-for-NULL-from-udev_device_get.patch (LP: #1532789)

 -- Rafael David Tinoco <email address hidden> Thu, 12 May 2016 11:03:37 +0200

Changed in multipath-tools (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Martin Pitt (pitti) wrote : Update Released

The verification of the Stable Release Update for multipath-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.