multipath -l can be very slow

Bug #1487169 reported by Walt Boring
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
os-brick
Fix Released
Medium
Kendall Nelson

Bug Description

When there are many volumes attached to a host, running multipath -l <device> to discover the multipath device and it's paths can be extremely slow.

Changed in os-brick:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to os-brick (master)

Reviewed: https://review.openstack.org/213389
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=3ea86f7d605d7109f4e13c1566ba1a14dc856443
Submitter: Jenkins
Branch: master

commit 3ea86f7d605d7109f4e13c1566ba1a14dc856443
Author: Walter A. Boring IV <email address hidden>
Date: Fri Aug 14 17:39:15 2015 -0700

    FC Stop calling multipath command line

    This patch changes how we discover Multipath devices for
    FibreChannel volume attaches.

    Running multipath -l <device> can become slower and slower
    as more and more volumes are attached to a host. To overcome this,
    there are ways of discovering multipath device paths without
    using the multipath -l command at all.

    When multipath daemon is running, and it discovers new volumes,
    it will create new device paths for the multipath device associated
    with that new volume. Those multipath device paths are predictable
    and show up after the multipath device is created. This avoids
    the repeated looping calls to multipath -l to discover the same paths.

    SCSI volumes have a WWN that's supposed to be in page 0x83 on the volume
    itself according to the SCSI SPC-3 spec. That WWN is where the multipath
    daemon gets it's multipath ID from and what is used to create the predictable
    multipath device paths on the system.

    When multipath friendly names are disabled, you get paths of
     /dev/disk/by-id/dm-uuid-mpath-<WWN>
     /dev/disk/by-id/scsi-<WWN>
     /dev/mapper/<WWN>

    When multipath friendly names are enabled, you get paths of
     /dev/disk/by-id/dm-uuid-mpath-<WWN>
     /dev/disk/by-id/dm-name-mpath<N>
     /dev/disk/by-id/scsi-mpath<N>
     /dev/mapper/mpath<N>

    This patch does 3 different attempts to find a multipath device path to
    use.

    First it looks in the common location of:
     /dev/disk/by-id/dm-uuid-mpath-<WWN>

    Then in the non friendly name path of:
     /dev/mapper/<WWN>

    And lastly using the fallback of calling multipath -l <device> to get:
     /dev/mapper/mpath<N>

    Partial-Bug: 1487169
    Change-Id: I9a9fffcb6882b1c2750b1e7927475093bde36d04

Revision history for this message
Walt Boring (walter-boring) wrote :

This one is still marked as in progress, because we have to fix the iSCSI connector to do the same fallback mechanism as the FC driver.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on os-brick (master)

Change abandoned by Sean McGinnis (<email address hidden>) on branch: master
Review: https://review.openstack.org/230391
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
Walt Boring (walter-boring) wrote :

Kendall is taking on the iSCSI version of this fix.

Changed in os-brick:
assignee: Walt Boring (walter-boring) → Kendall Nelson (kjnelson)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Keiichi KII (<email address hidden>) on branch: master
Review: https://review.openstack.org/230391
Reason: Sorry for my misoperation.
To remove multipath calling for iSCSI connector, we need to remove unneeded in_use flag checking for multipath.
The new patch is not related to this review.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to os-brick (master)

Fix proposed to branch: master
Review: https://review.openstack.org/267085

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to os-brick (master)

Reviewed: https://review.openstack.org/267085
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=ba2100a83f674fb5d86471afa829ec4dd29653d0
Submitter: Jenkins
Branch: master

commit ba2100a83f674fb5d86471afa829ec4dd29653d0
Author: Kendall Nelson <email address hidden>
Date: Tue Jan 12 14:12:27 2016 -0600

    Remove multipath -l logic from ISCSI connector

    This patch continues work on making the connect_volume methods
    more efficient. Using multipath -l to find the available multipath
    devices can take a lot of time listing all of the potential
    devices. Previously, the FC connector had been modified to skip
    the use of multipath -l.

    This patch takes the FC code and makes it into a generic method to
    be used in the ISCSI connector and the FC connector and updates tests.

    I also fixed argument order in assertEquals() in the
    test_connect_volume_with_multipath() function.

    Closes-Bug: #1487169
    Change-Id: If480df7c17a6f1c2ecebc0e00f5938d3056776fd

Changed in os-brick:
status: In Progress → Fix Released
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/os-brick 1.0.0

This issue was fixed in the openstack/os-brick 1.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on os-brick (master)

Change abandoned by Sean McGinnis (<email address hidden>) on branch: master
Review: https://review.openstack.org/264678
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.