IBM DS3400 Will Not Bring Up Second Path

@Joe

Please attach the latest multipath.conf and lvm.conf responsible for this
stream of messages. If you could move over that logs and other stuff
from salesforce into this bug that would be ideal. Thanks.

multipath.conf

syslog

The lvm.conf file has not been modified from what ships with Ubuntu.

So the paths not showing up with a config file error, a few well
placed regexp and a proper blacklist to filter out the internal disks
solved that. Also added scsi_dh_rdac to initial ramdisk. Now the paths
are up, and stay up. However when we tried testing failover using
a scsi "delete" of the primary member. The secondary never became
active.

We asked the customer to check with any additional SAN side
config, like automatic lun transfer, is necessary. Depending on how
that turns it we may actually have a real failover bug.

Working multipath config, taken almost verbatim from the IBM
documentation: Section 5-10.
ftp://ftp.software.ibm.com/systems/support/system_x_pdf/94y8402.pdf

# Canonical Customer Support: 08/12/11
# Version 1.2

blacklist {
  devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
  devnode "^hd[a-z]"
  # Ignore internal storage from the LSI RAID
  device {
    vendor LSILOGIC
    product .*
  }
}

devices {
  device {
    vendor "IBM"
    product ".*"
    path_grouping_policy group_by_prio
    path_checker rdac
    getuid_callout "/lib/udev/scsi_id -g -u -d /dev/%n"
    prio_callout "/sbin/mpath_prio_rdac /dev/%n"
    hardware_handler "1 rdac"
    failback immediate
# rr_weight uniform
    no_path_retry 30
    rr_min_io 100
    features "2 pg_init_retries 50"
  }
}

In an addition to the MP issues, LVM was misconfigured. We attempt
to repair this on one of the blades but the config file just wouldn't *take*.
I suspect there was something else going, especially since we didn't
even have direct access to the console and was forced to relay
directives to the customer to execute.

I set up a mock test environment to test the the filtering by
PCI device.

root@kickseed:/dev/disk/by-path# ls -l
total 0
lrwxrwxrwx 1 root root 8 2011-08-15 10:04 pci-0000:00:00.9-scsi-0:0:0:2 -> /dev/sdc
lrwxrwxrwx 1 root root 9 2011-08-15 10:04 pci-0000:00:00.9-scsi-0:0:0:2-part1 -> /dev/sdc1
lrwxrwxrwx 1 root root 8 2011-08-15 10:04 pci-0000:00:00.9-scsi-0:0:0:3 -> /dev/sdd
lrwxrwxrwx 1 root root 9 2011-08-15 10:04 pci-0000:00:00.9-scsi-0:0:0:3-part1 -> /dev/sdd1
lrwxrwxrwx 1 root root 8 2011-08-15 10:02 pci-0000:00:1f.2-scsi-0:0:0:0 -> /dev/sda
lrwxrwxrwx 1 root root 9 2011-08-15 10:03 pci-0000:00:1f.2-scsi-0:0:0:0-part1 -> /dev/sda1
lrwxrwxrwx 1 root root 8 2011-08-15 10:03 pci-0000:00:1f.2-scsi-0:0:0:1 -> /dev/sdb
lrwxrwxrwx 1 root root 9 2011-08-15 10:03 pci-0000:00:1f.2-scsi-0:0:0:1-part1 -> /dev/sdb1

Then, I commented out the existing "filter" directive on /etc/lvm/lvm.conf and tested:
    filter = ["a|/dev/disk/by-path/pci-0000:00:00.9-scsi.*|", "r/.*/" ]

and

    filter = ["a|/dev/disk/by-path/pci-0000:00:1f.2-scsi.*|", "r/.*/" ]

lvmdiskscan confirms the filter directives.

root@kickseed:/dev/disk/by-path# lvmdiskscan
  /dev/disk/by-path/pci-0000:00:1f.2-scsi-0:0:0:0-part1 [ 7.45 GiB]
  /dev/disk/by-path/pci-0000:00:1f.2-scsi-0:0:0:1-part1 [ 10.00 GiB]
  0 disks
  2 partitions
  0 LVM physical volume whole disks
  0 LVM physical volumes

root@kickseed:/dev/disk/by-path# lvmdiskscan
  /dev/disk/by-path/pci-0000:00:00.9-scsi-0:0:0:2-part1 [ 10.00 GiB]
  /dev/disk/by-path/pci-0000:00:00.9-scsi-0:0:0:3-part1 [ 10.00 GiB]
  0 disks
  2 partitions
  0 LVM physical volume whole disks
  0 LVM physical volumes

I'm going to leave this part of the effort to customer support. There are
no bugs in LVM filtering, simply execution issues.