open-iscsi +mpio with multipathd init script order errors

Bug #1135453 reported by Mike Burgener
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
multipath-tools (Ubuntu)
Invalid
Undecided
Peter Petrakis

Bug Description

when using open-iscsi and multipathd for a mpio setup there are several init script logical issues:

when shutting down, the system does first stop multipathd and then try to umount the filesystem and then stop open-iscsi to the system hangs forever on shutdown.

also when booting up it does mount the partition before multipathd and open-iscsi are ready and you get the ubuntu screen that the partition could not get mounted, do you want to skip.

after bootup process you can however mount the partition without any issue.

Tags: bot-comment
Revision history for this message
Mike Burgener (mburgener) wrote :

actually its on release 12.04

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1135453/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

tags: added: bot-comment
affects: ubuntu → multipath-tools (Ubuntu)
Revision history for this message
Ritesh Raj Sarraf (rrs) wrote : Re: [Bug 1135453] [NEW] open-iscsi +mpio with multipathd init script order errors

On Thu, Feb 28, 2013 at 6:51 PM, Launchpad Bug Tracker <
<email address hidden>> wrote:

> You have been subscribed to a public bug:
>
> when using open-iscsi and multipathd for a mpio setup there are several
> init script logical issues:
>
> when shutting down, the system does first stop multipathd and then try
> to umount the filesystem and then stop open-iscsi to the system hangs
> forever on shutdown.
>

Killing the daemon shouldn't have direct impact on umount of the map,
unless all the paths for that map go offline. If every operation goes in
parallel (with my understanding of the new gen init services like upstart
and systemd), and iscsi sessions get dropped before the umount, yes, you
definitely will run into the hang situation.

Again, the hang situation depends on how your multipath policy is
configured.

>
> also when booting up it does mount the partition before multipathd and
> open-iscsi are ready and you get the ubuntu screen that the partition
> could not get mounted, do you want to skip.
>
> after bootup process you can however mount the partition without any
> issue.
>

It is hard to make out what might be the cause from just the english
explanation. Perhaps the multipath maintainer here can translate this to *Steps
To Reproduce* and then investigate it further.

> ** Affects: multipath-tools (Ubuntu)
> Importance: Undecided
> Status: New
>
>
> ** Tags: bot-comment
> --
> open-iscsi +mpio with multipathd init script order errors
> https://bugs.launchpad.net/bugs/1135453
> You received this bug notification because you are subscribed to
> multipath-tools in Ubuntu.
>

--
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."

Revision history for this message
Mike Burgener (mburgener) wrote :

if i remember correctly, i made the config from the HP MSA recommendations for multipathd.conf

here my multipathd.conf config:

defaults {
         udev_dir /dev
         polling_interval 30
         selector "round-robin 0"
         path_grouping_policy multibus
         getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
         prio_callout /bin/true
         path_checker readsector0
         prio const
         rr_min_io 100
         rr_weight uniform
         failback immediate
         no_path_retry 12
         user_friendly_name yes
         hardware_handler "0"
}
blacklist {
         devnode "sda"
         devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
         devnode "^hd[a-z][[0-9]*]"
         devnode "^vd[a-z]"
         devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"
}
multipaths {
         multipath {
                 wwid mypseudorandomwwid901243012701704
                 alias mydevname
         }
}

Revision history for this message
Mike Burgener (mburgener) wrote :

here my iscsi config dir from /etc

Revision history for this message
Ritesh Raj Sarraf (rrs) wrote : Re: [Bug 1135453] Re: open-iscsi +mpio with multipathd init script order errors

On Friday 01 March 2013 05:49 PM, Mike Burgener wrote:
> defaults {
> udev_dir /dev
> polling_interval 30
> selector "round-robin 0"
> path_grouping_policy multibus
> getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
> prio_callout /bin/true
> path_checker readsector0
> prio const
> rr_min_io 100
> rr_weight uniform
> failback immediate
> no_path_retry 12
> user_friendly_name yes
> hardware_handler "0"
> }

The behavior you reported is pretty interesting. You reported that you
suffered the hang state because the umount occurred after the multipathd
and iscsid processes were killed by the init scripts (which I also
assumed given the new init systems, upstart and systemd).

But looking at your config, you should not typically see the hang
scenario. I was suspecting that you might be using the multipath
queue_if_no_path feature, but that is not the case here. The iscsi
replacement timeouts are also the defaults, i.e. 120 seconds, which
would mean that after 120 seconds of retry, iscsi will error out and the
errors will propagate to the upper layers.

Ideally, with the configuration settings that you have, you should see
the errors show up after 120 seconds, and NOT hang. I have no more ideas
on what might be going wrong in your setup. Perhaps the Ubuntu multipath
maintainer might have some insight to this.

--
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."

Revision history for this message
Mike Burgener (mburgener) wrote :

you seem to feel that it is normal that multipathd is stopped then the iscsi device logged out (so the blockdevice disappears) and then umount? this would end in a possible dataloss scenario isn't it?

btw. also bootup does not work so there must be some logical error i think.

of course the 120 hung timeout message from kernel arrives but i think it does never continue as it can no more "sync" disks? could that make sense?

regards

Mike

Revision history for this message
Ritesh Raj Sarraf (rrs) wrote :

On Fri, Mar 1, 2013 at 11:08 PM, Mike Burgener <email address hidden>wrote:

> you seem to feel that it is normal that multipathd is stopped then the
> iscsi device logged out (so the blockdevice disappears) and then umount?
>

No. The correct steps are to first umount the device, then flush the
multipath map, and then depending on the transport (iSCSI, FC or FCoE) act
further.
Never should we do it any other way. It is just asking for too much trouble.

But technically iscsid or multipatd daemon processes are not the core
performers. If you look at my Debian packages regular system v init
scripts, you'll see that killing the daemon and terminating the iscsi
sessions are 2 different tasks [1]. That's what I meant in my previous
mails.

[1]
http://anonscm.debian.org/gitweb/?p=pkg-iscsi/open-iscsi.git;a=blob;f=debian/open-iscsi.init;h=221fc9147f684bdb0bbcbda36799d5867bc617f2;hb=HEAD

this would end in a possible dataloss scenario isn't it?
>

Hard to predict. But yes, there could be a rare chance.

>
> btw. also bootup does not work so there must be some logical error i
> think.
>

Would it be possible for you to test this out on a sysv init? Or maybe even
try this on Debian? (But do note that the Ubuntu packages are not directly
identical to Debian ones).

> of course the 120 hung timeout message from kernel arrives but i think
> it does never continue as it can no more "sync" disks? could that make
> sense?
>

That might be the hung_task kernel messages. Yes.

>
> regards
>
> Mike
>
> --
> You received this bug notification because you are subscribed to
> multipath-tools in Ubuntu.
> https://bugs.launchpad.net/bugs/1135453
>
> Title:
> open-iscsi +mpio with multipathd init script order errors
>
> Status in “multipath-tools” package in Ubuntu:
> New
>
> Bug description:
> when using open-iscsi and multipathd for a mpio setup there are
> several init script logical issues:
>
> when shutting down, the system does first stop multipathd and then try
> to umount the filesystem and then stop open-iscsi to the system hangs
> forever on shutdown.
>
> also when booting up it does mount the partition before multipathd and
> open-iscsi are ready and you get the ubuntu screen that the partition
> could not get mounted, do you want to skip.
>
> after bootup process you can however mount the partition without any
> issue.
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1135453/+subscriptions
>

--
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."

Changed in multipath-tools (Ubuntu):
assignee: nobody → Peter Petrakis (peter-petrakis)
Revision history for this message
Peter Petrakis (peter-petrakis) wrote :

This might be self inflicted. First, we have really good multipath documentation now.

You're better off following our docs first and then supplementing it with the vendor
docs if it doesn't work. Many popular SANs have built-in support.

https://help.ubuntu.com/12.04/serverguide/dm-multipath-chapter.html

HP MSA's are already included in the built-in DB, your config file is likely
not needed so lets start by eliminating that as a variable by following these
directions.

https://help.ubuntu.com/12.04/serverguide/multipath-setting-up-dm-multipath.html#multipath-setup-overview

plus your path alias, and,
# update-initramfs -u -k all

To copy the multipath.conf to the ramdisk.

Make all these changes, reboot, and before you shutdown again grab:
 * sudo dmsetup table
 * sudo udisks --dump
 * sudo -i; echo 'show config' | multipathd -k

and attach to the bug. Thanks.

I'm hoping you have a newer MSA and this is just a mp jumbled
config.

Changed in multipath-tools (Ubuntu):
status: New → Incomplete
Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :

Thank you for taking the time to file a bug report.

After last comment it seems that better categorization is 'invalid'.

Since it seems likely to me that this is a local configuration problem,
rather than a bug in Ubuntu, I am marking this bug as 'Invalid'.

However, if you believe that this is really a bug in Ubuntu, then we would
be grateful if you would provide a more complete description of the problem
with steps to reproduce, explain why you believe this is a bug in Ubuntu
rather than a problem specific to your system, and then change the bug
status back to "New".

For local configuration issues, you can find assistance here:
http://www.ubuntu.com/support/community

Changed in multipath-tools (Ubuntu):
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.