Ubuntu
s390-tools package

fail to activate the second DS8K fcp device permanently

Bug #1683808 reported by bugproxy
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Fix Released
Medium
Unassigned
s390-tools (Ubuntu)
Fix Released
Medium
Dimitri John Ledkov
Ubuntu zesty-updates
Xenial
Fix Released
Medium
Dimitri John Ledkov
Ubuntu ubuntu-16.04.3
Yakkety
Fix Released
Medium
Dimitri John Ledkov
Ubuntu yakkety-updates
Zesty
Fix Released
Medium
Dimitri John Ledkov
Ubuntu zesty-updates
Artful
Fix Released
Medium
Dimitri John Ledkov
Ubuntu ubuntu-17.05

Bug Description

[Impact]

 * out of order devices fail to activate

[Test Case]

 * See steps to reproduce below

[Regression Potential]

 * s390-tools rebuild, includes rebuilding the bootloader, thus failure to boot should be tested in case of a no-change rebuild regression. The fix for the issue in question is a pure ordering issue, thus ordering the ids should resolve the issue at hand without any other negative side-effects

[Other Info]

 * Original report:

---Problem Description---
Fail to active the second DS8K fcp lun permanently

Machine Type = z13 lpar

---Debugger---
A debugger is not configured

---Steps to Reproduce---
 1. Map tow fcp luns from DS8880:
   root@ilz62u:~# lsluns
   Scanning for LUNs on adapter 0.0.1800
 at port 0x50050763070057f9:
  0x4000402800000000
  0x4001401000000000
   Scanning for LUNs on adapter 0.0.1900
 at port 0x50050763070b17f9:
  0x4000402800000000
  0x4001401000000000
2. active the two fcp temporarily
echo 0x4001401000000000 > /sys/bus/ccw/drivers/zfcp/0.0.1800/0x50050763070057f9/unit_add
  echo 0x4001401000000000 > /sys/bus/ccw/drivers/zfcp/0.0.1900/0x50050763070b17f9/unit_add

3. active them via tool "chzdev" permanently
   root@ilz62u:~# lszdev | grep 57f9
zfcp-lun 0.0.1800:0x50050763070057f9:0x4000402800000000 yes yes sdas sg44
zfcp-lun 0.0.1800:0x50050763070057f9:0x4001401000000000 yes yes sdau sg46

4. multipath show
root@ilz62u:~# multipath -ll | grep 2107s
d2ilsd2107s (36005076307ffd7f90000000000000110) dm-15 IBM,2107900
d1ilsd2107s (36005076307ffd7f90000000000000028) dm-14 IBM,2107900

5. reboot

6. the secod fcp lun "0x4001401000000000" fail to keep active any more
root@ilz62u:/sys/bus/ccw/drivers/zfcp/0.0.1800# ls 0x50050763070057f9
0x4000402800000000 access_denied failed in_recovery power status uevent unit_add unit_remove
root@ilz62u:/sys/bus/ccw/drivers/zfcp/0.0.1900# ls 0x50050763070b17f9
0x4000402800000000 access_denied failed in_recovery power status uevent unit_add unit_remove

root@ilz62u:/sys/bus/ccw/drivers/zfcp/0.0.1900# multipath -ll | grep 2107s
d1ilsd2107s (36005076307ffd7f90000000000000028) dm-14 IBM,2107900

Stack trace output:
 no

Oops output:
 no

System Dump Info:
  The system is not configured to capture a system dump.

== Comment: #1 - Heinz-Werner Seeck <email address hidden> - 2017-04-18 03:08:54 ==

First problem evaluation:
Could be a chzdev problem. Seeing two jump-labels in the udev rule for port 0x50050763070b17f9 on fcp device 0x1800 and 0x1900. The first label enables the LUN that is working, while the second one with the LUN number that is not working is likely ignored.

Confirmed as a bug in chzdev!

Workaround:
Use chzdev to configure and immediately deconfigure a non-existent FCP LUN for the FCP devices that displays the problem.

Example (ignore any warnings that might show up):

For FCP device 0x1800:

chzdev -e -p zfcp-lun 0x1800:0x0000000000000000:0x0000000000000000
chzdev -d -p zfcp-lun 0x1800:0x0000000000000000:0x0000000000000000

and for FCP device 0x1900:

chzdev -e -p zfcp-lun 0x1900:0x0000000000000000:0x0000000000000000
chzdev -d -p zfcp-lun 0x1900:0x0000000000000000:0x0000000000000000

Solution available : Patch applied:
Upstream patch, also applies to s390-tools v1.37.0 and v1.37.1.

See original description
Revision history for this message
bugproxy (bugproxy) wrote : sosrepoter

Default Comment by Bridge

tags: added: architecture-s39064 bugnameltc-153462 severity-medium targetmilestone-inin---
Revision history for this message
bugproxy (bugproxy) wrote : Upstream fix

Default Comment by Bridge

Changed in ubuntu:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu → linux (Ubuntu)
Revision history for this message
bugproxy (bugproxy) wrote : sosrepoter

Default Comment by Bridge

tags: added: targetmilestone-inin1704
removed: targetmilestone-inin---
Revision history for this message
bugproxy (bugproxy) wrote : Upstream fix

Default Comment by Bridge

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2017-04-18 10:09 EDT-------
This patch could also be applied to Xenial/Yaketty s390tools ... !

Joseph Salisbury (jsalisbury)
tags: added: kernel-da-key
Changed in linux (Ubuntu):
importance: Undecided → Medium
Dimitri John Ledkov (xnox)
affects: linux (Ubuntu) → s390-tools (Ubuntu)
Changed in s390-tools (Ubuntu Yakkety):
importance: Undecided → Medium
Changed in s390-tools (Ubuntu Xenial):
importance: Undecided → Medium
Changed in s390-tools (Ubuntu Zesty):
milestone: none → zesty-updates
Changed in s390-tools (Ubuntu Yakkety):
milestone: none → yakkety-updates
Changed in s390-tools (Ubuntu Xenial):
milestone: none → xenial-updates
Changed in s390-tools (Ubuntu Yakkety):
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in s390-tools (Ubuntu Xenial):
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in s390-tools (Ubuntu Zesty):
assignee: Skipper Bug Screeners (skipper-screen-team) → Dimitri John Ledkov (xnox)
status: New → Triaged
Changed in s390-tools (Ubuntu Yakkety):
status: New → Triaged
Changed in s390-tools (Ubuntu Xenial):
status: New → Triaged
milestone: xenial-updates → ubuntu-16.04.3
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package s390-tools - 1.37.0-0ubuntu4

---------------
s390-tools (1.37.0-0ubuntu4) artful; urgency=medium

  * chzdev: Fix ineffective zfcp-lun udev rule preventing activation of
    unordered WWPNs. LP: #1683808

 -- Dimitri John Ledkov <email address hidden> Wed, 03 May 2017 12:25:08 +0100

Changed in s390-tools (Ubuntu Artful):
status: Triaged → Fix Released
Dimitri John Ledkov (xnox)
description: updated
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Awaiting approval by SRU team

s390-tools (1.34.0-0ubuntu8.4) xenial; urgency=medium

  * chzdev: Fix ineffective zfcp-lun udev rule preventing activation of
    unordered WWPNs. LP: #1683808

Date: Thu, 06 Jul 2017 14:14:51 +0100
Changed-By: Dimitri John Ledkov <email address hidden>
Maintainer: Ubuntu Developers <email address hidden>
https://launchpad.net/ubuntu/+source/s390-tools/1.34.0-0ubuntu8.4

==

 OK: s390-tools_1.34.0.orig.tar.bz2
 OK: s390-tools_1.34.0-0ubuntu8.4.debian.tar.xz
 OK: s390-tools_1.34.0-0ubuntu8.4.dsc
     -> Component: main Section: admin

Upload Warnings:
Redirecting ubuntu xenial to ubuntu xenial-proposed.
This upload awaits approval by a distro manager

s390-tools (1.36.1-0ubuntu2.2) yakkety; urgency=medium

  * chzdev: Fix ineffective zfcp-lun udev rule preventing activation of
    unordered WWPNs. LP: #1683808

Date: Thu, 06 Jul 2017 14:11:42 +0100
Changed-By: Dimitri John Ledkov <email address hidden>
Maintainer: Ubuntu Developers <email address hidden>
https://launchpad.net/ubuntu/+source/s390-tools/1.36.1-0ubuntu2.2

==

 OK: s390-tools_1.36.1.orig.tar.bz2
 OK: s390-tools_1.36.1-0ubuntu2.2.debian.tar.xz
 OK: s390-tools_1.36.1-0ubuntu2.2.dsc
     -> Component: main Section: admin

Upload Warnings:
Redirecting ubuntu yakkety to ubuntu yakkety-proposed.
This upload awaits approval by a distro manager

s390-tools (1.37.0-0ubuntu3.1) zesty; urgency=medium

  * chzdev: Fix ineffective zfcp-lun udev rule preventing activation of
    unordered WWPNs. LP: #1683808

Date: Thu, 06 Jul 2017 14:08:32 +0100
Changed-By: Dimitri John Ledkov <email address hidden>
Maintainer: Ubuntu Developers <email address hidden>
https://launchpad.net/ubuntu/+source/s390-tools/1.37.0-0ubuntu3.1

==

 OK: s390-tools_1.37.0.orig.tar.bz2
 OK: s390-tools_1.37.0-0ubuntu3.1.debian.tar.xz
 OK: s390-tools_1.37.0-0ubuntu3.1.dsc
     -> Component: main Section: admin

Upload Warnings:
Redirecting ubuntu zesty to ubuntu zesty-proposed.
This upload awaits approval by a distro manager

Changed in s390-tools (Ubuntu Xenial):
status: Triaged → In Progress
Changed in s390-tools (Ubuntu Yakkety):
status: Triaged → In Progress
Changed in s390-tools (Ubuntu Zesty):
status: Triaged → In Progress
Changed in ubuntu-z-systems:
status: Triaged → In Progress
Revision history for this message
Andy Whitcroft (apw) wrote : Please test proposed package

Hello bugproxy, or anyone else affected,

Accepted s390-tools into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/s390-tools/1.37.0-0ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in s390-tools (Ubuntu Zesty):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-zesty
Changed in s390-tools (Ubuntu Yakkety):
status: In Progress → Fix Committed
tags: added: verification-needed-yakkety
Revision history for this message
Andy Whitcroft (apw) wrote :

Hello bugproxy, or anyone else affected,

Accepted s390-tools into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/s390-tools/1.36.1-0ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-yakkety to verification-done-yakkety. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-yakkety. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in s390-tools (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed-xenial
Revision history for this message
Andy Whitcroft (apw) wrote :

Hello bugproxy, or anyone else affected,

Accepted s390-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/s390-tools/1.34.0-0ubuntu8.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: In Progress → Fix Committed
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2017-07-17 21:04 EDT-------
Sir, our test environment is moving to z14, after that I will test the fix asap.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Started zesty container with s390-tools 1.37.0-0ubuntu3. Reproduced the problem using the simplified upstream test case:

$ chzdev -e -p zfcp-lun 0x0000:0x0000000000000000:0x0000000000000000
$ chzdev -e -p zfcp-lun 0x0000:0x0000000000000001:0x0000000000000000
$ chzdev -e -p zfcp-lun 0x0000:0x0000000000000000:0x0000000000000001

The generated rules were:

LABEL="start_zfcp_lun_0.0.0000"
SUBSYSTEM=="fc_remote_ports", ATTR{port_name}=="0x0000000000000000", GOTO="cfg_fc_0.0.0000_0x0000000000000000"
SUBSYSTEM=="fc_remote_ports", ATTR{port_name}=="0x0000000000000001", GOTO="cfg_fc_0.0.0000_0x0000000000000001"
SUBSYSTEM=="fc_remote_ports", ATTR{port_name}=="0x0000000000000000", GOTO="cfg_fc_0.0.0000_0x0000000000000000"
GOTO="end_zfcp_lun_0.0.0000"

LABEL="cfg_fc_0.0.0000_0x0000000000000000"
ATTR{[ccw/0.0.0000]0x0000000000000000/unit_add}="0x0000000000000000"
GOTO="end_zfcp_lun_0.0.0000"

LABEL="cfg_fc_0.0.0000_0x0000000000000001"
ATTR{[ccw/0.0.0000]0x0000000000000001/unit_add}="0x0000000000000000"
GOTO="end_zfcp_lun_0.0.0000"

LABEL="cfg_fc_0.0.0000_0x0000000000000000"
ATTR{[ccw/0.0.0000]0x0000000000000000/unit_add}="0x0000000000000001"
GOTO="end_zfcp_lun_0.0.0000"

LABEL="end_zfcp_lun_0.0.0000"

Observe how duplicate ..0 labels and goto end result in inability to reach the unit_add}="0x0000000000000001" paragraph.

Upgraded to 1.37.0-0ubuntu3.1. Disabled all of these devices and re-eneabled them.

$ chzdev -d -p zfcp-lun 0x0000:0x0000000000000000:0x0000000000000000
$ chzdev -d -p zfcp-lun 0x0000:0x0000000000000001:0x0000000000000000
$ chzdev -d -p zfcp-lun 0x0000:0x0000000000000000:0x0000000000000001
$ chzdev -e -p zfcp-lun 0x0000:0x0000000000000000:0x0000000000000000
$ chzdev -e -p zfcp-lun 0x0000:0x0000000000000001:0x0000000000000000
$ chzdev -e -p zfcp-lun 0x0000:0x0000000000000000:0x0000000000000001

The new rules combine the ...0 and ..1 devices under non-duplicate label, thus all states are now reachable.

LABEL="start_zfcp_lun_0.0.0000"
SUBSYSTEM=="fc_remote_ports", ATTR{port_name}=="0x0000000000000000", GOTO="cfg_fc_0.0.0000_0x0000000000000000"
SUBSYSTEM=="fc_remote_ports", ATTR{port_name}=="0x0000000000000001", GOTO="cfg_fc_0.0.0000_0x0000000000000001"
GOTO="end_zfcp_lun_0.0.0000"

LABEL="cfg_fc_0.0.0000_0x0000000000000000"
ATTR{[ccw/0.0.0000]0x0000000000000000/unit_add}="0x0000000000000000"
ATTR{[ccw/0.0.0000]0x0000000000000000/unit_add}="0x0000000000000001"
GOTO="end_zfcp_lun_0.0.0000"

LABEL="cfg_fc_0.0.0000_0x0000000000000001"
ATTR{[ccw/0.0.0000]0x0000000000000001/unit_add}="0x0000000000000000"
GOTO="end_zfcp_lun_0.0.0000"

LABEL="end_zfcp_lun_0.0.0000"

zesty verification is done.

tags: added: verification-done-zesty
removed: verification-needed-zesty
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Started yakkety container with s390-tools 1.36.1-0ubuntu2. Using same test-case as zesty, getting bad udev rules.
Upgrading to 1.36.1-0ubuntu2.2. Rerunning the test case. Getting correct udev rules with non-duplicate labels.

tags: added: verification-done-yakkety
removed: verification-needed-yakkety
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Started xenial container with s390-tools 1.34.0-0ubuntu8.2. Using the same test case as on yakkety, getting bad udev rules. Upgrading to 1.34.0-0ubuntu8.4 and the rules now are generated correctly without duplicate labels. Verification is completed.

tags: added: verification-done verification-done-xenial
removed: verification-needed verification-needed-xenial
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for s390-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package s390-tools - 1.37.0-0ubuntu3.1

---------------
s390-tools (1.37.0-0ubuntu3.1) zesty; urgency=medium

  * chzdev: Fix ineffective zfcp-lun udev rule preventing activation of
    unordered WWPNs. LP: #1683808

 -- Dimitri John Ledkov <email address hidden> Thu, 06 Jul 2017 14:08:32 +0100

Changed in s390-tools (Ubuntu Zesty):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package s390-tools - 1.34.0-0ubuntu8.4

---------------
s390-tools (1.34.0-0ubuntu8.4) xenial; urgency=medium

  * chzdev: Fix ineffective zfcp-lun udev rule preventing activation of
    unordered WWPNs. LP: #1683808

 -- Dimitri John Ledkov <email address hidden> Thu, 06 Jul 2017 14:14:51 +0100

Changed in s390-tools (Ubuntu Xenial):
status: Fix Committed → Fix Released
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package s390-tools - 1.36.1-0ubuntu2.2

---------------
s390-tools (1.36.1-0ubuntu2.2) yakkety; urgency=medium

  * chzdev: Fix ineffective zfcp-lun udev rule preventing activation of
    unordered WWPNs. LP: #1683808

 -- Dimitri John Ledkov <email address hidden> Thu, 06 Jul 2017 14:11:42 +0100

Changed in s390-tools (Ubuntu Yakkety):
status: Fix Committed → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2017-08-01 04:18 EDT-------
*** Bug 157109 has been marked as a duplicate of this bug. ***

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2017-08-01 08:49 EDT-------
*** Bug 157109 has been marked as a duplicate of this bug. ***

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2017-08-01 09:05 EDT-------
IBM Bugzilla Status -> CLOSED

Francis Ginther (fginther)
tags: added: id-5967809caa3068937bd9d599
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.