STC840.20:tuleta:tul516p01 formatting osprey for RAID fails

== Comment: #0 - Application Cdeadmin <email address hidden> - 2016-03-07 09:15:07 ==

== Comment: #1 - Application Cdeadmin <email address hidden> - 2016-03-07 09:15:10 ==
==== State: Open by: justinmc on 07 March 2016 08:09:05 ====

==========================Automatic entries==========================
Contact: McCright, Justin (Justin) *CONTRACTOR*
Backup: Thirukumaran V T (<email address hidden>), Deepti Umarani (<email address hidden>), Brian M. Carpenter(<email address hidden>)

System Name: tul516fp1
Type Model: 8247-22L
Serial #: 213F12A
FSP IP: 9.3.242.143 (tul516fp1.aus.stglabs.ibm.com)
FSP IP2: null ()
State: Operating
System Firmware Level:
Current Side Driver:.....fips840/b0224a_1608.840
Non-Current Side Driver:.fips840/b0222a_1608.840
System Refcode :

Lpars:
name,lpar_id,lpar_env,state,os_version
tul516p01,1,aixlinux,Running,Linux/Debian 4.4.0-9-generic Unknown
tul516p02,2,aixlinux,Running,Linux/Red Hat 3.10.0-327.el7.ppc64 7.2
tul516p08,8,vioserver,Open Firmware,Unknown

Lpar Refcode:
tul516p01,Linux ppc64le
tul516p02,Linux ppc64
tul516p08,AA00E1A9

Lpar Access: Please refer https://pcajet.austin.ibm.com/ for the lab password. (The Lab Test Passwords are now accessible only through the auto or manual install web apps. For example, from the manual install web app, enter your email address, check the Lab Passwords checkbox and then click on Submit.)

-------------------------------------------------------
HMC IP: 9.3.137.232 (tulhmc1.aus.stglabs.ibm.com)
HMC Version:
"version= Version: 8
 Release: 8.5.0
 Service Pack: 0
HMC Build level 20160215.1
","base_version=V8R8.5.0
"

Dumps in this HMC:
no dumps

Other systems in this HMC:
tul516fp1,Operating
tul503fp1,Operating

-------------------------------------------------------

LCB : http://vault.ibm.com/lcb

=================================Logs================================
FFDC :
na

HMC Log:
na

PHYP FFDC :
na

============================Manual entries===========================
Testcase : TER106369

Lpar under test : tul516p01

Problem Description :

Attempting to format the osprey in this partition for RAID fails:

                       Confirm Initialize and Format Disks

ATTENTION! System crash may occur if selected device is in use. Data loss will
occur on selected device. Proceed with caution.

Press 'c' to confirm your choice for 1=Initialize and format.
  q=Return to change your choice.

OPT Name Resource Path/Address Vendor Product ID Status
--- ------ -------------------------- -------- ------------------- -------------
 1 sdg 0:0:6:0 IBM PX04SRB192 Active

Initialize and format failed

I see no errors in the system log. Formatting the disk for JBOD works. Initializing the disk works. Formatting for RAID does not work after doing either of those.

The partition ran htx mdt.storage_4K_rand_read over the weekend on the osprey in JBOD mode without any errors.

I was able to initiate format for RAID on an AL14 on this partition.

htx save.it from the weekend run: /gsa/ausgsa/projects/s/sift/hst/trial_data/Tuleta/Ubuntu/Apollo_840/TER106369/FAIL201603070845

(130) root @ tul516p01: /root
# uname -a
Linux tul516p01 4.4.0-9-generic #24-Ubuntu SMP Mon Feb 29 19:28:28 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

(0) root @ tul516p01: /root
# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu Xenial Xerus (development branch)
Release: 16.04
Codename: xenial

Installed on 2016-03-04 in the afternoon.

(0) root @ tul516p01: /root
# iprconfig --version
iprconfig: 2.4.10.1

(0) root @ tul516p01: /root
# iprconfig -c show-details sg16
Manufacturer . . . . . . . . . . . . . . : IBM
Product ID . . . . . . . . . . . . . . . : PX04SRB192
Firmware Version . . . . . . . . . . . . : 37303038 (7008)
Serial Number. . . . . . . . . . . . . . : SUQALKYN
Capacity . . . . . . . . . . . . . . . . : 1920.39 GB
Resource Name. . . . . . . . . . . . . . : /dev/sdg

Physical location
PCI Address. . . . . . . . . . . . . . . : 0003:60:00.0
SCSI Host Number . . . . . . . . . . . . : 0
SCSI Channel . . . . . . . . . . . . . . : 0
SCSI Id. . . . . . . . . . . . . . . . . : 6
SCSI Lun . . . . . . . . . . . . . . . . : 0
Platform Location. . . . . . . . . . . . : U5887.001.G5CM00L-P1-D7

Extended Details
FRU Number . . . . . . . . . . . . . . . : 00LY373
EC Level . . . . . . . . . . . . . . . . :
Part Number. . . . . . . . . . . . . . . : 00LY371
Device Specific (Z0) . . . . . . . . . . : 000006329F001002
Device Specific (Z1) . . . . . . . . . . : 700870087008
Device Specific (Z2) . . . . . . . . . . : XMXM
Device Specific (Z3) . . . . . . . . . . : 15322
Device Specific (Z4) . . . . . . . . . . :
Device Specific (Z5) . . . . . . . . . . : 22
Device Specific (Z6) . . . . . . . . . . :

(0) root @ tul516p01: /root
# iprconfig -c show-config
Name PCI/SCSI Location Description Status
------ ------------------------- ------------------------- -----------------
       0003:60:00.0/0: PCI-E SAS RAID Adapter Operational
sdg 0003:60:00.0/0:0:6:0 Physical Disk Active
       0003:60:00.0/0:4:9:0 Physical Disk 31% Formatted
sda 0003:60:00.0/0:255:0:0 RAID 0 Array Optimized
       0003:60:00.0/0:0:0:0 RAID 0 Array Member Active
sdb 0003:60:00.0/0:255:1:0 RAID 0 Array Optimized
       0003:60:00.0/0:0:8:0 RAID 0 Array Member Active
sdc 0003:60:00.0/0:255:2:0 RAID 0 Array Optimized
       0003:60:00.0/0:4:0:0 RAID 0 Array Member Active
sdd 0003:60:00.0/0:255:3:0 RAID 0 Array Optimized
       0003:60:00.0/0:4:8:0 RAID 0 Array Member Active
sde 0003:60:00.0/0:255:4:0 RAID 0 Array Optimized
       0003:60:00.0/0:0:9:0 RAID 0 Array Member Active
       0003:60:00.0/0:0:38:0 Enclosure Active
       0003:60:00.0/0:2:38:0 Enclosure Active
       0003:60:00.0/0:4:38:0 Enclosure Active
       0003:60:00.0/0:6:38:0 Enclosure Active

(0) root @ tul516p01: /root
# iprconfig -c show-alt-config
Name Resource Path/Address Vendor Product ID Status
------ -------------------------- -------- ---------------- -----------------
sg0 0: IBM 57B3001SISIOA Operational
sg16 0:0:6:0 IBM PX04SRB192 Active
sg15 0:4:9:0 IBM AL14SEB030N 32% Formatted
sg6 0:255:0:0 IBM IPR-0 D51118F3 Optimized
sg2 0:0:0:0 IBM AL14SEB120N Active
sg7 0:255:1:0 IBM IPR-0 C6C132A6 Optimized
sg3 0:0:8:0 IBM AL14SEB120N Active
sg8 0:255:2:0 IBM IPR-0 2BC1D531 Optimized
sg5 0:4:0:0 IBM AL14SEB030N Active
sg9 0:255:3:0 IBM IPR-0 5F558780 Optimized
sg4 0:4:8:0 IBM AL14SEB030N Active
sg10 0:255:4:0 IBM IPR-0 30F9EC75 Optimized
sg1 0:0:9:0 IBM AL14SEB120N Active
sg13 0:0:38:0 IBM 5887 Active
sg14 0:2:38:0 IBM 5887 Active
sg11 0:4:38:0 IBM 5887 Active
sg12 0:6:38:0 IBM 5887 Active

===================================END===============================

== Comment: #3 - Justin A. McCright <email address hidden> - 2016-03-08 08:23:15 ==

== Comment: #6 - Application Cdeadmin <email address hidden> - 2016-03-08 08:59:57 ==
cde00 (<email address hidden>) added native attachment /tmp/AIXOS05616253/sosreport-tul516p01-20160308072115.tar.xz on 2016-03-08 07:41:53

== Comment: #14 - Application Cdeadmin <email address hidden> - 2016-03-17 12:43:17 ==
==== State: Open by: justinmc on 17 March 2016 11:32:08 ====

While investigating something else I broke tul516p01 and had to reinstall it. I have recreated the formatting issue on the fresh install.

== Comment: #15 - Gabriel Krisman Bertazi <email address hidden> - 2016-03-18 10:51:54 ==
Hello,

The problem seems to be that your adapter claims to not support 4K block size for AF devices, while your disk, PX04SRB192, is a 4K RI SSD. Since your adapter is a Cadet-E, 57B3 and is in the latest firmware version, I think this setup is not supported.

Inquiry page 0xD0 for your IOA:

?root@tul516p01:~/iprdd-iprutils/build# sg_inq /dev/sg0 --page=0xd0
? Only hex output supported. sg_vpd and sdparm decode more VPD pages.
?VPD INQUIRY, page code=0xd0:
?1f d0 00 14 00 00 04 02 a0 10 00 04 02 10 00 00 00 00 00 00 00 00 00 00

See that byte 10 shows 0x0, indicating that 4K is not supported.

== Comment: #16 - Application Cdeadmin <email address hidden> - 2016-03-18 12:53:56 ==
==== State: Assigned by: justinmc on 18 March 2016 11:40:03 ====

Can iprconfig provide a more useful error message in this case?

== Comment: #17 - Gabriel Krisman Bertazi <email address hidden> - 2016-03-22 09:28:33 ==
(In reply to comment #16)
>
> ==== State: Assigned by: justinmc on 18 March 2016 11:40:03 ====
>
> Can iprconfig provide a more useful error message in this case?

Hmm, the error message should already be there. Thanks for noticing. I think the reason for that is a bug when we read the device configuration.

Justin and I chatted on ST and I should test a patch for this, once his test system is back online.

Thanks,

== Comment: #18 - Gabriel Krisman Bertazi <email address hidden> - 2016-03-22 13:23:54 ==
(In reply to comment #17)
> (In reply to comment #16)
> >
> > ==== State: Assigned by: justinmc on 18 March 2016 11:40:03 ====
> >
> > Can iprconfig provide a more useful error message in this case?
>
> Hmm, the error message should already be there. Thanks for noticing. I
> think the reason for that is a bug when we read the device configuration.
>
> Justin and I chatted on ST and I should test a patch for this, once his test
> system is back online.
>
> Thanks,

Patch submitted to the ML: "iprutils: Mark devices as 4K even if IOA doesn't support 4K AF DASD."

If you want to test it, please use the iprconfig version available at:

/root/iprdd-iprutils/build/iprconfig

== Comment: #19 - Gabriel Krisman Bertazi <email address hidden> - 2016-03-28 09:33:55 ==
>
> Patch submitted to the ML: "iprutils: Mark devices as 4K even if IOA doesn't
> support 4K AF DASD."
>
> If you want to test it, please use the iprconfig version available at:
>
> /root/iprdd-iprutils/build/iprconfig

Patch is upstream:

commit 379bf9b449ddcb72a5c72bc1dd1327f58d2e2ff3
Author: Gabriel Krisman Bertazi <email address hidden>
Date: Fri Mar 25 14:31:43 2016 -0500

    iprutils: Mark devices as 4K even if IOA doesn't support 4K AF DASD

Please confirm it works for you.