IPMI USB SCSI endpoint discovery can fail on OpenPower machines

Bug #1509896 reported by Jeremy Kerr
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
ipmitool (Ubuntu)
Fix Released
Medium
Mathieu Trudel-Lapierre
Trusty
Fix Released
Medium
Mathieu Trudel-Lapierre
Vivid
Won't Fix
Medium
Mathieu Trudel-Lapierre
Wily
Won't Fix
Medium
Mathieu Trudel-Lapierre

Bug Description

[Impact]
The current ipmitool code can fail to find an IPMI device on my OpenPower machine, due to bug in the USB SCSI device discovery process.

This will only happen if the user has adjusted the number of virtual USB devices that the BMC exports. However, the failure mode isn't obvious, and it prevents the user from using the 'USB' IPMI channel, which is required for firmware upgrades.

This is fixed in upstream commit 607cfe6f4, can we backport this to the Ubuntu ipmitool for versions at 1.8.15 or earlier?

[Test case]
1- Increase number of virtual USB devices exported by BMC
2- Run ipmitool -I usb lan print

[Regression Potential]
Low potential for regression. This increases the number of devices allowed for SCSI; since following spec there may be up to 16 devices on a bus. This might cause issues still for out of spec systems with more devices on the bus, or buggy BIOS incorrectly cleaning up memory for devices that are not present (ie. non-zeroed memory that could make ipmi think there is a device when there is not).

Changed in ipmitool (Ubuntu):
assignee: nobody → Taco Screen team (taco-screen-team)
Steve Langasek (vorlon)
Changed in ipmitool (Ubuntu):
assignee: Taco Screen team (taco-screen-team) → Mathieu Trudel-Lapierre (mathieu-tl)
importance: Undecided → Medium
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ipmitool - 1.8.15-1ubuntu2

---------------
ipmitool (1.8.15-1ubuntu2) xenial; urgency=medium

  * debian/patches/0001-ID-394-plugins-usb-Fix-probe-for-SCSI-devices.patch:
    fix probing for SCSI devices in the USB plugin. (LP: #1509896)

 -- Mathieu Trudel-Lapierre <email address hidden> Wed, 28 Oct 2015 09:59:09 -0400

Changed in ipmitool (Ubuntu):
status: New → Fix Released
Revision history for this message
Jeremy Kerr (jk-ozlabs) wrote :

Is it possible to do the same for trusty too?

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

That was the plan, I just had not reached that yet :)

Revision history for this message
Jeremy Kerr (jk-ozlabs) wrote :

@mathieu-tl: ok, super. Sorry, I didn't mean to be badgering you there, just wanted to make sure it's tracked somewhere! :)

Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello Jeremy, or anyone else affected,

Accepted ipmitool into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ipmitool/1.8.13-1ubuntu0.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in ipmitool (Ubuntu Trusty):
status: New → Fix Committed
tags: added: verification-needed
Mathew Hodson (mhodson)
Changed in ipmitool (Ubuntu Trusty):
importance: Undecided → Medium
Changed in ipmitool (Ubuntu Vivid):
importance: Undecided → Medium
Changed in ipmitool (Ubuntu Wily):
importance: Undecided → Medium
Revision history for this message
Pavaman Subramaniyam (pavsubra) wrote :
Download full text (9.3 KiB)

I have installed the ipmitool_1.8.13-1ubuntu0.6_ppc64el.deb package on Ubuntu 14.04.3 OS and tried to update the OPAL 810 latest Firmware on a Habanero LC Hardware.

root@ltchabanero002:~# wget https://launchpad.net/ubuntu/+source/ipmitool/1.8.13-1ubuntu0.6/+build/8288800/+files/ipmitool_1.8.13-1ubuntu0.6_ppc64el.deb
--2015-11-16 03:10:11-- https://launchpad.net/ubuntu/+source/ipmitool/1.8.13-1ubuntu0.6/+build/8288800/+files/ipmitool_1.8.13-1ubuntu0.6_ppc64el.deb
Resolving launchpad.net (launchpad.net)... 91.189.89.222, 91.189.89.223
Connecting to launchpad.net (launchpad.net)|91.189.89.222|:443... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://launchpadlibrarian.net/225826018/ipmitool_1.8.13-1ubuntu0.6_ppc64el.deb [following]
--2015-11-16 03:10:12-- https://launchpadlibrarian.net/225826018/ipmitool_1.8.13-1ubuntu0.6_ppc64el.deb
Resolving launchpadlibrarian.net (launchpadlibrarian.net)... 91.189.89.228, 91.189.89.229
Connecting to launchpadlibrarian.net (launchpadlibrarian.net)|91.189.89.228|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 311704 (304K) [application/x-debian-package]
Saving to: âipmitool_1.8.13-1ubuntu0.6_ppc64el.debâ

100%[===================================================================================================================>] 311,704 601KB/s in 0.5s

2015-11-16 03:10:13 (601 KB/s) - âipmitool_1.8.13-1ubuntu0.6_ppc64el.debâ saved [311704/311704]

root@ltchabanero002:~# uname -a
Linux ltchabanero002 3.19.0-33-generic #38~14.04.1-Ubuntu SMP Fri Nov 6 18:17:15 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux
root@ltchabanero002:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="14.04.3 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.3 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"

root@ltchabanero002:~# cat /proc/cpuinfo | tail
processor : 79
cpu : POWER8 (raw), altivec supported
clock : 3491.000000MHz
revision : 2.0 (pvr 004d 0200)

timebase : 512000000
platform : PowerNV
model : TN71-BP012
machine : PowerNV TN71-BP012
firmware : OPAL v3

root@ltchabanero002:~# dpkg -i ipmitool_1.8.13-1ubuntu0.6_ppc64el.deb
(Reading database ... 139136 files and directories currently installed.)
Preparing to unpack ipmitool_1.8.13-1ubuntu0.6_ppc64el.deb ...
Unpacking ipmitool (1.8.13-1ubuntu0.6) over (1.8.13-1ubuntu0.5) ...
Setting up ipmitool (1.8.13-1ubuntu0.6) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
ureadahead will be reprofiled on next reboot

root@ltchabanero002:~# which ipmitool
/usr/bin/ipmitool
root@ltchabanero002:~# dpkg -S /usr/bin/ipmitool
ipmitool: /usr/bin/ipmitool
root@ltchabanero002:~# dpkg --list | grep ipmitool
ii ipmitool 1.8.13-1ubuntu0.6 ppc64el utility for IPMI control with kernel driver or LAN interface

root@ltchabanero002:~# wget https://w3.austin.ibm.com/afs/austin/projects/gfw/images/openpower8/810.1543.201511...

Read more...

Revision history for this message
Pavaman Subramaniyam (pavsubra) wrote :
Download full text (9.6 KiB)

But when I follow the same steps on updating the OPAL 810 Firmware on Firestone Hardware, I am getting error as shown.

root@ltc-fire8:~# uname -a
Linux ltc-fire8 3.19.0-32-generic #37~14.04.1-Ubuntu SMP Thu Oct 22 10:11:54 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux
root@ltc-fire8:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="14.04.3 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.3 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"

root@ltc-fire8:~# cat /proc/cpuinfo | tail
processor : 191
cpu : POWER8 (raw), altivec supported
clock : 2094.000000MHz
revision : 2.0 (pvr 004d 0200)

timebase : 512000000
platform : PowerNV
model : 8335-GTA
machine : PowerNV 8335-GTA
firmware : OPAL v3

root@ltc-fire8:~# wget https://launchpad.net/ubuntu/+source/ipmitool/1.8.13-1ubuntu0.6/+build/8288800/+files/ipmitool_1.8.13-1ubuntu0.6_ppc64el.deb
--2015-11-16 02:16:02-- https://launchpad.net/ubuntu/+source/ipmitool/1.8.13-1ubuntu0.6/+build/8288800/+files/ipmitool_1.8.13-1ubuntu0.6_ppc64el.deb
Resolving launchpad.net (launchpad.net)... 91.189.89.222, 91.189.89.223
Connecting to launchpad.net (launchpad.net)|91.189.89.222|:443... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://launchpadlibrarian.net/225826018/ipmitool_1.8.13-1ubuntu0.6_ppc64el.deb [following]
--2015-11-16 02:16:03-- https://launchpadlibrarian.net/225826018/ipmitool_1.8.13-1ubuntu0.6_ppc64el.deb
Resolving launchpadlibrarian.net (launchpadlibrarian.net)... 91.189.89.228, 91.189.89.229
Connecting to launchpadlibrarian.net (launchpadlibrarian.net)|91.189.89.228|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 311704 (304K) [application/x-debian-package]
Saving to: âipmitool_1.8.13-1ubuntu0.6_ppc64el.debâ

100%[===================================================================================================================>] 311,704 316KB/s in 1.0s

2015-11-16 02:16:05 (316 KB/s) - âipmitool_1.8.13-1ubuntu0.6_ppc64el.debâ saved [311704/311704]

root@ltc-fire8:~# dpkg -i ipmitool_1.8.13-1ubuntu0.6_ppc64el.deb
(Reading database ... 123258 files and directories currently installed.)
Preparing to unpack ipmitool_1.8.13-1ubuntu0.6_ppc64el.deb ...
Unpacking ipmitool (1.8.13-1ubuntu0.6) over (1.8.13-1ubuntu0.5) ...
Setting up ipmitool (1.8.13-1ubuntu0.6) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
ureadahead will be reprofiled on next reboot

root@ltc-fire8:~# which ipmitool
/usr/bin/ipmitool
root@ltc-fire8:~# dpkg -S /usr/bin/ipmitool
ipmitool: /usr/bin/ipmitool
root@ltc-fire8:~# dpkg --list | grep ipmitool
ii ipmitool 1.8.13-1ubuntu0.6 ppc64el utility for IPMI control with kernel driver or LAN interface

root@ltc-fire8:~# ipmitool -I usb lan print
Set in Progress : Set Complete
Auth Type Support : MD5
Auth Type Enable : Callback : MD5
                        : User : MD5
              ...

Read more...

Revision history for this message
Jeremy Kerr (jk-ozlabs) wrote :

Pavaman - thanks for verifying. The latter issue sounds like a BMC problem, which we're following-up separately.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ipmitool - 1.8.13-1ubuntu0.6

---------------
ipmitool (1.8.13-1ubuntu0.6) trusty; urgency=medium

  * debian/patches/lp1509896_fix_scsi_probe_607cfe6.patch: allow for up to
    16 devices on the SCSI bus when probing. Fixes the probing failing the
    use of the USB transport for IPMI when there are more than 8 SCSI devices
    on the bus, and follows the max devices for SCSI-3. (LP: #1509896)

 -- Mathieu Trudel-Lapierre <email address hidden> Wed, 04 Nov 2015 17:36:22 -0600

Changed in ipmitool (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Chris J Arges (arges) wrote : Update Released

The verification of the Stable Release Update for ipmitool has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

description: updated
Changed in ipmitool (Ubuntu Vivid):
status: New → In Progress
Changed in ipmitool (Ubuntu Wily):
status: New → In Progress
Changed in ipmitool (Ubuntu Vivid):
assignee: nobody → Mathieu Trudel-Lapierre (mathieu-tl)
Changed in ipmitool (Ubuntu Wily):
assignee: nobody → Mathieu Trudel-Lapierre (mathieu-tl)
Changed in ipmitool (Ubuntu Trusty):
assignee: nobody → Mathieu Trudel-Lapierre (mathieu-tl)
tags: removed: verification-done
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Jeremy, or anyone else affected,

Accepted ipmitool into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ipmitool/1.8.15-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in ipmitool (Ubuntu Wily):
status: In Progress → Fix Committed
tags: added: verification-needed
Changed in ipmitool (Ubuntu Vivid):
status: In Progress → Fix Committed
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Jeremy, or anyone else affected,

Accepted ipmitool into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ipmitool/1.8.15-1ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Frédéric Bonnard (frediz) wrote :

Verification successfully done on Trusty with : ipmitool 1.8.13-1ubuntu0.6

According to the bug fix in git :
----
    The current USB plugin doesn't find my IPMI channel:

      # ipmitool -I usb mc info
      Error in USB session setup

      Unable to setup interface usb
      Error loading interface usb

    This is beacuse I have more than 8 scsi-generic devices that identify
    as AMI:

      # grep -c ^AMI /proc/scsi/sg/device_strs
      13
----
On my machine :
root@abana:~# dpkg -l|grep ipmitool
ii ipmitool 1.8.13-1ubuntu0.6 ppc64el utility for IPMI control with kernel driver or LAN interface
root@abana:~# grep -c ^AMI /proc/scsi/sg/device_strs
13
root@abana:~# ipmitool -I usb mc info
Device ID : 32
Device Revision : 1
Firmware Revision : 2.06
IPMI Version : 2.0
Manufacturer ID : 0
Manufacturer Name : Unknown
Product ID : 43707 (0xaabb)
Product Name : Unknown (0xAABB)
Device Available : yes
Provides Device SDRs : no
Additional Device Support :
    Sensor Device
    SDR Repository Device
    SEL Device
    FRU Inventory Device
    IPMB Event Receiver
    IPMB Event Generator
    Chassis Device
Aux Firmware Rev Info :
    0xca
    0x48
    0x01
    0x00
root@abana:~# ipmitool -I usb lan print
Set in Progress : Set Complete
Auth Type Support : MD5
Auth Type Enable : Callback : MD5
                        : User : MD5
                        : Operator : MD5
                        : Admin : MD5
                        : OEM : MD5
IP Address Source : DHCP Address
IP Address : 9.101.18.66
Subnet Mask : 255.255.255.0
MAC Address : a0:42:3f:27:34:b8
SNMP Community String : AMI
IP Header : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl : 0.0 seconds
Default Gateway IP : 9.101.18.254
Default Gateway MAC : 00:12:79:04:36:80
Backup Gateway IP : 0.0.0.0
Backup Gateway MAC : 00:00:00:00:00:00
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 0,1,2,3,6,7,8,11,12,15,16,17
Cipher Suite Priv Max : caaaaaaaaaaaXXX
                        : X=Cipher Suite Unused
                        : c=CALLBACK
                        : u=USER
                        : o=OPERATOR
                        : a=ADMIN
                        : O=OEM
Bad Password Threshold : 0
Invalid password disable: no
Attempt Count Reset Int.: 0
User Lockout Interval : 0

tags: added: verification-done
removed: verification-needed
Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

Hi Frédéric;

Trusty has already been verified and landed. Did you mean vivid?

tags: added: verification-needed
removed: verification-done
Revision history for this message
Gianfranco Costamagna (costamagnagianfranco) wrote :

Hi @Jeremy, your patch was partially upstreamed, but since the important bits were, I deleted it.
I submitted a patch for the useless fp!=NULL check there
https://sourceforge.net/p/ipmitool/bugs/432/

(one is correctly removed, this one isn't important because outside of the loop)
cheers,

G.

Revision history for this message
Martin Pitt (pitti) wrote : Proposed package removed from archive

The version of ipmitool in the proposed pocket of Vivid that was purported to fix this bug report has been removed because the bugs that were to be fixed by the upload were not verified in a timely (105 days) fashion.

Changed in ipmitool (Ubuntu Vivid):
status: Fix Committed → Won't Fix
Revision history for this message
Martin Pitt (pitti) wrote :

The version of ipmitool in the proposed pocket of Wily that was purported to fix this bug report has been removed because the bugs that were to be fixed by the upload were not verified in a timely (105 days) fashion.

Changed in ipmitool (Ubuntu Wily):
status: Fix Committed → Won't Fix
tags: removed: verification-needed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.