incorrect sensor is read on Lenovo server BMC

Bug #1926299 reported by Dan Streetman
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
freeipmi (Ubuntu)
Fix Released
Low
Heather Lemon
Bionic
Fix Committed
Low
Heather Lemon
Focal
Fix Committed
Low
Heather Lemon
Groovy
Won't Fix
Low
Heather Lemon
Hirsute
Won't Fix
Low
Heather Lemon
Impish
Fix Released
Low
Heather Lemon

Bug Description

[impact]

ipmi-sensors reads the wrong sensor on a Lenovo BMC (although the problem may happen on other BMCs as well)

[test case]

run ipmi-sensors over the network using ipmi 1.5 or 2 protocol, and check the sensor output. Specifically for this case, the server is a Lenovo SR665 with only 1 cpu socket populated, and dimms 1-16 are connected to cpu 1 while dimms 17-32 are connected to cpu2, so dimms 17-32 should not return any values.

$ ipmi-sensors -u $USERNAME -p $PASSWD -D LAN_2_0 -l USER -h $BMCADDR | grep DIMM | grep Temp
110 | DIMM 1 Temp | Temperature | N/A | C | N/A
113 | DIMM 2 Temp | Temperature | N/A | C | N/A
116 | DIMM 3 Temp | Temperature | N/A | C | N/A
119 | DIMM 4 Temp | Temperature | N/A | C | N/A
122 | DIMM 5 Temp | Temperature | N/A | C | N/A
125 | DIMM 6 Temp | Temperature | N/A | C | N/A
128 | DIMM 7 Temp | Temperature | N/A | C | N/A
131 | DIMM 8 Temp | Temperature | N/A | C | N/A
134 | DIMM 9 Temp | Temperature | 52.00 | C | 'OK'
137 | DIMM 10 Temp | Temperature | 42.00 | C | 'OK'
140 | DIMM 11 Temp | Temperature | N/A | C | N/A
143 | DIMM 12 Temp | Temperature | N/A | C | N/A
146 | DIMM 13 Temp | Temperature | 27.00 | C | 'OK'
149 | DIMM 14 Temp | Temperature | 26.00 | C | 'OK'
152 | DIMM 15 Temp | Temperature | 24.00 | C | 'OK'
155 | DIMM 16 Temp | Temperature | 24.00 | C | 'OK'
158 | DIMM 17 Temp | Temperature | 218.00 | C | 'OK'
161 | DIMM 18 Temp | Temperature | 212.00 | C | 'OK'
165 | DIMM 19 Temp | Temperature | 212.00 | C | 'OK'
173 | DIMM 20 Temp | Temperature | N/A | C | N/A
176 | DIMM 21 Temp | Temperature | N/A | C | N/A
179 | DIMM 22 Temp | Temperature | N/A | C | N/A
182 | DIMM 23 Temp | Temperature | N/A | C | N/A
185 | DIMM 24 Temp | Temperature | N/A | C | N/A
188 | DIMM 25 Temp | Temperature | 0.00 | C | 'At or Below (<=) Lower Non-Critical Threshold'
191 | DIMM 26 Temp | Temperature | 0.00 | C | 'At or Below (<=) Lower Non-Critical Threshold'
194 | DIMM 27 Temp | Temperature | 0.00 | C | 'At or Below (<=) Lower Non-Critical Threshold'
197 | DIMM 28 Temp | Temperature | 0.00 | C | 'At or Below (<=) Lower Non-Critical Threshold'
200 | DIMM 29 Temp | Temperature | 0.00 | C | 'At or Below (<=) Lower Non-Critical Threshold'
203 | DIMM 30 Temp | Temperature | 0.00 | C | 'At or Below (<=) Lower Non-Critical Threshold'
206 | DIMM 31 Temp | Temperature | 0.00 | C | 'At or Below (<=) Lower Non-Critical Threshold'
209 | DIMM 32 Temp | Temperature | N/A | C | N/A

[regression potential]

any regression would likely result in incorrect values shown for some/all sensor readings

[scope]

this was recently fixed upstream, so it needed in all releases
https://github.com/chu11/freeipmi-mirror/pull/43

a new upstream release was made including the fix, so debian will pick it up in their next devel cycle

[other info]

the problem is that this particular BMC has chosen to use a non-zero 'LUN' number for these sensors; almost all BMCs use the standard LUN number 0, as the spec does state in section 19.3:

"Unless otherwise specified, commands that are listed as mandatory must be accessed via LUN 00b"

However the spec does allow for implementations to use alternate LUN numbers, specifically section 5.4 describes how the BMC would report the LUN number to the requesting software, and section 7.2 clarifies that while LUN 00b is reserved for devices beloning to the BMC itself and 10b is reserved for SMS messages, LUN numbers 01b and 11b are reserved for 'OEM' use. So it does appear valid for the BMC implementation to place sensor(s) under LUN 01b instead of the default 00b.

The cause of the invalid sensor readings in this particular case is because the BMC also defines sensors with the exact same sensor id number, but with LUN 00b. So when freeipmi attempts to read e.g. 'DIMM 17 Temp' sensor, it actually reads the current value for a completely different sensor, which (in this case) isn't even a temperature sensor, resulting in invalid temperature reading shown for the sensor, instead of showing it as N/A (since the DIMM slot isn't populated).

Note that ipmitool correctly uses the alternate LUN number so this bug does not exist with that tool, this affects only freeipmi tooling.

Changed in freeipmi (Ubuntu):
assignee: nobody → Heather Lemon (hypothetical-lemon)
Revision history for this message
Heather Lemon (hypothetical-lemon) wrote :
Changed in freeipmi (Ubuntu Hirsute):
assignee: nobody → Heather Lemon (hypothetical-lemon)
Changed in freeipmi (Ubuntu Groovy):
assignee: nobody → Heather Lemon (hypothetical-lemon)
Changed in freeipmi (Ubuntu Focal):
assignee: nobody → Heather Lemon (hypothetical-lemon)
Changed in freeipmi (Ubuntu Bionic):
assignee: nobody → Heather Lemon (hypothetical-lemon)
Revision history for this message
Heather Lemon (hypothetical-lemon) wrote :
Revision history for this message
Heather Lemon (hypothetical-lemon) wrote :

fix accepted upstream.. starting backport process

Revision history for this message
Heather Lemon (hypothetical-lemon) wrote :

attach hirsute debdiff

Changed in freeipmi (Ubuntu Bionic):
status: New → In Progress
Changed in freeipmi (Ubuntu Focal):
status: New → In Progress
Changed in freeipmi (Ubuntu Groovy):
status: New → In Progress
Changed in freeipmi (Ubuntu Hirsute):
status: New → In Progress
importance: Undecided → Low
Changed in freeipmi (Ubuntu Groovy):
importance: Undecided → Low
Changed in freeipmi (Ubuntu Focal):
importance: Undecided → Low
Changed in freeipmi (Ubuntu Bionic):
importance: Undecided → Low
Revision history for this message
Heather Lemon (hypothetical-lemon) wrote :

upload groovy debdiff

Revision history for this message
Heather Lemon (hypothetical-lemon) wrote :

hirsute re-uploaded again after missing d/p/

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "groovy-1926299.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Revision history for this message
Heather Lemon (hypothetical-lemon) wrote :

fix groovy wrong date uploaded.

Revision history for this message
Heather Lemon (hypothetical-lemon) wrote :

i misspelled groovy.. -_-

Revision history for this message
Heather Lemon (hypothetical-lemon) wrote :

focal debdiff

Dan Streetman (ddstreet)
Changed in freeipmi (Ubuntu Impish):
status: New → In Progress
importance: Undecided → Low
Revision history for this message
Heather Lemon (hypothetical-lemon) wrote :

bionic debdiff

tags: added: sts-sponsor verification-needed-bionic verification-needed-focal verification-needed-groovy verification-needed-hirsute
Revision history for this message
Dan Streetman (ddstreet) wrote :

uploaded to b/f/g/h/i thanks!

description: updated
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package freeipmi - 1.6.6-3ubuntu1

---------------
freeipmi (1.6.6-3ubuntu1) impish; urgency=medium

  * d/p/libfreeipmi-ipmi-sensor-read.c-Incorrect-sensor-read.patch
    incorrect sensor reading reported (LP: #1926299)
  * update Changelog

 -- Heather Lemon <email address hidden> Thu, 20 May 2021 18:13:30 +0000

Changed in freeipmi (Ubuntu Impish):
status: In Progress → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Dan, or anyone else affected,

Accepted freeipmi into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/freeipmi/1.6.6-3ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in freeipmi (Ubuntu Hirsute):
status: In Progress → Fix Committed
tags: added: verification-needed
Changed in freeipmi (Ubuntu Groovy):
status: In Progress → Fix Committed
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Dan, or anyone else affected,

Accepted freeipmi into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/freeipmi/1.6.4-3ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Dan, or anyone else affected,

Accepted freeipmi into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/freeipmi/1.6.4-3ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in freeipmi (Ubuntu Focal):
status: In Progress → Fix Committed
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Dan, or anyone else affected,

Accepted freeipmi into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/freeipmi/1.4.11-1.1ubuntu4.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in freeipmi (Ubuntu Bionic):
status: In Progress → Fix Committed
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (freeipmi/1.4.11-1.1ubuntu4.2)

All autopkgtests for the newly accepted freeipmi (1.4.11-1.1ubuntu4.2) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

nut/2.7.4-5.1ubuntu2 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#freeipmi

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Dan Streetman (ddstreet)
Changed in freeipmi (Ubuntu Groovy):
status: Fix Committed → Won't Fix
Mathew Hodson (mhodson)
tags: removed: verification-needed verification-needed-groovy
Revision history for this message
Brian Murray (brian-murray) wrote :

Could this please get verified?

Revision history for this message
Dan Streetman (ddstreet) wrote :

> Could this please get verified?

unfortunately this request came from a customer who then stopped responding to our requests for verification after this reached -proposed. Since verification for this requires access to specific server bmc, we were stuck being unable to verify this ourselves.

We could look around the company to try to find a machine with this bmc to verify this, or we could just drop this from -proposed entirely (or, just release it without direct verification).

Presumably, our customer stopped responding because the test package with this fix worked for them (they did verify that before we uploaded the fixed package as SRU), so the fix *should* work right, and it has been accepted upstream.

Revision history for this message
Eric Desrochers (slashd) wrote :

[sts-sponsor]

After a discussion with ddstreet about nobody being able (as it requires specific HW) nor reply to our request to test the proposed packages for this particular bug.

Since the packages of freeipmi are languishing in proposed for about 176 days (as of today) with no responses/outcomes/feebacks/....

I have asked the SRU verification team to drop the packages found in -proposed.

Note that as ddstreet mentioned, upstream has adopted the fix, so if someone in the Ubuntu community/customer report this bug again in the future and commit on testing the fix in Ubuntu we could consider to backport again.

- Eric

tags: added: verification-failed-bionic verification-failed-focal verification-failed-hirsute
removed: verification-needed-bionic verification-needed-focal verification-needed-hirsute
Revision history for this message
Brian Murray (brian-murray) wrote :

The Hirsute Hippo has reached End of Life, so this bug will not be fixed for that release.

Changed in freeipmi (Ubuntu Hirsute):
status: Fix Committed → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.