openipmi startup script removes kernel modules

Bug #1318317 reported by Oliver Mueller
50
This bug affects 7 people
Affects Status Importance Assigned to Milestone
openipmi (Ubuntu)
Fix Released
High
Nish Aravamudan
Trusty
Fix Released
High
Unassigned

Bug Description

I need some IPMI kernel modules, so I load them during startup. This worked quite well until I upgraded to Ubuntu 14.04.

As you can see I have the modules in /etc/modules:

# grep ipmi /etc/modules
ipmi_watchdog
ipmi_devintf
ipmi_poweroff
ipmi_si

Right after reboot, the modules are there:

# lsmod |grep -i ipm
ipmi_si 53257 2
ipmi_poweroff 14366 0
ipmi_devintf 17572 0
ipmi_watchdog 24912 0

but disappear after a few seconds/minute. I found out, that the cause is the /etc/init.d/openipmi script. The behaviour of the script is quite strange because a stop/start will behave differently then a restart.

I loaded the modules manually using modprobe:
ipmi_si 53257 2
ipmi_poweroff 14366 0
ipmi_devintf 17572 0
ipmi_watchdog 24912 0

when I execute:
# service openipmi stop

the module ipmi_devintf gets removed:

ipmi_si 53257 2
ipmi_poweroff 14366 0
ipmi_watchdog 24912 0

starting the service again with:

# service openipmi start

will remove ALL IPMI kernel modules:

# lsmod |grep -i ipm
#

One of the reasons seems to be the module ipmi_msghandler. This is already integrated and causes some of the problems.

When I remove it from /etc/init.d/openipmi like this:

65,66c65
< #MODULES_BASE="ipmi_msghandler"
< MODULES_BASE=""
---
> MODULES_BASE="ipmi_msghandler"
325,326c324,325
< # modprobe ipmi_msghandler > /dev/null 2>&1
< # modules_loaded ipmi_msghandler
---
> modprobe ipmi_msghandler > /dev/null 2>&1
> modules_loaded ipmi_msghandler

stoping and starting the service works again. But restarting the service will still remove all modules and will not load them anymore.

[Impact]
Issuing `service openipmi {start,stop,restart}` failures lead to unloading of IPMI modules that were previously loaded. Particularly on OpenPower systems, but also on others, this will result in 'ipmi_devintf' being unloaded during boot, and in-band IPMI being non-functional (amongst other IPMI functionality).

[Test Case]
On an OpenPower machine (this also happens on an appropriately configured (meaning without IPMI support) Intel qemu instance), load IPMI modules (in particular, ipmi_devintf, but any of them can be affected). Attempt to start the openipmi service with `service openipmi start`. Without the fix proposed, here, the ipmi_devintf module is unloaded. With the fix, the module stays loaded.

[Regression Potential]
The primary risk of regression is if an end-user was relying on the behavior of an error during openipmi initialization resulting in disabling of all IPMI functionality. I don't think that is reasonable behavior to rely on, though (and is not clearly documented anywhere as expected).

Revision history for this message
Tore Anderson (toreanderson) wrote :

This affects me, too. After boot, the necessary ipmi_{si,devintf} modules aren't loaded, so ipmitool and related monitoring doesn't work.

However this bug cannot possibly be in the "ejabberd" package, so I'm reassigning it to "openipmi".

affects: ejabberd (Ubuntu) → openipmi (Ubuntu)
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in openipmi (Ubuntu):
status: New → Confirmed
Revision history for this message
Jeremy Kerr (jk-ozlabs) wrote :

We see the same on OpenPower machines. Since we don't use the ipmi_si interface, the openipmi init script fails, which removes modules which we do need.

root@g43L:~# grep -B2 IPMI_SI /etc/default/openipmi
# Enable standard hardware interfaces (KCS, BT, SMIC)
# You probably want this enabled.
IPMI_SI=yes
root@g43L:~# modprobe ipmi_devintf
root@g43L:~# lsmod | grep ipmi
ipmi_devintf 13355 0
ipmi_powernv 5657 0
ipmi_msghandler 51374 2 ipmi_powernv,ipmi_devintf
root@g43L:~# service openipmi start
 * Starting ipmi drivers
   ...fail!
   ...done.
root@g43L:~# lsmod | grep ipmi
ipmi_powernv 5657 0
ipmi_msghandler 51374 1 ipmi_powernv
root@g43L:~#

This breaks any system daemon that requires in-band IPMI.

Workaround is to set IPMI_SI="no" in /etc/default/openipmi.

tags: added: taco-screen-team
Changed in openipmi (Ubuntu):
assignee: nobody → Taco Screen team (taco-screen-team)
Revision history for this message
Jeremy Kerr (jk-ozlabs) wrote :

From comment #3:

> This breaks any system daemon that requires in-band IPMI.

To provide a little more detail here: the opal-prd daemon, necessary for runtime system management on OpenPOWER machines, may communicate with the BMC using IPMI. As part of that daemon's initialisation, it ensures that the IPMI device interface is loaded (ie, the ipmi_devintf module), to ensure that the /dev/ipmiN interface is working.

However, after that daemon is started, the openipmi init script then *unloads* the ipmi_devintf module.

The same would happen if the user has added ipmi_devintf to /etc/modules, to get it loaded on boot.

At best, this unload means that the user will always need to issue a 'modprobe ipmi_devintf', on every reboot, before running any 'ipmitool' command using the in-band BT interface.

Since this also disabled communication between the opal-prd daemon and the BMC, it means that errors detected by opal-prd may not be reported to the BMC.

Jon Grimm (jgrimm)
Changed in openipmi (Ubuntu):
importance: Undecided → High
Revision history for this message
Nish Aravamudan (nacc) wrote :

Oliver, Tore & Jeremy, I'm taking a look at this issue now, and just wanted to confirm that you've not modified /etc/default/openipmi in your setups (except as the noted workaround by Jeremy)?

Nish Aravamudan (nacc)
Changed in openipmi (Ubuntu):
assignee: Taco Screen team (taco-screen-team) → Nish Aravamudan (nacc)
Revision history for this message
Nish Aravamudan (nacc) wrote :

Jeremy,

Does this patch have any effect on your OpenPower system?

Thanks,
Nish

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "lp1318317.v1.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch
Revision history for this message
Nish Aravamudan (nacc) wrote :

I've setup a PPA for x86 at: https://launchpad.net/~nacc/+archive/ubuntu/lp1318317, with some packages for testing on trusty & xenial. Still figuring out how to setup the PPA for ppc6el.

Revision history for this message
Nish Aravamudan (nacc) wrote :

Sorry, had some issues with that PPA, there is a new one at: https://launchpad.net/~nacc/+archive/ubuntu/lp1318317-1

Revision history for this message
Nish Aravamudan (nacc) wrote :

Nevermind, my apologies -- I was just impatient. Please use the original PPA (described at https://launchpad.net/~nacc/+archive/ubuntu/lp1318317)

Revision history for this message
Jeremy Kerr (jk-ozlabs) wrote :

thanks Nish! I just installed that package:

[jk@fstn2-p1 sources.list.d]$ sudo apt-get remove --purge openipmi
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  openipmi*
0 to upgrade, 0 to newly install, 1 to remove and 0 not to upgrade.
After this operation, 441 kB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 167006 files and directories currently installed.)
Removing openipmi (2.0.18-0ubuntu8) ...
Purging configuration files for openipmi (2.0.18-0ubuntu8) ...
Processing triggers for man-db (2.7.0.2-5) ...
[jk@fstn2-p1 sources.list.d]$ sudo apt-get install openipmi=2.0.18-0ubuntu7.2
Reading package lists... Done
Building dependency tree
Reading state information... Done
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
  openipmi
0 to upgrade, 1 to newly install, 0 to remove and 0 not to upgrade.
Need to get 189 kB of archives.
After this operation, 727 kB of additional disk space will be used.
Get:1 http://ppa.launchpad.net/nacc/lp1318317/ubuntu/ trusty/main openipmi ppc64el 2.0.18-0ubuntu7.2 [189 kB]
Fetched 189 kB in 2s (79.3 kB/s)
Selecting previously unselected package openipmi.
(Reading database ... 166978 files and directories currently installed.)
Preparing to unpack .../openipmi_2.0.18-0ubuntu7.2_ppc64el.deb ...
Unpacking openipmi (2.0.18-0ubuntu7.2) ...
Processing triggers for man-db (2.7.0.2-5) ...
Processing triggers for systemd (219-7ubuntu6) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up openipmi (2.0.18-0ubuntu7.2) ...
Processing triggers for systemd (219-7ubuntu6) ...
Processing triggers for ureadahead (0.100.0-19) ...

Now I get:

[jk@fstn2-p1 sources.list.d]$ sudo service openipmi start
Job for openipmi.service failed. See "systemctl status openipmi.service" and "journalctl -xe" for details.
[jk@fstn2-p1 sources.list.d]$ systemctl status openipmi.service
● openipmi.service - LSB: OpenIPMI Driver init script
   Loaded: loaded (/etc/init.d/openipmi)
   Active: failed (Result: exit-code) since Tue 2016-01-05 23:49:12 UTC; 5s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 4600 ExecStart=/etc/init.d/openipmi start (code=exited, status=1/FAILURE)

Jan 05 23:49:12 fstn2-p1 systemd[1]: Starting LSB: OpenIPMI Driver init script...
Jan 05 23:49:12 fstn2-p1 openipmi[4600]: * Starting ipmi drivers
Jan 05 23:49:12 fstn2-p1 openipmi[4600]: ...fail!
Jan 05 23:49:12 fstn2-p1 openipmi[4600]: ...done.
Jan 05 23:49:12 fstn2-p1 systemd[1]: openipmi.service: control process exited, code=exited status=1
Jan 05 23:49:12 fstn2-p1 systemd[1]: Failed to start LSB: OpenIPMI Driver init script.
Jan 05 23:49:12 fstn2-p1 systemd[1]: Unit openipmi.service entered failed state.
Jan 05 23:49:12 fstn2-p1 systemd[1]: openipmi.service failed.

But at least my modules are still loaded:

[jk@fstn2-p1 sources.list.d]$ lsmod | grep ipmi
ipmi_devintf 13227 0
ipmi_powernv 5657 0
ipmi_msghandler 51494 2 ipmi_powernv,ipmi_devintf
[jk@fstn2-p1 sources.list.d]$

.. which is a net improvement :)

Revision history for this message
Nish Aravamudan (nacc) wrote :

Right, that is because "you" (slash the default) have specified in the openmpi configuration file (/etc/default/openmpi) that IPMI_SI is needed on your system. Thus, when it fails to load successfully, it is still flagged as:

# no bits set = no errors
# bit 0 set = minimum modules aren't loaded
# bit 1 set = requested feature module isn't loaded
# bit 2 set = /dev/ipmi0 (or /dev/imb if using that instead) doesn't exist
# bit 3 set = /dev/watchdog doesn't exist
# bit 4 set = lockfile doesn't exist
# bit 5 set = modules are loaded even when asked to be unloaded

bit 0 being set (RETVAL | 1).

I guess we could change the logic in a different way such that bit 1 should be set in this case, rather than bit 0. But that would redefine what the "minimum" modules are in OpenIPMI, which per the init-script are ipmi_msghandler and ipmi_{si,smb} if those options are set in the config file.

So I think what I provided is probably the best combination for the configuration by default -- Robie did point out that my patch has a logical issue in that the last added line is really a no-op. So I'll spin up a new version and push it to my PPA. If you can test that one as well, just to be sure it's functionally the same, then we'll work on getting that into -proposed (once I learn how to do that :)

I'll provide another update once the new packages are present.

Thanks, Jeremy!

Revision history for this message
Nish Aravamudan (nacc) wrote :

I've uploaded an alternative change to my PPA, builds are going now, which only does not unload all modules in the error path.

The difference is, to minimize the impact to people that might be depending on the error path *not* loading more modules, I've put the return lines back in. So I'd like a re-test, if possible, Jeremy, to see if this still works for your case.

For Trusty use 2.0.18-0ubunt7.3 and for Xenial use 2.0.18-0ubuntu10 from my PPA. I'll fix the versions properly once we've agreed on the fix.

-Nish

Revision history for this message
Nish Aravamudan (nacc) wrote :

Jeremy, have you had a chance to test the updated version(s)?

Thanks,
Nish

Revision history for this message
Jeremy Kerr (jk-ozlabs) wrote :

Hi Nish,

I'll give it a try, thanks!

Revision history for this message
Jeremy Kerr (jk-ozlabs) wrote :

Nish: looks good to me:

[jk@hab2 ~]$ sudo service openipmi stop
[jk@hab2 ~]$ sudo modprobe ipmi_devintf
[jk@hab2 ~]$ lsmod | grep ipmi
ipmi_devintf 13227 0
ipmi_powernv 5657 0
ipmi_msghandler 51494 2 ipmi_powernv,ipmi_devintf
[jk@hab2 ~]$ sudo service openipmi start
Job for openipmi.service failed. See "systemctl status openipmi.service" and "journalctl -xe" for details.
[jk@hab2 ~]$ lsmod | grep ipmi
ipmi_devintf 13227 0
ipmi_powernv 5657 0
ipmi_msghandler 51494 2 ipmi_powernv,ipmi_devintf
[jk@hab2 ~]$

- even though the openipmi service itelf fails, it doesn't take down everyone else's access to /dev/ipmiX

Revision history for this message
Breno Leitão (breno-leitao) wrote :

Nish,

I understand that this bug should be targeted for 16.04 as for 14.04. Is it your understanding also?

Revision history for this message
Nish Aravamudan (nacc) wrote :

@Breno, yes, I think that's correct.

@Jeremy, were you able to test in both 16.04 and 14.04?

Nish Aravamudan (nacc)
Changed in openipmi (Ubuntu):
milestone: none → ubuntu-16.04
Revision history for this message
Nish Aravamudan (nacc) wrote :

openipmi (2.0.18-0ubuntu10) xenial; urgency=medium

  * debian/openipmi.init: Do not unload all modules if some configured
    HW modules are not successfully loaded (LP: #1318317).

 -- Nishanth Aravamudan <email address hidden> Thu, 03 Mar 2016 16:55:22 -0800

Revision history for this message
Nish Aravamudan (nacc) wrote :

@Breno, once the fix has been accepted for 16.04, I will request a SRU for 14.04.

Revision history for this message
Jeremy Kerr (jk-ozlabs) wrote :

@Nish: I tested against 14.04. We can take a look at 16.04 too, if you like.

Revision history for this message
Nish Aravamudan (nacc) wrote : Re: [Bug 1318317] Re: openipmi startup script removes kernel modules

On Mar 9, 2016 19:15, "Jeremy Kerr" <email address hidden> wrote:
>
> @Nish: I tested against 14.04. We can take a look at 16.04 too, if you
> like.

That'd be great, as we need to get it into 16.04 before we can get it into
14.04.

Revision history for this message
Jeremy Kerr (jk-ozlabs) wrote :

Looks good on 16.04 too:

$ sudo modprobe ipmi_devintf
$ sudo service openipmi start
Job for openipmi.service failed because the control process exited with error code. See "systemctl status openipmi.service" and "journalctl -xe" for details.
$ lsmod | grep ipmi
ipmi_devintf 13691 0
ipmi_powernv 6489 0
ipmi_msghandler 50947 2 ipmi_powernv,ipmi_devintf

Thanks!

Revision history for this message
Jeremy Kerr (jk-ozlabs) wrote :

(That's with your fixed package installed. Without, we see the ipmi_devintf module being removed)

Revision history for this message
Nish Aravamudan (nacc) wrote :

On 11.03.2016 [02:30:46 -0000], Jeremy Kerr wrote:
> (That's with your fixed package installed. Without, we see the
> ipmi_devintf module being removed)

Great, thanks! I'll ask for the fix to get merged and then submit an SRU
for 14.04.

Changed in openipmi (Ubuntu):
status: Confirmed → In Progress
Revision history for this message
Nish Aravamudan (nacc) wrote :

I have also sent the same patch upstream, with no response yet: https://sourceforge.net/p/openipmi/mailman/message/34928697/

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openipmi - 2.0.18-0ubuntu11

---------------
openipmi (2.0.18-0ubuntu11) xenial; urgency=medium

  * debian/openipmi.init: Do not unload all modules if some configured
    HW modules are not successfully loaded (LP: #1318317).

 -- Nishanth Aravamudan <email address hidden> Thu, 03 Mar 2016 16:55:22 -0800

Changed in openipmi (Ubuntu):
status: In Progress → Fix Released
Nish Aravamudan (nacc)
description: updated
Revision history for this message
Nish Aravamudan (nacc) wrote :
Mathew Hodson (mhodson)
Changed in openipmi (Ubuntu Trusty):
status: New → Triaged
importance: Undecided → High
Revision history for this message
Michael Terry (mterry) wrote :

Nish, I uploaded this to trusty-proposed. Let's wait for SRU process now. Thanks for the patch!

Changed in openipmi (Ubuntu Trusty):
status: Triaged → In Progress
Revision history for this message
Timo Aaltonen (tjaalton) wrote : Please test proposed package

Hello Oliver, or anyone else affected,

Accepted openipmi into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openipmi/2.0.18-0ubuntu7.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in openipmi (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Nish Aravamudan (nacc) wrote :

@Jeremy, could you test once more with the above version of openipmi in Trusty, to verify the SRU -- which will then allow it to propogate to -updates?

Revision history for this message
Breno Leitão (breno-leitao) wrote :

Jeremy is on vacation, but I just tested this version on -proposed, and it seems to be working fine:

ubuntu@ubuntu1404:~$ lsmod | grep ipmi
ipmi_watchdog 22705 0
ipmi_poweroff 11629 0
ipmi_devintf 13347 0
ipmi_msghandler 51294 3 ipmi_devintf,ipmi_poweroff,ipmi_watchdog
ubuntu@ubuntu1404:~$ service openipmi stop
 * Stopping ipmi drivers. * may be in use
                                                                                                                                          [ OK ]
ubuntu@ubuntu1404:~$ lsmod | grep ipmi
ipmi_watchdog 22705 0
ipmi_poweroff 11629 0
ipmi_devintf 13347 0
ipmi_msghandler 51294 3 ipmi_devintf,ipmi_poweroff,ipmi_watchdog
ubuntu@ubuntu1404:~$ sudo service openipmi stop
 * Stopping ipmi drivers. [ OK ]
ubuntu@ubuntu1404:~$ lsmod | grep ipmi
ipmi_watchdog 22705 0
ipmi_poweroff 11629 0
ipmi_msghandler 51294 2 ipmi_poweroff,ipmi_watchdog
ubuntu@ubuntu1404:~$ service openipmi start
 * Starting ipmi drivers [fail]
                                                                                                                                          [ OK ]
ubuntu@ubuntu1404:~$ lsmod | grep ipmi
ipmi_watchdog 22705 0
ipmi_poweroff 11629 0
ipmi_msghandler 51294 2 ipmi_poweroff,ipmi_watchdog

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openipmi - 2.0.18-0ubuntu7.2

---------------
openipmi (2.0.18-0ubuntu7.2) trusty; urgency=medium

  * debian/openipmi.init: Do not unload all modules if some configured
    HW modules are not successfully loaded.
    Backported from Xenial.
    -LP: #1318317

 -- Nishanth Aravamudan <email address hidden> Fri, 18 Mar 2016 14:45:39 -0700

Changed in openipmi (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
PabloAB (pabloab777) wrote :

Not sure if same bug but I'm on 18.04 with openipmi 2.0.22-1.1ubuntu2.1, 4.15.0-42-generic kernel on Asus UX303UB and openipmi.service fail to start: "Failed to start LSB: OpenIPMI Driver init script", as in https://askubuntu.com/q/898012/349837.

Revision history for this message
PabloAB (pabloab777) wrote :

FTR: lsmod | grep ipmi
ipmi_devintf 20480 0
ipmi_msghandler 53248 1 ipmi_devintf

And service won't start, as comment #11: "Failed to start LSB: OpenIPMI Driver init script".

Revision history for this message
Cesar Pulido (cesaar) wrote :

I'm on 18.10 amd64 with 2.0.25-2ubuntu1, 4.18.0-15-generic. systemd shows 'Failed to start LSB: OpenIPMI Driver init script.' on boot.

----------------------------------------

$ dpkg -l | grep openipmi
ii libopenipmi0 2.0.25-2ubuntu1 amd64 Intelligent Platform Management Interface - runtime
ii openipmi 2.0.25-2ubuntu1 amd64 Intelligent Platform Management Interface (for servers)

----------------------------------------

systemctl status openipmi.service
● openipmi.service - LSB: OpenIPMI Driver init script
   Loaded: loaded (/etc/init.d/openipmi; generated)
   Active: failed (Result: exit-code) since Thu 2019-03-07 01:32:34 -05; 33min ago
     Docs: man:systemd-sysv-generator(8)

Mar 07 01:32:28 development openipmi[1779]: /etc/init.d/openipmi: 55: [: 4.18: unexpected operator
Mar 07 01:32:28 development openipmi[1779]: * Starting ipmi drivers
Mar 07 01:32:26 development systemd[1]: Starting LSB: OpenIPMI Driver init script...
Mar 07 01:32:30 development openipmi[1779]: ...fail!
Mar 07 01:32:30 development openipmi[1779]: ...done.
Mar 07 01:32:34 development systemd[1]: openipmi.service: Control process exited, code=exited status=1
Mar 07 01:32:34 development systemd[1]: openipmi.service: Failed with result 'exit-code'.
Mar 07 01:32:34 development systemd[1]: Failed to start LSB: OpenIPMI Driver init script.

----------------------------------------

$ lsmod | grep ipmi
ipmi_devintf 20480 0
ipmi_msghandler 102400 1 ipmi_devintf

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.