Multiple issues found on Ubuntu 20.04 against drmgr

Default Comment by Bridge

------- Comment From <email address hidden> 2023-06-20 11:41 EDT-------
When can we expect this package would be available for ubuntu 20.04 customers?

Hello Ritu,
well, it's unfortunately not possible to just update the version in an Ubuntu release that is already out and in service.
We can only update the package to it's latest version (1.3.11) as part of the current development release (which is as of today 23.10, code-name 'mantic').

For all the other affected releases (in service) we need to either ask for an upstream commit that can be cherrypicked and fixes the issue, or in case this is not possible (means the cherrypick does not work cleanly), for a backport.
$ rmadison powerpc-utils
...
 powerpc-utils | 1.3.4-0ubuntu2 | focal | source, ppc64el
 powerpc-utils | 1.3.9-1ubuntu1 | jammy | source, ppc64el
 powerpc-utils | 1.3.9-1ubuntu2 | kinetic | source, ppc64el
 powerpc-utils | 1.3.10-2 | lunar | source, ppc64el
 powerpc-utils | 1.3.10-2 | mantic | source, ppc64el
And this needs to be done for all releases down to 20.04, to avoid any potential regressions with Ubuntu release upgrades.
(However, we may skip 22.10/kinetic, since it will reach it's end of live very shortly).

We have to stick to the so called 'stable release upgrade' (in short 'SRU') process:
https://wiki.ubuntu.com/StableReleaseUpdates
for such kind of fixes/updates.

So it's unfortunately needed to find the upstream commits (https://github.com/ibm-power-utilities/powerpc-utils/commits) that fixe all these glitches and we would like to ask the vendor for help on this.

------- Comment From <email address hidden> 2023-07-10 13:30 EDT-------
FYI, As of now we are looking this fix for ubuntu20.04.

@Ritu, so can you help with identifying the correct upstream commits that fix the issues mentioned?

------- Comment From <email address hidden> 2023-07-11 11:38 EDT-------
(In reply to comment #27)
> @Ritu, so can you help with identifying the correct upstream commits that
> fix the issues mentioned?

There are hundreds of commits between version 1.3.4 and 1.3.11.
Backporting the relevant fixes will yield an unstable, untested
Frankenstein's monster.

powerpc-utils 1.3.4 is *ancient*. It was tagged in 2017. It was
two releases out-of-date when Ubuntu 20.04 was released in 2020.

Keeping Ubuntu 20.04 chained to powerpc-utils 1.3.4 is inhibiting
support for new hardware. Every new version of powerpc-utils adds
support for the latest POWER hardware. Version 1.3.4 does not
have complete support for POWER 9. It has no support for POWER 10,
and it will have no support for the forthcoming POWER 11.

Can we offer a separate version of the package as an option alongside
1.3.4?

Somehow between bionic/18.04 and focal/20.04 there was not much activity and not a lot of updates on the powerpc-utils package, indeed:
$ rmadison powerpc-utils
 powerpc-utils | 1.2.18+r1.1.3-24ubuntu1 | trusty | source, powerpc, ppc64el
 powerpc-utils | 1.3.1-2 | xenial | source, powerpc, ppc64el
 powerpc-utils | 1.3.1-2ubuntu0.3 | xenial-updates | source, powerpc, ppc64el
 powerpc-utils | 1.3.4-0ubuntu2 | bionic | source, ppc64el
 powerpc-utils | 1.3.4-0ubuntu2 | focal | source, ppc64el
 powerpc-utils | 1.3.9-1ubuntu1 | jammy | source, ppc64el
 powerpc-utils | 1.3.9-1ubuntu2 | kinetic | source, ppc64el
 powerpc-utils | 1.3.10-2 | lunar | source, ppc64el
 powerpc-utils | 1.3.10-2 | mantic | source, ppc64el

We are happy to take requests to upgrade to the latest version, always as part of the current development release (always before Feature Freeze) - and we will consider updating powerpc-utils to latest v1.3.11 for mantic/23.10, the release that is currently in development.
I've now also subscribed myself to GH powerpc-utils for all new tags (releases) to get a notification.

Please notice that there is no P10 support in 20.04 (nor P11) and there will be never support for it - in 20.04, since it would mean that we have to lift the architectural level that we compile for.
That can only be done in the release that is currently in development.
And even trying to install will end up in an illegal instruction error - it's just not compiled for it (except one emulates an older architecture or runs in compat mode - if possible).

For P9 and P10 one actually has to migrate to the latest 22.04 LTS: https://ubuntu.com/download/server/power
20.04 is only still an option in case of P8.

I know that getting the commits to fix something in an older release can be painful, esp. if cherry-picking does not work cleanly and backports are needed - but it's due to the Ubuntu SRU process, that only allows important and critical fixes anyway, so not meant for version bumps, bigger patches or major new features.

We can of course (re-)build the latest version we have in the archive (currently v1.3.10) for focal in a PPA - and that is what I just did: https://launchpad.net/~fheimes/+archive/ubuntu/lp2023243
But this is also not ideal, since this version may for example have support for some P10 hardware features, but the underlying 20.04 has no support for this level - so the door is even more open for further issues one may run into.
So please handle the powerpc-utils package in ppa:fheimes/lp2023243 with care.

Added this successfully PPA build:
https://launchpad.net/~patriciasd/+archive/ubuntu/mantic.powerpc-utils.lp2023243

Added this successfully PPA build:
https://launchpad.net/~patriciasd/+archive/ubuntu/mantic.powerpc-utils.lp2023243

This bug was fixed in the package powerpc-utils - 1.3.11-0ubuntu1

---------------
powerpc-utils (1.3.11-0ubuntu1) mantic; urgency=medium

  * New upstream release (LP: #2023243)
  * d/p/0001-Change-program-path-for-udevadm-from-usr-bin-to-bin.patch id
    still required

 -- Patricia Domingues <email address hidden> Wed, 26 Jul 2023 16:44:24 -0300

------- Comment From <email address hidden> 2023-08-10 11:35 EDT-------
(In reply to comment #29)
>
> Somehow between bionic/18.04 and focal/20.04 there was not much activity and
> not a lot of updates on the powerpc-utils package, indeed:
> $ rmadison powerpc-utils
> powerpc-utils | 1.2.18+r1.1.3-24ubuntu1 | trusty | source, powerpc,
> ppc64el
> powerpc-utils | 1.3.1-2 | xenial | source, powerpc,
> ppc64el
> powerpc-utils | 1.3.1-2ubuntu0.3 | xenial-updates | source, powerpc,
> ppc64el
> powerpc-utils | 1.3.4-0ubuntu2 | bionic | source, ppc64el
> powerpc-utils | 1.3.4-0ubuntu2 | focal | source, ppc64el
> powerpc-utils | 1.3.9-1ubuntu1 | jammy | source, ppc64el
> powerpc-utils | 1.3.9-1ubuntu2 | kinetic | source, ppc64el
> powerpc-utils | 1.3.10-2 | lunar | source, ppc64el
> powerpc-utils | 1.3.10-2 | mantic | source, ppc64el
>
> We are happy to take requests to upgrade to the latest version, always as
> part of the current development release (always before Feature Freeze) - and
> we will consider updating powerpc-utils to latest v1.3.11 for mantic/23.10,
> the release that is currently in development.
> I've now also subscribed myself to GH powerpc-utils for all new tags
> (releases) to get a notification.
>
> Please notice that there is no P10 support in 20.04 (nor P11) and there will
> be never support for it - in 20.04, since it would mean that we have to lift
> the architectural level that we compile for.
> That can only be done in the release that is currently in development.
> And even trying to install will end up in an illegal instruction error -
> it's just not compiled for it (except one emulates an older architecture or
> runs in compat mode - if possible).

I think you have misunderstood what I meant when I said "support for P10 and P11". I was not referring to architectural features, e.g. new machine instructions. I was referring to new hypervisor features only available on hypervisor versions shipped for newer hardware.

> For P9 and P10 one actually has to migrate to the latest 22.04 LTS:
> https://ubuntu.com/download/server/power
> 20.04 is only still an option in case of P8.
>
> I know that getting the commits to fix something in an older release can be
> painful, esp. if cherry-picking does not work cleanly and backports are
> needed - but it's due to the Ubuntu SRU process, that only allows important
> and critical fixes anyway, so not meant for version bumps, bigger patches or
> major new features.

This fix is critical and important. If drmgr is misbehaving then the NovaLink partition cannot reliably orchestrate hotplug operations on the other LPARs. This cripples the machine's ability to interoperate with other machines in the cloud.

> We can of course (re-)build the latest version we have in the archive
> (currently v1.3.10) for focal in a PPA - and that is what I just did:
> https://launchpad.net/~fheimes/+archive/ubuntu/lp2023243
> But this is also not ideal, since this version may for example have support
> for some P10 hardware features, but the underlying 20.0...

------- Comment From <email address hidden> 2023-08-30 15:56 EDT-------
We need to add commit "powerpc-utils/devtree: Parse 'ibm,drc-info' property" (b0586b5) in order for drmgr to add a slot and remove a slot.

I believe this is because Canonical has backported kernel patch "powerpc/pseries: Enable support for ibm,drc-info property" to the latest kernel level in Focal. I believe the slots were not being added correctly and removed correctly due to parsing failures.

---

In addition, I want to recommend we add commit "drmgr: don't bother with hp children for non PCI slots" (60d9f54). When removing a slot there are some harmless error messages:

root@novap9zz:/home/neo# /home/neo/powerpc-utils-1.3.4-patched/src/drmgr/drmgr -c slot -s 'U9009.42A.13C647W-V1-C5' -r
Validating I/O DLPAR capability...yes.
failed to open /sys/bus/pci/slots/U9009.42A.13C647W-V1-C5/power: No such file or directory
failed to disable hotplug children

Non-pci devices do not have hotpluggable children. These error messages could lead a user to open another bug.

---

After applying both powerpc-utils commits I listed above and building, drmgr appears to work correctly:

root@novap9zz:/home/neo# /home/neo/powerpc-utils-1.3.4-patched/src/drmgr/drmgr -c slot -s 'U9009.42A.13C647W-V1-C5' -a
Validating I/O DLPAR capability...yes.
root@novap9zz:/home/neo# tail -n 31 /var/log/drmgr
########## Aug 30 19:15:04 2023 ##########
drmgr: -c slot -s U9009.42A.13C647W-V1-C5 -a
Validating I/O DLPAR capability...yes.
Getting node types 0x00000003
Could not find DRC property group in path: /proc/device-tree/pci@800000020000017.
Could not find DRC property group in path: /proc/device-tree/ibm,platform-keystore.
Could not find DRC property group in path: /proc/device-tree/memory@0.
Could not find DRC property group in path: /proc/device-tree/pci@800000020000017.
Could not find DRC property group in path: /proc/device-tree/ibm,platform-facilities.
Could not find DRC property group in path: /proc/device-tree/interrupt-controller@0.
Could not find DRC property group in path: /proc/device-tree/rtas.
Could not find DRC property group in path: /proc/device-tree/ibm,serial.
Could not find DRC property group in path: /proc/device-tree/options.
Could not find DRC property group in path: /proc/device-tree/openprom.
Could not find DRC property group in path: /proc/device-tree/ibm,dynamic-reconfiguration-memory.
Could not find DRC property group in path: /proc/device-tree/interrupt-controller@800000025000017.
Could not find DRC property group in path: /proc/device-tree/event-sources.
Could not find DRC property group in path: /proc/device-tree/aliases.
Could not find DRC property group in path: /proc/device-tree/packages.
Could not find DRC property group in path: /proc/device-tree/chosen.
Acquiring drc index 0x30000005
get-sensor for 30000005: 0, 2
Setting allocation state to 'alloc usable'
Setting indicator state to 'unisolate'
Configuring connector for drc index 30000005
Adding device-tree node /proc/device-tree/vdevice/vty-server@30000005
ofdt update: add_node /vdevice/vty-server@30000005 name 11 vty-server
Getting node types 0x00000003
Could not find DRC property group in path: /proc/device-tree/pci@8...

------- Comment From <email address hidden> 2023-09-07 15:25 EDT-------
(In reply to comment #36)
> We need to add commit "powerpc-utils/devtree: Parse 'ibm,drc-info' property"
> (b0586b5) in order for drmgr to add a slot and remove a slot.
>
> I believe this is because Canonical has backported kernel patch
> "powerpc/pseries: Enable support for ibm,drc-info property" to the latest
> kernel level in Focal. I believe the slots were not being added correctly
> and removed correctly due to parsing failures.
>
> ---
>
> In addition, I want to recommend we add commit "drmgr: don't bother with hp
> children for non PCI slots" (60d9f54). When removing a slot there are some
> harmless error messages:
>
>
> root@novap9zz:/home/neo#
> /home/neo/powerpc-utils-1.3.4-patched/src/drmgr/drmgr -c slot -s
> 'U9009.42A.13C647W-V1-C5' -r
> Validating I/O DLPAR capability...yes.
> failed to open /sys/bus/pci/slots/U9009.42A.13C647W-V1-C5/power: No such
> file or directory
> failed to disable hotplug children
>
>
> Non-pci devices do not have hotpluggable children. These error messages
> could lead a user to open another bug.
>
> ---
>
> After applying both powerpc-utils commits I listed above and building, drmgr
> appears to work correctly:
>
>
> root@novap9zz:/home/neo#
> /home/neo/powerpc-utils-1.3.4-patched/src/drmgr/drmgr -c slot -s
> 'U9009.42A.13C647W-V1-C5' -a
> Validating I/O DLPAR capability...yes.
> root@novap9zz:/home/neo# tail -n 31 /var/log/drmgr
> ########## Aug 30 19:15:04 2023 ##########
> drmgr: -c slot -s U9009.42A.13C647W-V1-C5 -a
> Validating I/O DLPAR capability...yes.
> Getting node types 0x00000003
> Could not find DRC property group in path:
> /proc/device-tree/pci@800000020000017.
> Could not find DRC property group in path:
> /proc/device-tree/ibm,platform-keystore.
> Could not find DRC property group in path: /proc/device-tree/memory@0.
> Could not find DRC property group in path:
> /proc/device-tree/pci@800000020000017.
> Could not find DRC property group in path:
> /proc/device-tree/ibm,platform-facilities.
> Could not find DRC property group in path:
> /proc/device-tree/interrupt-controller@0.
> Could not find DRC property group in path: /proc/device-tree/rtas.
> Could not find DRC property group in path: /proc/device-tree/ibm,serial.
> Could not find DRC property group in path: /proc/device-tree/options.
> Could not find DRC property group in path: /proc/device-tree/openprom.
> Could not find DRC property group in path:
> /proc/device-tree/ibm,dynamic-reconfiguration-memory.
> Could not find DRC property group in path:
> /proc/device-tree/interrupt-controller@800000025000017.
> Could not find DRC property group in path: /proc/device-tree/event-sources.
> Could not find DRC property group in path: /proc/device-tree/aliases.
> Could not find DRC property group in path: /proc/device-tree/packages.
> Could not find DRC property group in path: /proc/device-tree/chosen.
> Acquiring drc index 0x30000005
> get-sensor for 30000005: 0, 2
> Setting allocation state to 'alloc usable'
> Setting indicator state to 'unisolate'
> Configuring connector for drc index 30000005
> Adding device-tree node /proc/device-tree/vdevice/vty-server@30000005
> ofdt...

------- Comment (attachment only) From <email address hidden> 2023-09-07 15:23 EDT-------

------- Comment (attachment only) From <email address hidden> 2023-09-07 15:24 EDT-------

thanks for sending the patches.

both are upstream since v1.3.8

patricia@lh:~$ rmadison powerpc-utils
 powerpc-utils | 1.2.18+r1.1.3-24ubuntu1 | trusty | source, powerpc, ppc64el
 powerpc-utils | 1.3.1-2 | xenial | source, powerpc, ppc64el
 powerpc-utils | 1.3.1-2ubuntu0.3 | xenial-updates | source, powerpc, ppc64el
 powerpc-utils | 1.3.4-0ubuntu2 | bionic | source, ppc64el
 powerpc-utils | 1.3.4-0ubuntu2 | focal | source, ppc64el
 powerpc-utils | 1.3.9-1ubuntu1 | jammy | source, ppc64el
 powerpc-utils | 1.3.10-2 | lunar | source, ppc64el
 powerpc-utils | 1.3.11-1 | mantic | source, ppc64el

which means jammy and lunar already have them. So we can delete jammy and lunar as affected releases.

A test build of a patched package for Focal is available at this PPA:
https://launchpad.net/~patriciasd/+archive/ubuntu/focal.powerpc-utils.lp2023243

attaching the debdiff file for FOCAL

I uploaded the powerpc-utils patch to Focal, thanks.

I'm assuming folks at IBM are willing to help with the verification once it's accepted?

Hello, thanks Simon. Yes, IBM is aware that we will need then for the verification.

------- Comment From <email address hidden> 2023-09-29 14:05 EDT-------
Attaching the latest error message and stack trace

neo@p8-110-neo:~$ mkvterm --id 8
[PVME03000147-0404] Virtual adapter defined for virtual slot number 5 is in use and cannot be removed.

Error creating a vterm device:

neo@p8-110-neo:~$ mkvterm --id 8
[PVME03000147-0404] Virtual adapter defined for virtual slot number 5 is in use and cannot be removed.

Error creating a vterm device:

neo@p8-110-neo:~$

root@p8-110-neo:/home/neo# /usr/sbin/mkvtermutil --id 8
+ . gettext.sh
++ test 'X\t' = 'X\t'
++ echo=echo
++ test -z ''
++ case "$0" in
+ hvcsadmin=/usr/sbin/hvcsadmin
+ hvcsdevlimit=1024
+ pvmutil=/usr/sbin/pvmutil
+ socat=/usr/bin/socat
+ rmvterm=/usr/sbin/rmvtermutil
+ openvt=/bin/openvt
+ '[' -f /usr/bin/linuxvnc ']'
+ linuxvnc=/usr/bin/linuxvnc
+ flock=/usr/bin/flock
+ modprobe=/sbin/modprobe
+ slattach=/sbin/slattach
+ ifconfig=/sbin/ifconfig
+ tcpdump=/usr/sbin/tcpdump
+ Xvfb=/usr/bin/Xvfb
+ x11vnc=/usr/bin/x11vnc
+ java=/opt/pvm-rest/lib/wlp/java/bin/java
+ xinit=/usr/bin/xinit
+ bash=/bin/bash
+ xterm=/usr/bin/xterm
+ pgrep=/usr/bin/pgrep
+ giveUsage=0
+ lparId=0
+ logFileDir=/var/log/pvm/mkvtermLogs
+ logFilePath=
+ lparIsIBMi=false
+ vnc=0
+ localVNC=0
+ ttyNumber=-1
+ vncPort=
+ vncPasswd=
+ vncDesktop=
+ lockFile=/tmp/vterm.lck
+ lockTimeout=15
+ lockFD=3
+ logFile=
+ socatParms=STDIO,raw,echo=0,escape=0x1d
+ socatParmsVNC=STDIO,raw,echo=0
+ useXterm=false
+ xtermParms='-fa xft:DejaVuSansMono:size=12 +sb -rv -e '
+ xtermSuffix1=' -- /usr/bin/Xvfb'
+ xtermSuffix2=' -screen 0 804x484x8 -nolisten tcp'
+ vncPortFile=/var/run/vterm_rfbport_lp
+ ptsLink=/var/run/5250_lp
+ socatParms5250=pty,raw,echo=0,link=/var/run/5250_lp
+ prefixLocal5250=10.252
+ prefixRemote5250=10.253
+ xvfbParms='-screen 0 976x615x8'
+ emulatorLocation=/opt/pvm/5250/
+ emulatorSettingsLocation=/etc/pvm/data/
+ emulatorSettingsLocationForLpar=/etc/pvm/data/lpar/
+ emulatorParms='-Xmx40m -cp /opt/pvm/5250/acshmchod.jar com.ibm.eNetwork.hmchod.HmcHod '
+ emulatorParmsSuffix1=' -port 2300 -title emulator_settings X X '
+ emulatorParmsSuffix2=' -- /usr/bin/Xvfb'
+ emulatorParmsSuffix3=' -screen 0 1024x768x24 -nolisten tcp'
+ RET_FAIL=1
+ RET_ITEMS_NOT_INSTALLED=2
+ RET_VTERM_ALREADY_STARTED=3
+ RET_OUT_OF_VTERMS=4
+ RET_USAGE=99
+ socatPid=-1
+ slattachPid=-1
+ declare -A lang_code
+ lang_code=([930]=0 [e930]=1 [939]=2 [1399]=3 [e1399]=4)
+ codePage=(930 930 939 1399 1399)
+ codePageKey=(KEY_JAPAN_KATAKANA KEY_JAPAN_KATAKANA_EX KEY_JAPAN_ENGLISH_EX KEY_JAPAN_LATIN_EXTENDED KEY_JAPAN_LATIN_EXTENDED_JIS2004)
+ fontName=(Monospaced)
+ pdfFont=(KozMinPro-Regular-Acro)
++ getopt -o h --long id:,help,vnc,local,passwdfile:,log:,vncport:,vncdesktop:,codepage: -- --id 8
+ TEMP=' --id '\''8'\'' --'
+ '[' 0 -ne 0 ']'
+ eval set -- ' --id '\''8'\'' --'
++ set -- --id 8 --
+ true
+ case "$1" in
+ lparId=8
+ shift 2
+ true
+ case "$1" in
+ break
+ [[ 8 -eq 0 ]]
+ [[ 8 =~ ^[0-9]+$ ]]
+ [[ ! -d /var/log/pvm/mkvtermLogs ]]
+ logFilePath=/var/log/pvm/mkvtermLogs/vtermLog_8.log
+ cat /dev/null
+ [[ -n '' ]]
+ [[ -n '' ]]
+ [[ -n '' ]]
+ /usr/sbin/pvmutil -o getrmcstatus
+ grep lpar_id=8,
+ grep os400
+ '[' 1 -eq 0...

------- Comment From <email address hidden> 2023-10-10 11:16 EDT-------
Updating this bug with the latest findings.

Please ignore the past error output. Something was off with that system. This is actually the current error that is being seen:

-----
root@novap9fleetw:/home/neo# /usr/sbin/mkvtermutil_temp --id 2
+ export TEXTDOMAINDIR
+ . gettext.sh
++ test 'X\t' = 'X\t'
++ echo=echo
++ test -z ''
++ case "$0" in
+ hvcsadmin=/usr/sbin/hvcsadmin
+ hvcsdevlimit=1024
+ pvmutil=/usr/sbin/pvmutil
+ socat=/usr/bin/socat
+ rmvterm=/usr/sbin/rmvtermutil
+ openvt=/bin/openvt
+ '[' -f /usr/bin/linuxvnc ']'
+ linuxvnc=/usr/bin/linuxvnc
+ flock=/usr/bin/flock
+ modprobe=/sbin/modprobe
+ slattach=/sbin/slattach
+ ifconfig=/sbin/ifconfig
+ tcpdump=/usr/sbin/tcpdump
+ Xvfb=/usr/bin/Xvfb
+ x11vnc=/usr/bin/x11vnc
+ java=/opt/pvm-rest/lib/wlp/java/bin/java
+ xinit=/usr/bin/xinit
+ bash=/bin/bash
+ xterm=/usr/bin/xterm
+ pgrep=/usr/bin/pgrep
+ giveUsage=0
+ lparId=0
+ logFileDir=/var/log/pvm/mkvtermLogs
+ logFilePath=
+ lparIsIBMi=false
+ vnc=0
+ localVNC=0
+ ttyNumber=-1
+ vncPort=
+ vncPasswd=
+ vncDesktop=
+ lockFile=/tmp/vterm.lck
+ lockTimeout=15
+ lockFD=3
+ logFile=
+ socatParms=STDIO,raw,echo=0,escape=0x1d
+ socatParmsVNC=STDIO,raw,echo=0
+ useXterm=false
+ xtermParms='-fa xft:DejaVuSansMono:size=12 +sb -rv -e '
+ xtermSuffix1=' -- /usr/bin/Xvfb'
+ xtermSuffix2=' -screen 0 804x484x8 -nolisten tcp'
+ vncPortFile=/var/run/vterm_rfbport_lp
+ ptsLink=/var/run/5250_lp
+ socatParms5250=pty,raw,echo=0,link=/var/run/5250_lp
+ prefixLocal5250=10.252
+ prefixRemote5250=10.253
+ xvfbParms='-screen 0 976x615x8'
+ emulatorLocation=/opt/pvm/5250/
+ emulatorSettingsLocation=/etc/pvm/data/
+ emulatorSettingsLocationForLpar=/etc/pvm/data/lpar/
+ emulatorParms='-Xmx40m -cp /opt/pvm/5250/acshmchod.jar com.ibm.eNetwork.hmchod.HmcHod '
+ emulatorParmsSuffix1=' -port 2300 -title emulator_settings X X '
+ emulatorParmsSuffix2=' -- /usr/bin/Xvfb'
+ emulatorParmsSuffix3=' -screen 0 1024x768x24 -nolisten tcp'
+ RET_FAIL=1
+ RET_ITEMS_NOT_INSTALLED=2
+ RET_VTERM_ALREADY_STARTED=3
+ RET_OUT_OF_VTERMS=4
+ RET_USAGE=99
+ socatPid=-1
+ slattachPid=-1
+ declare -A lang_code
+ lang_code=([930]=0 [e930]=1 [939]=2 [1399]=3 [e1399]=4)
+ codePage=(930 930 939 1399 1399)
+ codePageKey=(KEY_JAPAN_KATAKANA KEY_JAPAN_KATAKANA_EX KEY_JAPAN_ENGLISH_EX KEY_JAPAN_LATIN_EXTENDED KEY_JAPAN_LATIN_EXTENDED_JIS2004)
+ fontName=(Monospaced)
+ pdfFont=(KozMinPro-Regular-Acro)
++ getopt -o h --long id:,help,vnc,local,passwdfile:,log:,vncport:,vncdesktop:,codepage: -- --id 2
+ TEMP=' --id '\''2'\'' --'
+ '[' 0 -ne 0 ']'
+ eval set -- ' --id '\''2'\'' --'
++ set -- --id 2 --
+ true
+ case "$1" in
+ lparId=2
+ shift 2
+ true
+ case "$1" in
+ break
+ [[ 2 -eq 0 ]]
+ [[ 2 =~ ^[0-9]+$ ]]
+ [[ ! -d /var/log/pvm/mkvtermLogs ]]
+ logFilePath=/var/log/pvm/mkvtermLogs/vtermLog_2.log
+ cat /dev/null
+ [[ -n '' ]]
+ [[ -n '' ]]
+ [[ -n '' ]]
+ /usr/sbin/pvmutil -o getrmcstatus
+ grep lpar_id=2,
+ grep os400
+ '[' 1 -eq 0 ']'
+ [[ -n '' ]]
+ [[ -n '' ]]
+ [[ -n '' ]]
+ [[ false = true ]]
+ [[ -n '' ]]
+ [[ false = false ]]
+ [[ -e /usr/bin/xinit ]]
+ [[ -e /usr/bin/Xvfb ]]
+ [[ -e /usr/bin/xterm ]]
+ [[ -e /usr/bin/x11vnc...

Hello Jamie, many thanks for checking and updating this, specially `There is no longer an issue seen in drmgr with adding and removing a slot. Therefore, the patches did correct the drmgr issue.`

Please, if you could file a new/separate bug with the new issue you are seeing now. Thanks!

Hi, SRU team member here. I read the comments, and considering the problem statement from the bug description:
```
1. drmgr command fails as it doesn't find a slot
2. drmgr throws error while removing the slot
3. ctrl+c closes the terminal opened for any vm
4. ctrl+] doesn't close the terminal. But comes out with an extra 'Enter'
```

I didn't see confirmation in the comments that problems (3) and (4) are also fixed by the proposed two patches. comment #24 states that there are missing devices in /dev, and it's not clear to me if that has relation to the bugs being fixed here, or if they are new bugs. Or even if the powerpc-utils package would be responsible for that (sounds like an issue with some udev rule?).

We can accept the package as is, and the testing with the proposed packages will tell, as the testing must verify that all 4 issues are fixed, but I would like to give a chance to the people involved to clarify this issue first.

It might also well be that the patched package is already better than what we have in focal currently, and the missing devices are indeed a new separate bug, in which case I would echo what Patricia said in comment #25 and ask for a new bug to be filed.

In short, can someone please clarify on whether the two patches are enough and that we can proceed with accepting the package into focal-proposed, or if more investigation is needed?

------- Comment From <email address hidden> 2023-11-10 13:38 EDT-------
I am currently unsure with the test systems provided that have Focal installed - why the /dev/hvcs* device nodes are present sometimes and not present other times. Could the Novalink team please comment if they found a workaround for this? Or is this an intermittent issue?

However - on a system without this absent device node issue, I installed the updated powerpc-utils, and I can confirm that the drmgr issues are resolved.

Unfortunately, issues #3 and #4 are still present. ctrl+c does close the terminal, and ctrl+] does not close the terminal (but does eventually with an extra 'Enter'). I believe #3 and #4 are related to socat. Seems to be the same issues mentioned in bug 192082. Comment #16 mentions the patch.

In bug 192082 it also mentioned that socat version 1.7.3.3 introduced this issue. This is the same version of socat Focal has.

Do we want to open another bug for issues #3 and #4 since it seems to be related to socat and not drmgr?

Thanks Jamie for re-confirming that #1 and #2 are fixed.

And yes, I would also prefer to handle any other issue (#3 and #4) separately - means in a dedicated bug (we are happy to do the split up).
At the end this would allow us to make progress here.

@SRU team (ahasenack), with that, would it be okay to proceed with this SRU only covering issues #1 and #2? (The SRU justification in the bug description was adjusted, #3 and #4 were removed here).

Hello bugproxy, or anyone else affected,

Accepted powerpc-utils into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/powerpc-utils/1.3.4-0ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

> @SRU team (ahasenack), with that, would it be okay to proceed with this SRU only covering issues #1 and
> #2? (The SRU justification in the bug description was adjusted, #3 and #4 were removed here).

Yes :)

The fix for this bug has been awaiting testing feedback in the -proposed repository for focal for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

------- Comment From <email address hidden> 2024-03-06 15:22 EDT-------
The version of powerpc-utils tested was 1.3.4-0ubuntu2.1 from the -proposed repository.

Much testing was done with adding and removing slots. See logs below:

########## Mar 06 16:58:32 2024 ##########
drmgr: -c slot -s U9080.M9S.13072B8-V1-C7 -a -w 3
Validating I/O DLPAR capability...yes.
Getting node types 0x00000003
Could not find DRC property group in path: /proc/device-tree/ibm,platform-keystore.
Could not find DRC property group in path: /proc/device-tree/memory@0.
Could not find DRC property group in path: /proc/device-tree/ibm,platform-facilities.
Could not find DRC property group in path: /proc/device-tree/interrupt-controller@0.
Could not find DRC property group in path: /proc/device-tree/rtas.
Could not find DRC property group in path: /proc/device-tree/ibm,serial.
Could not find DRC property group in path: /proc/device-tree/options.
Could not find DRC property group in path: /proc/device-tree/openprom.
Could not find DRC property group in path: /proc/device-tree/ibm,dynamic-reconfiguration-memory.
Could not find DRC property group in path: /proc/device-tree/event-sources.
Could not find DRC property group in path: /proc/device-tree/aliases.
Could not find DRC property group in path: /proc/device-tree/packages.
Could not find DRC property group in path: /proc/device-tree/chosen.
Acquiring drc index 0x30000007
get-sensor for 30000007: 0, 2
Setting allocation state to 'alloc usable'
Setting indicator state to 'unisolate'
Configuring connector for drc index 30000007
Adding device-tree node /proc/device-tree/vdevice/vty-server@30000007
ofdt update: add_node /vdevice/vty-server@30000007 name 11 vty-server
Getting node types 0x00000003
performing kernel op for U9080.M9S.13072B8-V1-C7, file is /sys/bus/pci/slots/control/add_slot
########## Mar 06 16:58:33 2024 ##########

########## Mar 06 16:59:07 2024 ##########
drmgr: -c slot -s U9080.M9S.13072B8-V1-C7 -r -w 3
Validating I/O DLPAR capability...yes.
Getting node types 0x00000003
get-sensor for 30000007: 0, 1
The sensor-state of drc_index 0x30000007 is 1
performing kernel op for U9080.M9S.13072B8-V1-C7, file is /sys/bus/pci/slots/control/remove_slot
Releasing drc index 0x30000007
get-sensor for 30000007: 0, 1
Setting isolation state to 'isolate'
Setting allocation state to 'alloc unusable'
get-sensor for 30000007: 0, 2
drc_index 30000007 sensor-state: 2
Resource is not available to the partition.
Removing device-tree node /proc/device-tree/vdevice/vty-server@30000007
########## Mar 06 16:59:07 2024 ##########

Testing was periodically stalled when I did an strace on pvmutil creating a vterm device (e.g. 'pvmutil -o create_vterm_dev --id 3'). mkvterm calls pvmutil. pvmutil is what uses drmgr. There was confusion when the drmgr logs showed drmgr working properly, however an strace showed writing the slot information to .../add_slot was returning -ENODEV. It was determined that this was not an issue with drmgr, as drmgr was writing the correct slot information.

Therefore, the conclusion is the version of powerpc-utils in -proposed is working as expected.

I am going to open two additional bugs. One for socat, and one for ...

Many thanks for your testing and feedback on focal @jldolan.
With that I'm updating the tags to verification done.

(The two additional bugs about kernel and socat will then be addressed in their own LP bug, I already noticed one of them.)

This bug was fixed in the package powerpc-utils - 1.3.4-0ubuntu2.1

---------------
powerpc-utils (1.3.4-0ubuntu2.1) focal; urgency=medium

  * Cherrypick patches to solve issues with drmgr (LP: #2023243)

 -- Patricia Domingues <email address hidden> Thu, 14 Sep 2023 12:40:06 -0300

The verification of the Stable Release Update for powerpc-utils has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.