Comment 24 for bug 2023243

------- Comment From <email address hidden> 2023-10-10 11:16 EDT-------
Updating this bug with the latest findings.

Please ignore the past error output. Something was off with that system. This is actually the current error that is being seen:

-----
root@novap9fleetw:/home/neo# /usr/sbin/mkvtermutil_temp --id 2
+ export TEXTDOMAINDIR
+ . gettext.sh
++ test 'X\t' = 'X\t'
++ echo=echo
++ test -z ''
++ case "$0" in
+ hvcsadmin=/usr/sbin/hvcsadmin
+ hvcsdevlimit=1024
+ pvmutil=/usr/sbin/pvmutil
+ socat=/usr/bin/socat
+ rmvterm=/usr/sbin/rmvtermutil
+ openvt=/bin/openvt
+ '[' -f /usr/bin/linuxvnc ']'
+ linuxvnc=/usr/bin/linuxvnc
+ flock=/usr/bin/flock
+ modprobe=/sbin/modprobe
+ slattach=/sbin/slattach
+ ifconfig=/sbin/ifconfig
+ tcpdump=/usr/sbin/tcpdump
+ Xvfb=/usr/bin/Xvfb
+ x11vnc=/usr/bin/x11vnc
+ java=/opt/pvm-rest/lib/wlp/java/bin/java
+ xinit=/usr/bin/xinit
+ bash=/bin/bash
+ xterm=/usr/bin/xterm
+ pgrep=/usr/bin/pgrep
+ giveUsage=0
+ lparId=0
+ logFileDir=/var/log/pvm/mkvtermLogs
+ logFilePath=
+ lparIsIBMi=false
+ vnc=0
+ localVNC=0
+ ttyNumber=-1
+ vncPort=
+ vncPasswd=
+ vncDesktop=
+ lockFile=/tmp/vterm.lck
+ lockTimeout=15
+ lockFD=3
+ logFile=
+ socatParms=STDIO,raw,echo=0,escape=0x1d
+ socatParmsVNC=STDIO,raw,echo=0
+ useXterm=false
+ xtermParms='-fa xft:DejaVuSansMono:size=12 +sb -rv -e '
+ xtermSuffix1=' -- /usr/bin/Xvfb'
+ xtermSuffix2=' -screen 0 804x484x8 -nolisten tcp'
+ vncPortFile=/var/run/vterm_rfbport_lp
+ ptsLink=/var/run/5250_lp
+ socatParms5250=pty,raw,echo=0,link=/var/run/5250_lp
+ prefixLocal5250=10.252
+ prefixRemote5250=10.253
+ xvfbParms='-screen 0 976x615x8'
+ emulatorLocation=/opt/pvm/5250/
+ emulatorSettingsLocation=/etc/pvm/data/
+ emulatorSettingsLocationForLpar=/etc/pvm/data/lpar/
+ emulatorParms='-Xmx40m -cp /opt/pvm/5250/acshmchod.jar com.ibm.eNetwork.hmchod.HmcHod '
+ emulatorParmsSuffix1=' -port 2300 -title emulator_settings X X '
+ emulatorParmsSuffix2=' -- /usr/bin/Xvfb'
+ emulatorParmsSuffix3=' -screen 0 1024x768x24 -nolisten tcp'
+ RET_FAIL=1
+ RET_ITEMS_NOT_INSTALLED=2
+ RET_VTERM_ALREADY_STARTED=3
+ RET_OUT_OF_VTERMS=4
+ RET_USAGE=99
+ socatPid=-1
+ slattachPid=-1
+ declare -A lang_code
+ lang_code=([930]=0 [e930]=1 [939]=2 [1399]=3 [e1399]=4)
+ codePage=(930 930 939 1399 1399)
+ codePageKey=(KEY_JAPAN_KATAKANA KEY_JAPAN_KATAKANA_EX KEY_JAPAN_ENGLISH_EX KEY_JAPAN_LATIN_EXTENDED KEY_JAPAN_LATIN_EXTENDED_JIS2004)
+ fontName=(Monospaced)
+ pdfFont=(KozMinPro-Regular-Acro)
++ getopt -o h --long id:,help,vnc,local,passwdfile:,log:,vncport:,vncdesktop:,codepage: -- --id 2
+ TEMP=' --id '\''2'\'' --'
+ '[' 0 -ne 0 ']'
+ eval set -- ' --id '\''2'\'' --'
++ set -- --id 2 --
+ true
+ case "$1" in
+ lparId=2
+ shift 2
+ true
+ case "$1" in
+ break
+ [[ 2 -eq 0 ]]
+ [[ 2 =~ ^[0-9]+$ ]]
+ [[ ! -d /var/log/pvm/mkvtermLogs ]]
+ logFilePath=/var/log/pvm/mkvtermLogs/vtermLog_2.log
+ cat /dev/null
+ [[ -n '' ]]
+ [[ -n '' ]]
+ [[ -n '' ]]
+ /usr/sbin/pvmutil -o getrmcstatus
+ grep lpar_id=2,
+ grep os400
+ '[' 1 -eq 0 ']'
+ [[ -n '' ]]
+ [[ -n '' ]]
+ [[ -n '' ]]
+ [[ false = true ]]
+ [[ -n '' ]]
+ [[ false = false ]]
+ [[ -e /usr/bin/xinit ]]
+ [[ -e /usr/bin/Xvfb ]]
+ [[ -e /usr/bin/xterm ]]
+ [[ -e /usr/bin/x11vnc ]]
+ useXterm=true
++ expr 1024 - 1
+ '[' '!' -e /dev/hvcs1023 ']'
+ echo '/sbin/modprobe -r hvcs'
+ /sbin/modprobe -r hvcs
+ echo '/sbin/modprobe hvcs hvcs_parm_num_devs=1024'
+ /sbin/modprobe hvcs hvcs_parm_num_devs=1024
+ [[ false = true ]]
+ pvmhvcsadmin=/tmp/pvmhvcsadmin
+ '[' -s /tmp/pvmhvcsadmin ']'
+ hvcsadmin=/tmp/pvmhvcsadmin
+ chmod +x /tmp/pvmhvcsadmin
+ grep -F '=~ /.+(3\d+)$/' /tmp/pvmhvcsadmin
+ '[' 1 -eq 0 ']'
++ /usr/sbin/pvmutil -o create_vterm_dev --id 2
+ createDev=drc_index=30000006,drc_name=U9080.M9S.13072B8-V1-C6
+ '[' 0 -ne 0 ']'
+ exec
+ /usr/bin/flock -x -w 15 3
+ '[' 0 -ne 0 ']'
+ trap '$flock -u $lockFD' SIGHUP SIGINT SIGTERM EXIT
+ echo '/tmp/pvmhvcsadmin -all'
++ /tmp/pvmhvcsadmin -all
+ hvcsCloseAll=
+ echo 'hvcsCloseAll '
++ /tmp/pvmhvcsadmin -status
++ grep 'partition:2 '
+ hvcsConsoleString=
+ echo 'hvcsConsoleString '
+ [[ -z '' ]]
+ gettext 'Warning - vterm not found. You likely have run out of vterm devices.'
Warning - vterm not found. You likely have run out of vterm devices.+ echo

+ gettext 'Attempting to clean up unused devices. Please wait...'
Attempting to clean up unused devices. Please wait...+ echo

+ echo 'Running command "/usr/sbin/pvmutil -o delete_vterm_dev --id 2"'
++ /usr/sbin/pvmutil -o delete_vterm_dev --id 2
+ deleteDev=
+ '[' 0 -ne 0 ']'
+ echo 'Running command "/usr/sbin/rmvtermutil --inactive"'
+ /usr/sbin/rmvtermutil --inactive
+ echo 'Running command "/usr/sbin/pvmutil -o create_vterm_dev --id 2"'
++ /usr/sbin/pvmutil -o create_vterm_dev --id 2
+ createDev=drc_index=30000006,drc_name=U9080.M9S.13072B8-V1-C6
+ '[' 0 -ne 0 ']'
+ echo 'Running command "/tmp/pvmhvcsadmin -status"'
+ /tmp/pvmhvcsadmin -status
+ grep 'no hvcs adapters'
+ [[ 0 -eq 0 ]]
+ echo 'Running command "/sbin/modprobe -r hvcs"'
+ /sbin/modprobe -r hvcs
+ echo 'Running command " /sbin/modprobe hvcs hvcs_parm_num_devs=1024"'
+ /sbin/modprobe hvcs hvcs_parm_num_devs=1024
+ echo 'Running command "/tmp/pvmhvcsadmin -status "'
++ /tmp/pvmhvcsadmin -status
++ grep 'partition:2 '
+ hvcsConsoleString=
+ echo 'hvcsConsoleString '
+ [[ -z '' ]]
+ gettext 'Error - vterm still not found. You may need to close active vterm devices.'
Error - vterm still not found. You may need to close active vterm devices.+ echo

+ echo 'Running command "/usr/sbin/pvmutil -o delete_vterm_dev --id 2"'
+ /usr/sbin/pvmutil -o delete_vterm_dev --id 2
+ exit 4
+ /usr/bin/flock -u 3
-----

There is no longer an issue seen in drmgr with adding and removing a slot. Therefore, the patches did correct the drmgr issue. From /var/log/drmgr:

-----
########## Oct 10 14:49:00 2023 ##########
drmgr: -c slot -s U9080.M9S.13072B8-V1-C6 -a -w 3
Validating I/O DLPAR capability...yes.
Getting node types 0x00000003
Could not find DRC property group in path: /proc/device-tree/ibm,platform-keystore.
Could not find DRC property group in path: /proc/device-tree/memory@0.
Could not find DRC property group in path: /proc/device-tree/ibm,platform-facilities.
Could not find DRC property group in path: /proc/device-tree/interrupt-controller@0.
Could not find DRC property group in path: /proc/device-tree/rtas.
Could not find DRC property group in path: /proc/device-tree/ibm,serial.
Could not find DRC property group in path: /proc/device-tree/options.
Could not find DRC property group in path: /proc/device-tree/openprom.
Could not find DRC property group in path: /proc/device-tree/ibm,dynamic-reconfiguration-memory.
Could not find DRC property group in path: /proc/device-tree/event-sources.
Could not find DRC property group in path: /proc/device-tree/aliases.
Could not find DRC property group in path: /proc/device-tree/packages.
Could not find DRC property group in path: /proc/device-tree/chosen.
Acquiring drc index 0x30000006
get-sensor for 30000006: 0, 2
Setting allocation state to 'alloc usable'
Setting indicator state to 'unisolate'
Configuring connector for drc index 30000006
Adding device-tree node /proc/device-tree/vdevice/vty-server@30000006
ofdt update: add_node /vdevice/vty-server@30000006 name 11 vty-server
Getting node types 0x00000003
performing kernel op for U9080.M9S.13072B8-V1-C6, file is /sys/bus/pci/slots/control/add_slot
########## Oct 10 14:49:00 2023 ##########

########## Oct 10 14:49:00 2023 ##########
drmgr: -c slot -s U9080.M9S.13072B8-V1-C6 -r -w 3
Validating I/O DLPAR capability...yes.
Getting node types 0x00000003
get-sensor for 30000006: 0, 1
The sensor-state of drc_index 0x30000006 is 1
performing kernel op for U9080.M9S.13072B8-V1-C6, file is /sys/bus/pci/slots/control/remove_slot
Releasing drc index 0x30000006
get-sensor for 30000006: 0, 1
Setting isolation state to 'isolate'
Setting allocation state to 'alloc unusable'
get-sensor for 30000006: 0, 2
drc_index 30000006 sensor-state: 2
Resource is not available to the partition.
Removing device-tree node /proc/device-tree/vdevice/vty-server@30000006
########## Oct 10 14:49:00 2023 ##########
-----

The issue now is when modprobe loads hvcs, the character device nodes are not being created.

-----
root@novap9fleetw:/home/neo# lsmod |grep hvcs
root@novap9fleetw:/home/neo# modprobe hvcs hvcs_parm_num_devs=1024
root@novap9fleetw:/home/neo# lsmod |grep hvcs
hvcs 22244 0
hvcserver 4733 1 hvcs
root@novap9fleetw:/home/neo# ls -l /dev | grep hvcs
----

This is why mkvterm eventually fails. pvmutil is working when it calls drmgr. But then mkvterm fails because it calls 'hvcsadmin -status', which outputs no adapters are found. hvcsadmin outputs this because it does not find any /dev/hvcs* device nodes. I checked the hvcs code in the kernel, and it expects the major number assignment to be dynamic. I am unsure why udev is not creating the device nodes when the module is loaded. They are being created in a RHEL environment.