[UBUNTU 22.04] s390/qeth: cache link_info for ethtool

Status changed to 'Confirmed' because the bug affects multiple users.

Hello Frank,

I mirrored this while Boris is on vacation for your awareness - please adjust accordingly.
Thanks

------- Comment From <email address hidden> 2022-08-10 04:10 EDT-------
clarification from development on the overall situation:

Symptoms:
A) Since
02d5cb5bb20b ("qeth: Accurate ethtool output")
query_card_info is called for each qeth_get_link_ksettings
Frequent calls to ethtool?s get_link_ksetting can put pressure on the zVM VSwitch
that lead to timeouts in the qeth driver that responds with recovering the affected interface.

B) Since
e6e771b3d897 ("s390/qeth: detach netdevice while card is offline")
there is a window condition that query_card_info can interfere with recovery and
lead to failing recovery.

7a07a29e4f67 ("s390/qeth: cache link_info for ethtool") solves both problems.

Backport considerations:
A) 02d5cb5bb20b ("qeth: Accurate ethtool output")
went into kernel v3.14 and was backported to RHEL7.9

B) e6e771b3d897 ("s390/qeth: detach netdevice while card is offline")
went into kernel v5.1 and was backported to RHEL8.2

canonical-kernel-team does not monitor or respond to userpsace ethtool package bugs; this bug should have gone to the server team triage.

------- Comment From <email address hidden> 2022-08-30 08:13 EDT-------
(In reply to comment #12)
> canonical-kernel-team does not monitor or respond to userpsace ethtool
> package bugs; this bug should have gone to the server team triage.

This is a _kernel_ patch that fixes output used by ethtool.

Yepp, it's all about kernel patches.
This bug was wrongly marked as affecting the ethtool (well, it affects the ethtool, but not in the 'affects' send of Launchpad), which caused some confusion.
I just changed that, pointing now to the kernel (which is "linux (Ubuntu)" in the above affects section).

I prepared a test kernel with the proposed patch (that was a clear cherry pick for 5.15). Frank do you think you can help me validating the test kernel?

You can find a tarball with the debian packages for the test kernel at:

https://kernel.ubuntu.com/~mhcerri/test/lp1984103/linux-unsigned-debs-5.15.0-48.54+lp1984103.1_s390x.tgz

------- Comment From <email address hidden> 2022-08-31 12:47 EDT-------
new test kernel works as expected:

Linux version 5.15.0-47-generic

# grep qcrdinfo /5.15.0-48-genericsys/kernel/debug/s390dbf/qeth*/hex_ascii | wc -l
7
# ethtool encbd00 | grep Duplex
Duplex: Full
# grep qcrdinfo /sys/kernel/debug/s390dbf/qeth*/hex_ascii | wc -l
9

afterupdate to Linux version 5.15.0-48-generic

# grep qcrdinfo /sys/kernel/debug/s390dbf/qeth*/hex_ascii | wc -l
0
# ethtool encbd00 | grep Duplex
Duplex: Full
# grep qcrdinfo /sys/kernel/debug/s390dbf/qeth*/hex_ascii | wc -l
0
# ethtool encbd00
Settings for encbd00:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseSR/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10000baseSR/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: on
Port: FIBRE
PHYAD: 0
Transceiver: internal
Link detected: yes

SRU request submitted to UKT mailing list:
https://lists.ubuntu.com/archives/kernel-team/2022-September/133102.html

This bug is awaiting verification that the linux/5.15.0-50.56 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Verification done on jammy (on a slightly different setup):
root@s1lp15:~# uname -a
Linux s1lp15 5.15.0-27-generic #28-Ubuntu SMP Thu Apr 14 04:55:23 UTC 2022 s390x s390x s390x GNU/Linux
root@s1lp15:~# ethtool encc000 | grep Duplex
 Duplex: Full
root@s1lp15:~# grep qcrdinfo /sys/kernel/debug/s390dbf/qeth*/hex_ascii | wc -l
3
root@s1lp15:~# grep qcrdinfo /sys/kernel/debug/s390dbf/qeth*/hex_ascii
/sys/kernel/debug/s390dbf/qeth_card_0.0.c000/hex_ascii:00 01664021373:134002 2 - 0001 000003ff81066676 71 63 72 64 69 6e 66 6f | qcrdinfo
/sys/kernel/debug/s390dbf/qeth_card_0.0.c000/hex_ascii:00 01664021384:308308 2 - 0007 000003ff81066676 71 63 72 64 69 6e 66 6f | qcrdinfo
/sys/kernel/debug/s390dbf/qeth_card_0.0.c000/hex_ascii:00 01664021384:308804 2 - 0007 000003ff81066676 71 63 72 64 69 6e 66 6f | qcrdinfo
root@s1lp15:~# ethtool encc000
Settings for encc000:
 Supported ports: [ FIBRE ]
 Supported link modes: 10000baseSR/Full
 Supported pause frame use: No
 Supports auto-negotiation: Yes
 Supported FEC modes: Not reported
 Advertised link modes: 10000baseSR/Full
 Advertised pause frame use: No
 Advertised auto-negotiation: Yes
 Advertised FEC modes: Not reported
 Speed: 10000Mb/s
 Duplex: Full
 Auto-negotiation: on
 Port: FIBRE
 PHYAD: 0
 Transceiver: internal
 Link detected: yes
(adjusting tags accordingly)

This bug was fixed in the package linux - 5.19.0-18.18

---------------
linux (5.19.0-18.18) kinetic; urgency=medium

  * kinetic/linux: 5.19.0-18.18 -proposed tracker (LP: #1990366)

  * 5.19.0-17.17: kernel NULL pointer dereference, address: 0000000000000084
    (LP: #1990236)
    - Revert "UBUNTU: SAUCE: apparmor: Fix regression in stacking due to label
      flags"
    - Revert "UBUNTU: [Config] disable SECURITY_APPARMOR_RESTRICT_USERNS"
    - Revert "UBUNTU: SAUCE: Revert "hwrng: virtio - add an internal buffer""
    - Revert "UBUNTU: SAUCE: Revert "hwrng: virtio - don't wait on cleanup""
    - Revert "UBUNTU: SAUCE: Revert "hwrng: virtio - don't waste entropy""
    - Revert "UBUNTU: SAUCE: Revert "hwrng: virtio - always add a pending
      request""
    - Revert "UBUNTU: SAUCE: Revert "hwrng: virtio - unregister device before
      reset""
    - Revert "UBUNTU: SAUCE: Revert "virtio-rng: make device ready before making
      request""
    - Revert "UBUNTU: [Config] update configs after apply new apparmor patch set"
    - Revert "UBUNTU: SAUCE: apparmor: add user namespace creation mediation"
    - Revert "UBUNTU: SAUCE: selinux: Implement userns_create hook"
    - Revert "UBUNTU: SAUCE: bpf-lsm: Make bpf_lsm_userns_create() sleepable"
    - Revert "UBUNTU: SAUCE: security, lsm: Introduce security_create_user_ns()"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: AppArmor: Remove the exclusive
      flag"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: LSM: Add /proc attr entry for full
      LSM context"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: LSM: Removed scaffolding function
      lsmcontext_init"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: netlabel: Use a struct lsmblob in
      audit data"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: Audit: Add record for multiple
      object contexts"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: audit: multiple subject lsm values
      for netlabel"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: Audit: Add record for multiple task
      security contexts"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: Audit: Allow multiple records in an
      audit_buffer"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: LSM: Add a function to report
      multiple LSMs"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: Audit: Create audit_stamp
      structure"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: Audit: Keep multiple LSM data in
      audit_names"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: LSM: security_secid_to_secctx
      module selection"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: binder: Pass LSM identifier for
      confirmation"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: NET: Store LSM netlabel data in a
      lsmblob"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: LSM: security_secid_to_secctx in
      netlink netfilter"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: LSM: Use lsmcontext in
      security_dentry_init_security"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: LSM: Use lsmcontext in
      security_inode_getsecctx"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: LSM: Use lsmcontext in
      security_secid_to_secctx"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: LSM:...

This bug was fixed in the package linux - 5.15.0-50.56

---------------
linux (5.15.0-50.56) jammy; urgency=medium

  * jammy/linux: 5.15.0-50.56 -proposed tracker (LP: #1990148)

  * CVE-2022-3176
    - io_uring: refactor poll update
    - io_uring: move common poll bits
    - io_uring: kill poll linking optimisation
    - io_uring: inline io_poll_complete
    - io_uring: correct fill events helpers types
    - io_uring: clean cqe filling functions
    - io_uring: poll rework
    - io_uring: remove poll entry from list when canceling all
    - io_uring: bump poll refs to full 31-bits
    - io_uring: fail links when poll fails
    - io_uring: fix wrong arm_poll error handling
    - io_uring: fix UAF due to missing POLLFREE handling

  * ip/nexthop: fix default address selection for connected nexthop
    (LP: #1988809)
    - selftests/net: test nexthop without gw

  * ip/nexthop: fix default address selection for connected nexthop
    (LP: #1988809) // icmp_redirect.sh in ubuntu_kernel_selftests failed on
    Jammy 5.15.0-49.55 (LP: #1990124)
    - ip: fix triggering of 'icmp redirect'

linux (5.15.0-49.55) jammy; urgency=medium

  * jammy/linux: 5.15.0-49.55 -proposed tracker (LP: #1989785)

  * amdgpu module crash after 5.15 kernel update (LP: #1981883)
    - drm/amdgpu: fix check in fbdev init

  * scsi: hisi_sas: Increase debugfs_dump_index after dump is  completed
    (LP: #1982070)
    - scsi: hisi_sas: Increase debugfs_dump_index after dump is completed

  * [UBUNTU 22.04] s390/qeth: cache link_info for ethtool (LP: #1984103)
    - s390/qeth: cache link_info for ethtool

  * WARN in trace_event_dyn_put_ref (LP: #1987232)
    - tracing/perf: Fix double put of trace event when init fails

  * Jammy update: v5.15.60 upstream stable release (LP: #1989221)
    - x86/speculation: Make all RETbleed mitigations 64-bit only
    - selftests/bpf: Extend verifier and bpf_sock tests for dst_port loads
    - selftests/bpf: Check dst_port only on the client socket
    - block: fix default IO priority handling again
    - tools/vm/slabinfo: Handle files in debugfs
    - ACPI: video: Force backlight native for some TongFang devices
    - ACPI: video: Shortening quirk list by identifying Clevo by board_name only
    - ACPI: APEI: Better fix to avoid spamming the console with old error logs
    - crypto: arm64/poly1305 - fix a read out-of-bound
    - KVM: x86: do not report a vCPU as preempted outside instruction boundaries
    - KVM: x86: do not set st->preempted when going back to user space
    - KVM: selftests: Make hyperv_clock selftest more stable
    - tools/kvm_stat: fix display of error when multiple processes are found
    - selftests: KVM: Handle compiler optimizations in ucall
    - KVM: x86/svm: add __GFP_ACCOUNT to __sev_dbg_{en,de}crypt_user()
    - arm64: set UXN on swapper page tables
    - btrfs: zoned: prevent allocation from previous data relocation BG
    - btrfs: zoned: fix critical section of relocation inode writeback
    - Bluetooth: hci_bcm: Add BCM4349B1 variant
    - Bluetooth: hci_bcm: Add DT compatible for CYW55572
    - dt-bindings: bluetooth: broadcom: Add BCM4349B1 DT binding
    - Bluetooth: btusb: Add support of IMC Netw...

This bug is awaiting verification that the linux-gkeop-5.15/5.15.0-1005.7~20.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug was not requested for linux-gkeop-5.15 nor for focal.
Hence I'll just set the tag 'verification-done-focal' to unblock the SRU process.

This bug is awaiting verification that the linux-bluefield/5.15.0-1010.12 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug was not requested for linux-bluefield.
Hence I'll just set the tag 'verification-done-jammy' to unblock the further process.

This bug is awaiting verification that the linux-nvidia/5.15.0-1011.11 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug was not opened against linux-nvidia/5.15.0-1011.11 and is also not relevant for this kernel.
However, I'm setting the tag to done to unblock the process.

------- Comment From <email address hidden> 2022-12-12 08:36 EDT-------
still fine with latest updates:

# uname -a
Linux a83lp78.lnxne.boe 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:57:26 UTC 2022 s390x s390x s390x GNU/Linux
# grep qcrdinfo /sys/kernel/debug/s390dbf/qeth*/hex_ascii | wc -l
0
# ethtool encbd00 | grep Duplex
Duplex: Full
# grep qcrdinfo /sys/kernel/debug/s390dbf/qeth*/hex_ascii | wc -l
0

This bug is awaiting verification that the linux-mtk/5.15.0-1030.34 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-mtk' to 'verification-done-jammy-linux-mtk'. If the problem still exists, change the tag 'verification-needed-jammy-linux-mtk' to 'verification-failed-jammy-linux-mtk'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

his bug only affected s390x, updating all other verification requests to done (to unblock potential processes).