Intel E810-XXV - NETDEV WATCHDOG: (ice): transmit queue timed out

This is the log from the HWE kernel:

[33219.508873] ------------[ cut here ]------------
[33219.508877] NETDEV WATCHDOG: enp161s0f1 (ice): transmit queue 35 timed out
[33219.508932] WARNING: CPU: 48 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x21f/0x230
[33219.508940] Modules linked in: sch_ingress nf_conntrack_netlink geneve ip6_udp_tunnel udp_tunnel xt_CT dm_crypt scsi_transport_iscsi veth nfnetlink_cttimeout openvswitch nsh nf_conncount unix_diag nft_masq zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_vsock vmw_vsock_virtio_transport_common vhost vhost_iotlb vsock xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink bridge sunrpc nvme_fabrics 8021q garp mrp stp llc bonding tls binfmt_misc ipmi_ssif intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd dell_wmi kvm_amd video ledtrig_audio nls_iso8859_1 irdma sparse_keymap kvm i40e irqbypass dell_smbios dcdbas ib_uverbs rapl dell_wmi_descriptor wmi_bmof ib_core ccp ptdma k10temp acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter mac_hid sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops
[33219.509051] reed_solomon pstore_blk pstore_zone efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear cdc_ether usbnet mii mgag200 i2c_algo_bit drm_shmem_helper drm_kms_helper syscopyarea crct10dif_pclmul sysfillrect sysimgblt crc32_pclmul bcache polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 nvme aesni_intel crypto_simd nvme_core ahci xhci_pci cryptd ice tg3 libahci drm megaraid_sas i2c_piix4 xhci_pci_renesas nvme_common wmi
[33219.509114] CPU: 48 PID: 0 Comm: swapper/48 Tainted: P O 6.2.0-32-generic #32~22.04.1-Ubuntu
[33219.509116] Hardware name: Dell Inc. PowerEdge R7525/03WYW4, BIOS 2.12.4 07/26/2023
[33219.509118] RIP: 0010:dev_watchdog+0x21f/0x230
[33219.509122] Code: 00 e9 31 ff ff ff 4c 89 e7 c6 05 66 83 78 01 01 e8 56 00 f8 ff 44 89 f1 4c 89 e6 48 c7 c7 08 4f e4 b7 48 89 c2 e8 61 df 2b ff <0f> 0b e9 22 ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90
[33219.509123] RSP: 0018:ffffb42719fd0e70 EFLAGS: 00010246
[33219.509125] RAX: 0000000000000000 RBX: ffff9bd91b3e74c8 RCX: 0000000000000000
[33219.509126] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[33219.509127] RBP: ffffb42719fd0e98 R08: 0000000000000000 R09: 0000000000000000
[33219.509128] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9bd91b3e7000
[33219.509129] R13: ffff9bd91b3e741c R14: 0000000000000023 R15: 0000000000000000
[33219.509130] FS: 0000000000000000(0000) GS:ffff9b573de00000(0000) knlGS:0000000000000000
[33219.509132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[33219.509133] CR2: 000055fd64034000 CR3: 0000010273ae2004 CR4: 0000000000770ee0
[33219.509135] PKRU: 55555554
[33219.509135] Call Trace:
[33219.509137] <IRQ>
[33219.509140] ? show_regs+0x72/0x90
[33219.509145] ? dev_watchdog+0x21f/0x230
[33219.509147] ? __warn+0x8...

This change was made by a bot.

I added logs from a machine that I'm not sure was affected (infra01), adding more logs below for the one that is certainly affected (cloud002).

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

I have not tested without the bond, but I believe this issue probably is not directly related to the fact that the interface is bonded, which would mean removing the bond will not help. While I will try to test this if possible (depends on customer doing reconfiguration of switch side), I appreciate any suggestion or workaround that could unblock the deployment.

https://<email address hidden>/msg12747.html

similar issue

Got a suggestion to try disabling TSO which helped in similar cases (same queue timeout error) in e1000e driver. Will report back soon.

Disabling TSO on both legs of the bond in all hosts did not help. After 2h30min working well, it happened again.

Removing lacp bonding (using just one interface without any kind of bonding) seemed to help, I'm not seeing the issue anymore. Still testing.

I ran into this issue on 22.04 LTS (using HWE kernel 6.2) on a 100G dual-port E810 NIC.
Also with LACP only, active-backup works without issues.

To bring this more to the attention of the driver devs, I posted to the intel-wired-lan ML: https://lists.osuosl.org/pipermail/intel-wired-lan/Week-of-Mon-20231120/038096.html

Hi Christian

In my tests, I also saw the same issues with active-backup too.

Do you know a way to reproduce this issue? I'm having a hard time to find a consistent reproducer, currently I need to deploy a complete openstack, run a ser of load tests on it and eventually the problem shows up, but it takes many hours and not on all hosts.

It would be much easier to have just one machine and trigger the issue in some other way.

Also, this is fixed upstream, some changes between 1.8.x and 1.9.x of upstream source drivers fixed the problem (they are at 1.12.x now, so it has been fixed for quite a while now). The problem is that whatever the fix is, is has not been imported to kernels 5.15 (jammy GA), 6.2 (jammy HWE), 6.5 (cosmic GA). I could not reliably test upstream mainline 6.6 because there is no ubuntu currently shipping this package and the pure upstream kernel breaks a lot of stuff in ubuntu.

I mention this because of your post in the intel mailing list. They will probably not be able to help much.

Let me know if you find a consistent reproducer.

1) Andre, after I switched to active-backup the issue is gone (so far). But yeah, we are looking for a reproducer as well. It's hard to narrow down some random issue - also likely for Intel.

2) But I just received an email from an Intel developer with a suggested change to the driver to narrow down the issue further. I quote ...

--- cut ---

Could you edit file (from kernel source tree base) drivers/net/ethernet/intel/ice/ice_lag.c .
Then find the functions ice_init_lag()and ice_deinit_lag().

Then add this line to the beginning of the functions

return 0; and return; respectively.

the patch nomenclature would look something like this:

* Memory will be freed in ice_deinit_lag
*/
int ice_init_lag(struct ice_pf *pf)
{
        struct device *dev = ice_pf_to_dev(pf);
        struct ice_lag *lag;
        struct ice_vsi *vsi;
        int err;

+ return 0;
        pf->lag = kzalloc(sizeof(*lag), GFP_KERNEL);
        if (!pf->lag)
                return -ENOMEM;
        lag = pf->lag;

………

* This function is meant to only be called on driver remove/shutdown
*/
void ice_deinit_lag(struct ice_pf *pf)
{
        struct ice_lag *lag;

+ return;
        lag = pf->lag;

Then re-build the driver and try to reproduce the problem?

--- cut ---

So in essence I believe this just skips offloading the bonding / LACP to the HW.
I will set this up on one or two of our machines to test. Would you please also try this on your systems?

I have tried this (patches suggested in comment #40) and the problem seems to have gone away. It may be too soon to say but my test scenario (which never gave me a false negative before) finished without issues.

Of course this is not a 'fix', so I'm curious to see what the OP has to say about this result.

What is the cards firmware ?

$ ethtool -i <iface> |grep firmware-version

@Bartosz

$ ethtool -i enp65s0f0 |grep firmware-version
firmware-version: 4.20 0x8001784b 22.0.9

This is the latest firmware supported by Dell. You will find 4.3 available on Intel website, but it is not available yet through dell firmware tools.

@Andre,

I successfully installed on the machines with NVMUpdate64 tool. That is on HPE machines.

$ sudo ./nvmupdate64e

Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.39.56.8
Copyright(C) 2013 - 2023 Intel Corporation.

WARNING: To avoid damage to your device, do not stop the update or reboot or power off the system during this update.
Inventory in progress. Please wait [**+.......]

Num Description Ver.(hex) DevId S:B Status
=== ================================== ============ ===== ====== ==============
01) Intel(R) Ethernet Network Adapter 4.48(4.30) 159B 00:016 Up to date
    E810-XXV-2
02) Intel(R) Ethernet Network Adapter N/A(N/A) 1521 00:072 Update not
    I350-T4 for OCP NIC 3.0 available

Tool execution completed with the following status: An error occurred accessing the device.
Press any key to exit.

Yeah, I knew about that 4.30 update in Intel website, but it is not present on Dell tools yet and the customer did not want to void their warranty (potentially), so I did not try it. That is something to keep in mind while we debug it.

FWIW, we updated our NICs to 4.30 as they were individually purchased and not part of pre-built servers and also have this issue.

So in essence the issue also exists with the latest firmware.

Script to verify AQ 0x000A capabilities

@Christian,
Can you verify your device capabilities returned from 0x000A looking for SRIOV lag?
I have attached a script "parse_aq_0xA.py" you need to load driver with dyndbg=+p and replace a buffer in script.
Note: buffer has to come from CQ CMD: opcode 0x000A
Expected result:
(...)
resp cap: 0x92 -- this capability we are looking for
resp maj_ver: 0x1
resp min_ver: 0x0
resp number: 0x1 -- This is value we want to check.
resp logical_id: 0x0
resp phys_id: 0x0

If it is set to 0x1 patch [1] will disable lag handler and simplify initialization logic to something like in comment #40
Buffer available in the script comes from CVL4.20 NVM

[1] - https://lists.osuosl.org/pipermail/intel-wired-lan/Week-of-Mon-20231211/038588.html

@Robert,
first thanks a lot for pursuing this issue!

1) I certainly can provide the debugging info. May I ask if ...

  a) the system in question would need to have an active LAG (LACP) for this to be helpful? We did switch to active-backup on all our machines due to this very issue.

  b) this requires FW version 4.20? All our machines currently run 4.30 already.

  c) this requires a certain kernel / ice driver version?

2) There now is FW 4.40 out [1]. But there seem to be no fixes related to LAG / LACP, some regarding SRIO-V though. But I guess you are convinced the issue is not within the FW, but rather the ice driver?

[1] - https://www.intel.com/content/www/us/en/download/19624/non-volatile-memory-nvm-update-utility-for-intel-ethernet-network-adapter-e810-series.html

Hey @Christian,
1a) No need, AQ 0x000A returns NVM capabilities regardless of configuration applied (it's done during driver init)
1b) That's the point, I noticed you upgraded to 4.3 which I currently don't have access to and I wanted to verify capabilities on 4.3. NVM caps should be similar on the same NVM version in single head of family so values I had access to would be the same you had on 4.2 (meaning there is no point of collecting these)
1c) No, any "recent" kernel/driver version will support enabling debug logs by adding dyndbg=+p param to module. We only care for logs which are retrieved from NVM and printed with debug flags.

2) The issue based on recent patches from Intel is caused by performing LAG related operations without proper support from NVM. Release notes does not always tell every feature change so there is a possibility that 4.4 introduced sriov_lag capability but I cannot verify it.
Worst case scenario is that NVM 4.4 will introduce sriov_lag capability, meaning patches added recently to upstream kernel will have no effect, and also issue will still reproduce. In this scenario currently there will be no 'workaround' for it.
Best case scenario is that NVM 4.4 will introduce sriov_lag capability and issue will no longer reproduce.
In this scenario no additional patches to the driver will be required.

@Stefan Could you kindly elaborate on the "Fix Commmited"? Was there any change to the kernel that would fix this issue? Is this fixed with 4.40 NVM from Intel?

Reading Roberts post (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2036239/comments/50) again, it seems that he is only guessing that there was something fixed by Intel.

Hey Christian, Intel proposed change [1]
which is targeting this problem and based on our testing in fact it solves the problem.
This change is currently added to Ubuntu Kernels.

I'm also keeping an eye on [2] but right now I don't yet see "business need" to incorporate it to Ubuntu Kernel.
This patch furthers limit problematic part of the code by adding (in addition to NVM caps check) verification based on DDP package.

1 - https://lists.osuosl.org/pipermail/intel-wired-lan/Week-of-Mon-20231211/038588.html
2 - https://lists.osuosl.org/pipermail/intel-wired-lan/Week-of-Mon-20240122/039100.html

Fix already included in 6.8

Switching status for Noble to In Progress.
Target release for Noble is 6.8 (which includes fix) but it's not out yet, status will be changed once 6.8 will be introduced.

@Robert thanks for keeping this bug alive and updated!

1) More debug info required?

@Robert, reading your post https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2036239/comments/50 again,
I am wondering if you asked me to provided more debug info with NVM 4.4 on my E810 NICs? Would this help in any way?

2) @smb changed this bug to "fix commited" for Jammy - is this really the correct state?
As @Andre said in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2036239/comments/41, just manually commenting out some lines in the ice kernel module is "not a fix".

3) Will the two "fixes" you referred to in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2036239/comments/52 make it to any other kernel than 6.8? Either by Intel or by Ubuntu applying them there? Otherwise I am wondering if and when 6.8 will be, once out, made available as HWE for Jammy?

@christian-rhomann "Fix committed" here means that the patches have been merged into Ubuntu's kernel tree for that specific release. The patch Robert submitted is the one from upstream, not the test patch from the comments here.
E.g. for Jammy:
https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/commit/?h=master-next&id=fc26d7737e3a

Thx a log Heitor! With no mention of some new package fixing this I did not correlate that to any patch to the kernel.
Will the be fixed in the HWE kernel as well then?

Yes, HWE kernels based on the Jammy/Mantic/Noble kernels should get this fix automatically when the GA versions get released.

This bug is awaiting verification that the linux/5.15.0-100.110 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux' to 'verification-done-jammy-linux'. If the problem still exists, change the tag 'verification-needed-jammy-linux' to 'verification-failed-jammy-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Houston, we have a problem...

This bug is notoriously difficult to reproduce. The only environment that presented it is now in production and will not be available for testing anymore. Which means that this cannot be tested, unless anyone can suggest a new way of reproducing it.

This bug is awaiting verification that the linux/6.5.0-25.25 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-mantic-linux' to 'verification-done-mantic-linux'. If the problem still exists, change the tag 'verification-needed-mantic-linux' to 'verification-failed-mantic-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Jammy update:
Due to lack of reproduction environment I have been performing following regression test:
1. Setup:
   nic: 2port E810-XXV
        both interfaces set up in bonding
   kernel: 5.15.0-100-generic
2. Test cases:
   0) verified that code from the change is used during driver init
   a) stress traffic for 48h (multiple streams of iperf (tcp))
   b) if up/down during stress traffic
   c) pf reset during stress traffic
Look for any issues related to traffic processing, look for tx_hangs
3. Result: No issues have been detected during test execution

Mantic tests in progress.

Hi Robert! Thanks for testing this on jammy. I marked the tag as verified ('verification-done-jammy-linux') to reflect that.
Could you share the results from mantic? we need to release this next week and we need a confirmation this works as expected.
If your test looks fine, please remove 'verification-needed-mantic-linux' tag and add 'verification-done-mantic-linux'.
Thanks!

Hi Roxana,
Mantic verification is still not finished.
I did some touch tests without stress traffic.
I'm trying to get my hands on E810 device to finish testing, I'll update ticket once it's done.
Wishful ETA EOW 09.

This bug is awaiting verification that the linux-ibm-gt-fips/5.15.0-1055.58+fips1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-ibm-gt-fips' to 'verification-done-jammy-linux-ibm-gt-fips'. If the problem still exists, change the tag 'verification-needed-jammy-linux-ibm-gt-fips' to 'verification-failed-jammy-linux-ibm-gt-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

LP update:

Mantic update:
Due to lack of reproduction environment I have been performing following regression test:
1. Setup:
   nic: 2port E810-C
        both interfaces set up in bonding
   kernel: 6.5.0-25-generic
2. Test cases:
   0) verified that code from the change is used during driver init
   a) stress traffic for 12h (multiple streams of iperf (tcp))
   b) if up/down during stress traffic
   c) reload driver during stress traffic
Look for any issues related to traffic processing, look for tx_hangs
3. Result: No issues have been detected during test execution

This bug was fixed in the package linux - 6.5.0-25.25

---------------
linux (6.5.0-25.25) mantic; urgency=medium

  * mantic/linux: 6.5.0-25.25 -proposed tracker (LP: #2052615)

  * Packaging resync (LP: #1786013)
    - debian/dkms-versions -- update from kernel-versions (main/2024.02.05)

  * [SRU][22.04.04]: mpi3mr driver update (LP: #2045233)
    - scsi: mpi3mr: Invoke soft reset upon TSU or event ack time out
    - scsi: mpi3mr: Update MPI Headers to version 3.00.28
    - scsi: mpi3mr: Add support for more than 1MB I/O
    - scsi: mpi3mr: WRITE SAME implementation
    - scsi: mpi3mr: Enhance handling of devices removed after controller reset
    - scsi: mpi3mr: Update driver version to 8.5.0.0.0
    - scsi: mpi3mr: Split off bus_reset function from host_reset
    - scsi: mpi3mr: Add support for SAS5116 PCI IDs
    - scsi: mpi3mr: Add PCI checks where SAS5116 diverges from SAS4116
    - scsi: mpi3mr: Increase maximum number of PHYs to 64 from 32
    - scsi: mpi3mr: Add support for status reply descriptor
    - scsi: mpi3mr: driver version upgrade to 8.5.0.0.50
    - scsi: mpi3mr: Refresh sdev queue depth after controller reset
    - scsi: mpi3mr: Clean up block devices post controller reset
    - scsi: mpi3mr: Block PEL Enable Command on Controller Reset and Unrecoverable
      State
    - scsi: mpi3mr: Fetch correct device dev handle for status reply descriptor
    - scsi: mpi3mr: Support for preallocation of SGL BSG data buffers part-1
    - scsi: mpi3mr: Support for preallocation of SGL BSG data buffers part-2
    - scsi: mpi3mr: Support for preallocation of SGL BSG data buffers part-3
    - scsi: mpi3mr: Update driver version to 8.5.1.0.0

  * The display becomes frozen after some time when a HDMI device is connected.
    (LP: #2049027)
    - drm/i915/dmc: Don't enable any pipe DMC events

  * Audio balancing setting doesn't work with the cirrus codec (LP: #2051050)
    - ALSA: hda/cs8409: Suppress vmaster control for Dolphin models

  * partproke is broken on empty loopback device (LP: #2049689)
    - block: Move checking GENHD_FL_NO_PART to bdev_add_partition()

  * CVE-2023-51780
    - atm: Fix Use-After-Free in do_vcc_ioctl

  * CVE-2023-6915
    - ida: Fix crash in ida_free when the bitmap is empty

  * Update Ubuntu.md (LP: #2051176)
    - [Packaging] update Ubuntu.md

  * test_021_aslr_dapper_libs from ubuntu_qrt_kernel_security failed on K-5.19 /
    J-OEM-6.1 / J-6.2 AMD64 (LP: #1983357)
    - [Config]: set ARCH_MMAP_RND_{COMPAT_, }BITS to the maximum

  * Intel E810-XXV - NETDEV WATCHDOG: (ice): transmit queue timed out
    (LP: #2036239)
    - ice: Add driver support for firmware changes for LAG
    - ice: alter feature support check for SRIOV and LAG

  * Mantic update: upstream stable patchset 2024-01-29 (LP: #2051584)
    - Upstream stable to v6.1.67, v6.6.6
    - vdpa/mlx5: preserve CVQ vringh index
    - hrtimers: Push pending hrtimers away from outgoing CPU earlier
    - i2c: designware: Fix corrupted memory seen in the ISR
    - netfilter: ipset: fix race condition between swap/destroy and kernel side
      add/del/test
    - zstd: Fix array-index-out-of-bounds UBSAN warning
    - tg3: Move the [rt]x_dropped counters...

This bug is awaiting verification that the linux-aws/6.5.0-1015.15 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-mantic-linux-aws' to 'verification-done-mantic-linux-aws'. If the problem still exists, change the tag 'verification-needed-mantic-linux-aws' to 'verification-failed-mantic-linux-aws'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-azure/6.5.0-1016.16 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-mantic-linux-azure' to 'verification-done-mantic-linux-azure'. If the problem still exists, change the tag 'verification-needed-mantic-linux-azure' to 'verification-failed-mantic-linux-azure'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-hwe-6.5/6.5.0-25.25~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-hwe-6.5' to 'verification-done-jammy-linux-hwe-6.5'. If the problem still exists, change the tag 'verification-needed-jammy-linux-hwe-6.5' to 'verification-failed-jammy-linux-hwe-6.5'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug was fixed in the package linux - 5.15.0-100.110

---------------
linux (5.15.0-100.110) jammy; urgency=medium

  * jammy/linux: 5.15.0-100.110 -proposed tracker (LP: #2052616)

  * i915 regression introduced with 5.5 kernel (LP: #2044131)
    - drm/i915: Skip some timing checks on BXT/GLK DSI transcoders

  * Audio balancing setting doesn't work with the cirrus codec (LP: #2051050)
    - ALSA: hda/cs8409: Suppress vmaster control for Dolphin models

  * partproke is broken on empty loopback device (LP: #2049689)
    - block: Move checking GENHD_FL_NO_PART to bdev_add_partition()

  * CVE-2023-0340
    - vhost: use kzalloc() instead of kmalloc() followed by memset()

  * CVE-2023-51780
    - atm: Fix Use-After-Free in do_vcc_ioctl

  * CVE-2023-6915
    - ida: Fix crash in ida_free when the bitmap is empty

  * CVE-2024-0646
    - net: tls, update curr on splice as well

  * CVE-2024-0565
    - smb: client: fix OOB in receive_encrypted_standard()

  * CVE-2023-51781
    - appletalk: Fix Use-After-Free in atalk_ioctl

  * Jammy update: v5.15.143 upstream stable release (LP: #2050858)
    - vdpa/mlx5: preserve CVQ vringh index
    - hrtimers: Push pending hrtimers away from outgoing CPU earlier
    - i2c: designware: Fix corrupted memory seen in the ISR
    - netfilter: ipset: fix race condition between swap/destroy and kernel side
      add/del/test
    - tg3: Move the [rt]x_dropped counters to tg3_napi
    - tg3: Increment tx_dropped in tg3_tso_bug()
    - kconfig: fix memory leak from range properties
    - drm/amdgpu: correct chunk_ptr to a pointer to chunk.
    - platform/x86: asus-wmi: Adjust tablet/lidflip handling to use enum
    - platform/x86: asus-wmi: Add support for ROG X13 tablet mode
    - platform/x86: asus-wmi: Simplify tablet-mode-switch probing
    - platform/x86: asus-wmi: Simplify tablet-mode-switch handling
    - platform/x86: asus-wmi: Move i8042 filter install to shared asus-wmi code
    - of: dynamic: Fix of_reconfig_get_state_change() return value documentation
    - platform/x86: wmi: Allow duplicate GUIDs for drivers that use struct
      wmi_driver
    - platform/x86: wmi: Skip blocks with zero instances
    - ipv6: fix potential NULL deref in fib6_add()
    - octeontx2-pf: Add missing mutex lock in otx2_get_pauseparam
    - octeontx2-af: Check return value of nix_get_nixlf before using nixlf
    - hv_netvsc: rndis_filter needs to select NLS
    - r8152: Rename RTL8152_UNPLUG to RTL8152_INACCESSIBLE
    - r8152: Add RTL8152_INACCESSIBLE checks to more loops
    - r8152: Add RTL8152_INACCESSIBLE to r8156b_wait_loading_flash()
    - r8152: Add RTL8152_INACCESSIBLE to r8153_pre_firmware_1()
    - r8152: Add RTL8152_INACCESSIBLE to r8153_aldps_en()
    - mlxbf-bootctl: correctly identify secure boot with development keys
    - platform/mellanox: Add null pointer checks for devm_kasprintf()
    - platform/mellanox: Check devm_hwmon_device_register_with_groups() return
      value
    - arcnet: restoring support for multiple Sohard Arcnet cards
    - net: stmmac: fix FPE events losing
    - octeontx2-af: fix a use-after-free in rvu_npa_register_reporters
    - i40e: Fix unexpected MFS warning message
    - net: bn...

This bug is awaiting verification that the linux-intel-iotg/5.15.0-1050.56 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-intel-iotg' to 'verification-done-jammy-linux-intel-iotg'. If the problem still exists, change the tag 'verification-needed-jammy-linux-intel-iotg' to 'verification-failed-jammy-linux-intel-iotg'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-kvm/5.15.0-1052.57 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-kvm' to 'verification-done-jammy-linux-kvm'. If the problem still exists, change the tag 'verification-needed-jammy-linux-kvm' to 'verification-failed-jammy-linux-kvm'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-oracle/5.15.0-1053.59 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-oracle' to 'verification-done-jammy-linux-oracle'. If the problem still exists, change the tag 'verification-needed-jammy-linux-oracle' to 'verification-failed-jammy-linux-oracle'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-raspi/5.15.0-1048.51 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-raspi' to 'verification-done-jammy-linux-raspi'. If the problem still exists, change the tag 'verification-needed-jammy-linux-raspi' to 'verification-failed-jammy-linux-raspi'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-aws/5.15.0-1056.61 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-aws' to 'verification-done-jammy-linux-aws'. If the problem still exists, change the tag 'verification-needed-jammy-linux-aws' to 'verification-failed-jammy-linux-aws'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-aws-fips/5.15.0-1056.61+fips1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-aws-fips' to 'verification-done-jammy-linux-aws-fips'. If the problem still exists, change the tag 'verification-needed-jammy-linux-aws-fips' to 'verification-failed-jammy-linux-aws-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-gcp-fips/5.15.0-1055.63+fips2 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-gcp-fips' to 'verification-done-jammy-linux-gcp-fips'. If the problem still exists, change the tag 'verification-needed-jammy-linux-gcp-fips' to 'verification-failed-jammy-linux-gcp-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-nvidia-6.5/6.5.0-1014.14 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-nvidia-6.5' to 'verification-done-jammy-linux-nvidia-6.5'. If the problem still exists, change the tag 'verification-needed-jammy-linux-nvidia-6.5' to 'verification-failed-jammy-linux-nvidia-6.5'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-nvidia-tegra/5.15.0-1023.23 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-nvidia-tegra' to 'verification-done-jammy-linux-nvidia-tegra'. If the problem still exists, change the tag 'verification-needed-jammy-linux-nvidia-tegra' to 'verification-failed-jammy-linux-nvidia-tegra'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-nvidia-tegra-5.15/5.15.0-1023.23~20.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux-nvidia-tegra-5.15' to 'verification-done-focal-linux-nvidia-tegra-5.15'. If the problem still exists, change the tag 'verification-needed-focal-linux-nvidia-tegra-5.15' to 'verification-failed-focal-linux-nvidia-tegra-5.15'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-nvidia-tegra-igx/5.15.0-1010.10 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-nvidia-tegra-igx' to 'verification-done-jammy-linux-nvidia-tegra-igx'. If the problem still exists, change the tag 'verification-needed-jammy-linux-nvidia-tegra-igx' to 'verification-failed-jammy-linux-nvidia-tegra-igx'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-xilinx-zynqmp/5.15.0-1029.33 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-xilinx-zynqmp' to 'verification-done-jammy-linux-xilinx-zynqmp'. If the problem still exists, change the tag 'verification-needed-jammy-linux-xilinx-zynqmp' to 'verification-failed-jammy-linux-xilinx-zynqmp'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!