Bionic ubuntu ethtool doesn't check ring parameters boundaries

Bug #1874444 reported by Arthur Kiyanovski
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
High
Guilherme G. Piccoli
Xenial
Fix Released
High
Guilherme G. Piccoli
Bionic
Fix Released
High
Guilherme G. Piccoli

Bug Description

[Impact]
* There's a bad behavior in the ena driver ringparam setting on kernels 4.4 and 4.15, if an invalid ringparam is provided to ethtool.

* Upstream Linux kernel implemented ring parameter boundaries check in commit: 37e2d99b59c4 ("ethtool: Ensure new ring parameters are within bounds during SRINGPARAM") [ git.kernel.org/linus/37e2d99b59c4 ].
Due to this commit, the community doesn't usually allow ring parameter boundary checks in driver code.

* Xenial/Bionic kernels don't include this patch, and some network drivers (like ena) rely on this patch for boundary checking of ring params. So, we are hereby requesting the commit inclusion in these kernel versions.

[Test case]
1. In AWS, create a new c5.4xlarge instance with the Ubuntu 18.04 official ami (uses the ENA network driver) and update to latest kernel/reboot.

2. Run ethtool -g ens5
output:
Ring parameters for ens5:
Pre-set maximums:
RX: 16384
RX Mini: 0
RX Jumbo: 0
TX: 1024
Current hardware settings:
RX: 1024
RX Mini: 0
RX Jumbo: 0
TX: 1024

3. Change the TX/RX ring size to a legal number within boundaries - works!

4. Change the TX/RX ring size to an illegal number (such as 2048 for TX) with the command - "sudo ethtool -G ens5 tx 2048".

Expected behavior - "Cannot set device ring parameters: Invalid argument"
Actual behavior - causes a driver hang since boundaries are not checked by ethtool, effectively hanging the instance (given that AWS has no console to allow system manipulation).

[Regression Potential]

Since that the commit is present in kernels v4.16+ (including Ubuntu) and is quite small and self-contained, the regression risk is very reduced.

One potential "regression" would be if some driver has bugs and provide bad values on get_ringparams, then the validation would be broken (allowing illegal values or refusing legal ones), but this wouldn't be a regression in the hereby proposed patch itself, it'd be only exposed by the patch.

CVE References

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1874444/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

tags: added: bot-comment
affects: ubuntu → ethtool (Ubuntu)
Revision history for this message
Ben Hutchings (benh-debian) wrote :

This can't be fixed in ethtool, it has to be fixed in the kernel.

affects: ethtool (Ubuntu) → linux (Ubuntu)
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1874444

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
tags: added: bionic
Revision history for this message
Arthur Kiyanovski (akiyano) wrote :

Tried running apport-collect but since my machine has no guy I couldn't authorize.
Tried running apport-cli --save file.out - this also failed with:
"No pending crash reports. Try --help for more information."

I have no idea how to continue here, and I don't see the point as it is fairly easy to run an ubuntu instance with the latest ubuntu 18.04 ami to reproduce the issue and I filled all the necessary information on how to fix the issue as well.

Moving the status to Confirmed.

Changed in linux (Ubuntu):
status: Incomplete → Opinion
status: Opinion → Confirmed
Revision history for this message
Arthur Kiyanovski (akiyano) wrote :

Hi,

Bumping this ticket up.
Is there still something missing in the ticket that I can provide to make it workable?

Thanks!

Revision history for this message
Guilherme G. Piccoli (gpiccoli) wrote :

Hi Arthur, thanks a lot for the report and for providing a test case and a fix suggestion - basically you did all the work heheh
I've just built kernels 4.4 (Xenial) and 4.15 (Bionic) with your suggested fix and indeed it prevented the failure; I'm working on the SRU request, so we can get that merged for the next kernel cycle [0].
Cheers,

Guilherme

[0] https://kernel.ubuntu.com

tags: added: sts xenial
Changed in linux (Ubuntu):
status: Confirmed → In Progress
Changed in linux (Ubuntu Xenial):
status: New → In Progress
Changed in linux (Ubuntu Bionic):
status: New → In Progress
Changed in linux (Ubuntu Xenial):
importance: Undecided → High
Changed in linux (Ubuntu Bionic):
importance: Undecided → High
Changed in linux (Ubuntu):
assignee: nobody → Guilherme G. Piccoli (gpiccoli)
Changed in linux (Ubuntu Xenial):
assignee: nobody → Guilherme G. Piccoli (gpiccoli)
Changed in linux (Ubuntu Bionic):
assignee: nobody → Guilherme G. Piccoli (gpiccoli)
Changed in linux (Ubuntu):
importance: Undecided → High
description: updated
Revision history for this message
Guilherme G. Piccoli (gpiccoli) wrote :

SRU submitted to kernel mailing-list: https://lists.ubuntu.com/archives/kernel-team/2020-May/109697.html

Cheers,

Guilherme

Changed in linux (Ubuntu Xenial):
status: In Progress → Fix Committed
Changed in linux (Ubuntu Bionic):
status: In Progress → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-bionic
tags: added: verification-needed-xenial
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Revision history for this message
Guilherme G. Piccoli (gpiccoli) wrote :

I was able to verify it on AWS using the test procedure on description, in the following linux-aws proposed kernels:
4.4.0-1108-aws (Xenial / 16.04)
4.15.0-1068-aws (Bionic / 18.04)

Also, I've done a code inspection in generic proposed kernels (4.4.0-180/Xenial and 4.15.0-102/Bionic) and the patch is there, hence marking it as verified for both kernels.
Cheers,

Guilherme

tags: added: verification-done-bionic verification-done-xenial
removed: verification-needed-bionic verification-needed-xenial
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (18.8 KiB)

This bug was fixed in the package linux - 4.15.0-106.107

---------------
linux (4.15.0-106.107) bionic; urgency=medium

  * CVE-2020-0543
    - SAUCE: x86/cpu: Add a steppings field to struct x86_cpu_id
    - SAUCE: x86/cpu: Add 'table' argument to cpu_matches()
    - SAUCE: x86/speculation: Add Special Register Buffer Data Sampling (SRBDS)
      mitigation
    - SAUCE: x86/speculation: Add SRBDS vulnerability and mitigation documentation
    - SAUCE: x86/speculation: Add Ivy Bridge to affected list

linux (4.15.0-103.104) bionic; urgency=medium

  * bionic/linux: 4.15.0-103.104 -proposed tracker (LP: #1881272)

  * "BUG: unable to handle kernel paging request" when testing
    ubuntu_kvm_smoke_test.kvm_smoke_test with B-KVM in proposed (LP: #1881072)
    - KVM: VMX: Explicitly reference RCX as the vmx_vcpu pointer in asm blobs
    - KVM: VMX: Mark RCX, RDX and RSI as clobbered in vmx_vcpu_run()'s asm blob

linux (4.15.0-102.103) bionic; urgency=medium

  * bionic/linux: 4.15.0-102.103 -proposed tracker (LP: #1878856)

  * Packaging resync (LP: #1786013)
    - update dkms package versions

  * debian/scripts/file-downloader does not handle positive failures correctly
    (LP: #1878897)
    - [Packaging] file-downloader not handling positive failures correctly

  * Kernel log flood "ceph: Failed to find inode for 1" (LP: #1875884)
    - ceph: don't check quota for snap inode
    - ceph: quota: cache inode pointer in ceph_snap_realm

  * [UBUNTU 18.04] zpcictl --reset - contribution for kernel (LP: #1870320)
    - s390/pci: Recover handle in clp_set_pci_fn()
    - s390/pci: Fix possible deadlock in recover_store()

  * Bionic update: upstream stable patchset 2020-05-12 (LP: #1878256)
    - drm/edid: Fix off-by-one in DispID DTD pixel clock
    - drm/qxl: qxl_release leak in qxl_draw_dirty_fb()
    - drm/qxl: qxl_release leak in qxl_hw_surface_alloc()
    - drm/qxl: qxl_release use after free
    - btrfs: fix block group leak when removing fails
    - btrfs: fix partial loss of prealloc extent past i_size after fsync
    - mmc: sdhci-xenon: fix annoying 1.8V regulator warning
    - mmc: sdhci-pci: Fix eMMC driver strength for BYT-based controllers
    - ALSA: hda/realtek - Two front mics on a Lenovo ThinkCenter
    - ALSA: hda/hdmi: fix without unlocked before return
    - ALSA: pcm: oss: Place the plugin buffer overflow checks correctly
    - PM: ACPI: Output correct message on target power state
    - PM: hibernate: Freeze kernel threads in software_resume()
    - dm verity fec: fix hash block number in verity_fec_decode
    - RDMA/mlx5: Set GRH fields in query QP on RoCE
    - RDMA/mlx4: Initialize ib_spec on the stack
    - vfio: avoid possible overflow in vfio_iommu_type1_pin_pages
    - vfio/type1: Fix VA->PA translation for PFNMAP VMAs in vaddr_get_pfn()
    - iommu/qcom: Fix local_base status check
    - scsi: target/iblock: fix WRITE SAME zeroing
    - iommu/amd: Fix legacy interrupt remapping for x2APIC-enabled system
    - ALSA: opti9xx: shut up gcc-10 range warning
    - nfs: Fix potential posix_acl refcnt leak in nfs3_set_acl
    - dmaengine: dmatest: Fix iteration non-stop logic
    - selinux: properly handle multiple messages in ...

Changed in linux (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (31.3 KiB)

This bug was fixed in the package linux - 4.4.0-184.214

---------------
linux (4.4.0-184.214) xenial; urgency=medium

  * CVE-2020-0543
    - SAUCE: x86/cpu: Add a steppings field to struct x86_cpu_id
    - SAUCE: x86/cpu: Add 'table' argument to cpu_matches()
    - SAUCE: x86/speculation: Add Special Register Buffer Data Sampling (SRBDS)
      mitigation
    - SAUCE: x86/speculation: Add SRBDS vulnerability and mitigation documentation
    - SAUCE: x86/speculation: Add Ivy Bridge to affected list

linux (4.4.0-181.211) xenial; urgency=medium

  * xenial/linux: 4.4.0-181.211 -proposed tracker (LP: #1881170)

  * CVE-2020-12769
    - spi: spi-dw: Add lock protect dw_spi rx/tx to prevent concurrent calls

  * I2C bus on Dell Edge Gateway stops working after upgrading to
    Ubuntu-4.4.0-180.210 (LP: #1881124)
    - SAUCE: Revert: Revert "ACPI / LPSS: allow to use specific PM domain during
      ->probe()"

linux (4.4.0-180.210) xenial; urgency=medium

  * xenial/linux: 4.4.0-180.210 -proposed tracker (LP: #1878873)

  * Xenial update: 4.4.223 upstream stable release (LP: #1878232)
    - mwifiex: fix PCIe register information for 8997 chipset
    - drm/qxl: qxl_release use after free
    - drm/qxl: qxl_release leak in qxl_draw_dirty_fb()
    - staging: rtl8192u: Fix crash due to pointers being "confusing"
    - usb: gadget: f_acm: Fix configfs attr name
    - usb: gadged: pch_udc: get rid of redundant assignments
    - usb: gadget: pch_udc: reorder spin_[un]lock to avoid deadlock
    - usb: gadget: udc: core: don't starve DMA resources
    - MIPS: Fix macro typo
    - MIPS: ptrace: Drop cp0_tcstatus from regoffset_table[]
    - MIPS: BMIPS: Fix PRID_IMP_BMIPS5000 masking for BMIPS5200
    - MIPS: smp-cps: Stop printing EJTAG exceptions to UART
    - MIPS: scall: Handle seccomp filters which redirect syscalls
    - MIPS: BMIPS: BMIPS5000 has I cache filing from D cache
    - MIPS: BMIPS: Clear MIPS_CACHE_ALIASES earlier
    - MIPS: BMIPS: local_r4k___flush_cache_all needs to blast S-cache
    - MIPS: BMIPS: Pretty print BMIPS5200 processor name
    - MIPS: Fix HTW config on XPA kernel without LPA enabled
    - MIPS: BMIPS: Adjust mips-hpt-frequency for BCM7435
    - MIPS: math-emu: Fix BC1{EQ,NE}Z emulation
    - MIPS: Fix BC1{EQ,NE}Z return offset calculation
    - MIPS: perf: Fix I6400 event numbers
    - MIPS: KVM: Fix translation of MFC0 ErrCtl
    - MIPS: SMP: Update cpu_foreign_map on CPU disable
    - MIPS: c-r4k: Fix protected_writeback_scache_line for EVA
    - MIPS: Octeon: Off by one in octeon_irq_gpio_map()
    - bpf, mips: fix off-by-one in ctx offset allocation
    - MIPS: RM7000: Double locking bug in rm7k_tc_disable()
    - MIPS: Define AT_VECTOR_SIZE_ARCH for ARCH_DLINFO
    - mips/panic: replace smp_send_stop() with kdump friendly version in panic
      path
    - ARM: dts: armadillo800eva Correct extal1 frequency to 24 MHz
    - ARM: imx: select SRC for i.MX7
    - ARM: dts: kirkwood: gpio pin fixes for linkstation ls-wxl/wsxl
    - ARM: dts: kirkwood: gpio pin fixes for linkstation ls-wvl/vl
    - ARM: dts: kirkwood: gpio-leds fixes for linkstation ls-wxl/wsxl
    - ARM: dts: kirkwood: gpio-leds fixes for linkstation ls-wvl/v...

Changed in linux (Ubuntu Xenial):
status: Fix Committed → Fix Released
Changed in linux (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Arthur Kiyanovski (akiyano) wrote :

Question - when is this fix expected to be released to the official Bionic and Xenial AMIs in AWS?
Thank!

Revision history for this message
Guilherme G. Piccoli (gpiccoli) wrote :

Hi Arthur, I discussed this topic today with the responsible for AMIs on AWS, and I was then told that new AMIs should be built for testing next hours. After internal testing, they should get released, I don't have an ETA.

Cheers,

Guilherme

Revision history for this message
Arthur Kiyanovski (akiyano) wrote :

Hi Guilherme,

Thanks for your reply.
No need for an exact ETA, but can you give me some sort of feeling, maybe how often do these amis get released? once a month? 2 months?

Cheers,
Arthur

Revision history for this message
Guilherme G. Piccoli (gpiccoli) wrote :

Hi Arthur, good news - there are currently Bionic and Xenial AMIs with the fix! I've tried the following AMIs, and they have the fix:

ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20200610 - ami-025201fa53cf4d031
ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20200610 - ami-0a0ddd875a1ea2c7f

Note that the first one (Ubuntu 18.04) includes kernel 5.3, which contains the fix.
Let me know if you have more questions, and thanks again for the report!
Cheers,

Guilherme

Revision history for this message
Arthur Kiyanovski (akiyano) wrote :

Hi Guilherme,

Thanks for the update! I tested and it works!

Cheers,
Arthur

Revision history for this message
Guilherme G. Piccoli (gpiccoli) wrote :

Great news Arthur, I'm glad the issue is solved!
Thanks for the report and testing,

Guilherme

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.