Possible io_uring regression with QEMU on Ubuntu's kernel

Bug #1935017 reported by Juhyung Park
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Invalid
Undecided
Unassigned
Groovy
Fix Released
High
Kamal Mostafa

Bug Description

Hi everyone.
Possible io_uring regression with QEMU on Ubuntu's kernel

With the latest Ubuntu 20.04's HWE kernel 5.8.0-59, I'm noticing some weirdness when using QEMU/libvirt with the following storage configuration:

<disk type="block" device="disk">
  <driver name="qemu" type="raw" cache="none" io="io_uring" discard="unmap" detect_zeroes="unmap"/>
  <source dev="/dev/disk/by-id/md-uuid-df271a1e:9dfb7edb:8dc4fbb8:c43e652f-part1" index="1"/>
  <backingStore/>
  <target dev="vda" bus="virtio"/>
  <alias name="virtio-disk0"/>
  <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
</disk>

QEMU version is 5.2+dfsg-9ubuntu3 and libvirt version is 7.0.0-2ubuntu2.

The guest VM is unable to handle I/O properly with io_uring, and nuking io="io_uring" fixes the issue.
On one machine (EPYC 7742), the partition table cannot be read and on another (Ryzen 9 3950X), ext4 detects weirdness with journaling and ultimately remounts the guest disk to R/O:

[ 2.712321] virtio_blk virtio5: [vda] 3906519775 512-byte logical blocks (2.00 TB/1.82 TiB)
[ 2.714054] vda: detected capacity change from 0 to 2000138124800
[ 2.963671] blk_update_request: I/O error, dev vda, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 2.964909] Buffer I/O error on dev vda, logical block 0, async page read
[ 2.966021] blk_update_request: I/O error, dev vda, sector 1 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 2.967177] Buffer I/O error on dev vda, logical block 1, async page read
[ 2.968330] blk_update_request: I/O error, dev vda, sector 2 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 2.969504] Buffer I/O error on dev vda, logical block 2, async page read
[ 2.970767] blk_update_request: I/O error, dev vda, sector 3 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 2.971624] Buffer I/O error on dev vda, logical block 3, async page read
[ 2.972170] blk_update_request: I/O error, dev vda, sector 4 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 2.972728] Buffer I/O error on dev vda, logical block 4, async page read
[ 2.973308] blk_update_request: I/O error, dev vda, sector 5 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 2.973920] Buffer I/O error on dev vda, logical block 5, async page read
[ 2.974496] blk_update_request: I/O error, dev vda, sector 6 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 2.975093] Buffer I/O error on dev vda, logical block 6, async page read
[ 2.975685] blk_update_request: I/O error, dev vda, sector 7 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 2.976295] Buffer I/O error on dev vda, logical block 7, async page read
[ 2.980074] blk_update_request: I/O error, dev vda, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 2.981104] Buffer I/O error on dev vda, logical block 0, async page read
[ 2.981786] blk_update_request: I/O error, dev vda, sector 1 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 2.982083] ixgbe 0000:06:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63 XDP Queue count = 0
[ 2.982442] Buffer I/O error on dev vda, logical block 1, async page read
[ 2.983642] ldm_validate_partition_table(): Disk read failed.

Kernel 5.8.0-55 is fine, and the only io_uring-related change between 5.8.0-55 and 5.8.0-59 is the commit 4b982bd0f383 ("io_uring: don't mark S_ISBLK async work as unbounded").

I'm hesitant to run lspci -vnvn and post other bug reporting logs as the machine includes some proprietary hardware (quite unrelated to this specific issue), but this issue is reproducible on multiple machines.
The another machine that I was able to reproduce this is running a production software that needs to run 24/7, so I'm also hesitant to gather logs there too.

Thanks,
Regards

Update:

It was the commit 87c9cfe0fa1fb ("block: don't ignore REQ_NOWAIT for direct IO").
(Upstream commit f8b78caf21d5bc3fcfc40c18898f9d52ed1451a5)

I've double checked by resetting the Git to Ubuntu-hwe-5.8-5.8.0-59.66_20.04.1 and reverting that patch alone.
It fixes the issue.

It seems like this patch was backported to multiple stable trees, so I'm not exactly sure why only Canonical's 5.8 is affected.
FWIW, 5.8.0-61 is also affected.

CVE References

Changed in linux (Ubuntu):
status: New → In Progress
importance: Undecided → High
assignee: nobody → Kamal Mostafa (kamalmostafa)
Revision history for this message
Kamal Mostafa (kamalmostafa) wrote :
Stefan Bader (smb)
Changed in linux (Ubuntu Groovy):
assignee: nobody → Kamal Mostafa (kamalmostafa)
importance: Undecided → High
status: New → In Progress
Changed in linux (Ubuntu):
assignee: Kamal Mostafa (kamalmostafa) → nobody
importance: High → Undecided
status: In Progress → Invalid
Changed in linux (Ubuntu Groovy):
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (60.6 KiB)

This bug was fixed in the package linux - 5.8.0-63.71

---------------
linux (5.8.0-63.71) groovy; urgency=medium

  * Possible io_uring regression with QEMU on Ubuntu's kernel (LP: #1935017)
    - SAUCE: Revert "block: don't ignore REQ_NOWAIT for direct IO"

linux (5.8.0-62.70) groovy; urgency=medium

  * CVE-2021-33909
    - SAUCE: seq_file: Disallow extremely large seq buffer allocations

linux (5.8.0-61.68) groovy; urgency=medium

  * test_pmtu_vti4_link_add_mtu() test from net/pmtu.sh in
    ubuntu_kernel_selftests cannot finish properly on 5.11 and 5.8
    (LP: #1933969)
    - ip6_gre: proper dev_{hold|put} in ndo_[un]init methods
    - sit: proper dev_{hold|put} in ndo_[un]init methods
    - ip6_tunnel: sit: proper dev_{hold|put} in ndo_[un]init methods
    - ipv6: remove extra dev_hold() for fallback tunnels

linux (5.8.0-60.67) groovy; urgency=medium

  * groovy/linux: 5.8.0-60.67 -proposed tracker (LP: #1932438)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync getabis
    - [Packaging] update helper scripts
    - update dkms package versions

  * Add support for IO functions of AAEON devices (LP: #1929504)
    - ODM: mfd: Add support for IO functions of AAEON devices
    - ODM: gpio: add driver for AAEON devices
    - ODM: watchdog: add driver for AAEON devices
    - ODM: hwmon: add driver for AAEON devices
    - ODM: leds: add driver for AAEON devices
    - ODM: [Config] update config for AAEON devices

  * Add support for selective build of special drivers (LP: #1912789)
    - [Packaging] Add support for ODM drivers
    - [Packaging] Turn on ODM support for amd64
    - [Packaging] Fix ODM support in actual build

  * Upstream v5.9 introduced 'module' patches that removed exported symbols
    (LP: #1932065)
    - SAUCE: Revert "modules: inherit TAINT_PROPRIETARY_MODULE"
    - SAUCE: Revert "modules: return licensing information from find_symbol"
    - SAUCE: Revert "modules: rename the licence field in struct symsearch to
      license"
    - SAUCE: Revert "modules: unexport __module_address"
    - SAUCE: Revert "modules: unexport __module_text_address"
    - SAUCE: Revert "modules: mark each_symbol_section static"
    - SAUCE: Revert "modules: mark find_symbol static"
    - SAUCE: Revert "modules: mark ref_module static"

  * Disable hv-kvp-daemon.service on certain instance types (LP: #1932081)
    - [Packaging]: Add kernel command line condition to hv-kvp-daemon service

  * Groovy update: upstream stable patchset 2021-06-10 (LP: #1931637)
    - KEYS: trusted: Fix memory leak on object td
    - tpm: fix error return code in tpm2_get_cc_attrs_tbl()
    - tpm, tpm_tis: Extend locality handling to TPM2 in tpm_tis_gen_interrupt()
    - tpm, tpm_tis: Reserve locality in tpm_tis_resume()
    - KVM: x86/mmu: Remove the defunct update_pte() paging hook
    - PM: runtime: Fix unpaired parent child_count for force_resume
    - kvm: Cap halt polling at kvm->max_halt_poll_ns
    - ath11k: fix thermal temperature read
    - fs: dlm: fix debugfs dump
    - tipc: convert dest node's address to network order
    - ASoC: Intel: bytcr_rt5640: Enable jack-detect support on Asus T100TAF
    - net: stmmac: Set FIFO sizes for ipq806x
   ...

Changed in linux (Ubuntu Groovy):
status: Fix Committed → Fix Released
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-focal
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.