vmwgfx fails to reserve graphics buffer on aarch64 leading to blank display

This change was made by a bot.

This same issue impacts Ubuntu 22.04.2 HWE 5.19.0-32 kernels on aarch64.

Kernel logging shows that the vmwgfx driver encounters an error when loading. The error indicates that a memory range could not be acquired by the driver. The error is eerily similar to https://bugzilla.kernel.org/show_bug.cgi?id=215678 which was fixed in kernel source in 5.19.

The behavior seen in this kernel is a regression to past kernel version behavior. Is there anyone at Ubuntu that would care to look at this issue? It's been crickets for over a month.

Is it possible to identify the offending commit?

I'm not an Ubuntu kernel engineer, so I can't help identify the offending commit.
What I do see is the following in dmesg:

[ 11.135360] vmwgfx 0000:00:0f.0: BAR 2: can't reserve [mem 0x70000000-0x77ffffff 64bit pref]
[ 11.135366] vmwgfx: probe of 0000:00:0f.0 failed with error -16

And a cat /proc/iomem shows this:

50000000-7fffffff : PCI Bus 0000:00
  70000000-77ffffff : 0000:00:0f.0
    70000000-702fffff : BOOTFB

The kernel has refused to release BOOTFB, and that explains why vmwgfx could not obtain it.

This did not happen in 5.19.0-29, and does not happen in mainline 6.1.12. Example from mainline kernel 6.1.12 dmesg (same system, only chose to boot from 6.1.2 kernel):

[ 9.173979] vmwgfx 0000:00:0f.0: [drm] Register MMIO at 0x0x000000003d000000 size is 4096 kiB
[ 9.173985] vmwgfx 0000:00:0f.0: [drm] VRAM at 0x0000000070000000 size is 131072 kiB
[ 9.173989] vmwgfx 0000:00:0f.0: [drm] Running on SVGA version 3.
[ 9.173992] vmwgfx 0000:00:0f.0: [drm] Capabilities: cursor, cursor bypass, alpha cursor, pitchlock, irq mask, traces, command buffers, command buffers 2, gbobject, dx, hp cmd queue, no bb restriction, cap2 register,
[ 9.173994] vmwgfx 0000:00:0f.0: [drm] Capabilities2: grow otable, intra surface copy, dx2, gb memsize 2, screendma reg, otable ptdepth2, non ms to ms stretchblt, cursor mob, mshint, cb max size 4mb, dx3, frame type, trace full fb, extra regs, lo staging,
[ 9.173995] vmwgfx 0000:00:0f.0: [drm] DMA map mode: Caching DMA mappings.
[ 9.174030] vmwgfx 0000:00:0f.0: [drm] Legacy memory limits: VRAM = 4096 kB, FIFO = 256 kB, surface = 524288 kB
[ 9.174031] vmwgfx 0000:00:0f.0: [drm] MOB limits: max mob size = 262144 kB, max mob pages = 196608
[ 9.174032] vmwgfx 0000:00:0f.0: [drm] Maximum display memory size is 262144 kiB
[ 9.176279] vmwgfx 0000:00:0f.0: [drm] No GMR memory available. Graphics memory resources are very limited.
[ 9.176967] vmwgfx 0000:00:0f.0: [drm] Screen Target display unit initialized
[ 9.177214] vmwgfx 0000:00:0f.0: [drm] Using command buffers with DMA pool.
[ 9.177218] vmwgfx 0000:00:0f.0: [drm] Available shader model: Legacy.
[ 9.177749] Console: switching to colour frame buffer device 128x48
[ 9.181076] [drm] Initialized vmwgfx 2.20.0 20211206 for 0000:00:0f.0 on minor 0

and /proc/iomem:

50000000-7fffffff : PCI Bus 0000:00
  70000000-77ffffff : 0000:00:0f.0
    70000000-77ffffff : vmwgfx probe

Here is VMware's analysis: https://communities.vmware.com/t5/VMware-Fusion-Discussions/EDITED-Workaround-for-Ubuntu-22-10-displaying-blank-screen-after/m-p/2955292/highlight/true#M182008 .

They point the finger at this backport: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/kinetic/commit/?id=89314ff239e1933357419fa91b20190150f114a8 , which they say only works as part of a larger series, I guess https://patchwork.freedesktop.org/series/106040/ , although I didn't check which of those made it into Linus' tree. I don't know anything about this subsystem or which patches from the series are actually needed to fix this issue, but the series does seem fairly hefty for an SRU, so perhaps a revert would be preferable?

Whatever patch series is needed does seem to be included in Linus' 5.19 tree as the mainline kernel 5.19.17 (supposedly built from Linus' sources) does not have this issue.

Status changed to 'Confirmed' because the bug affects multiple users.

Hi everyone,

I have built a test kernel based on the current 5.19.0-38-generic kernel for
both Kinetic and Jammy HWE. It has the below patch reverted:

commit 5e01376124309b4dbd30d413f43c0d9c2f60edea
Author: Thomas Zimmermann <email address hidden>
Date: Mon Jul 18 09:23:18 2022 +0200
Subject: video/aperture: Disable and unregister sysfb devices via aperture helpers
Link: https://github.com/torvalds/linux/commit/5e01376124309b4dbd30d413f43c0d9c2f60edea

The revert patch is below:

https://paste.ubuntu.com/p/F5dxtqPqxC/

Could someone please test the test kernel to see if it fixes the problem on
their kinetic VM running on VMWare?

You may want to make the grub menu selectable first, in case the test kernel
does not work, so you can easily swap back to 5.19.0-29-generic which works.

Edit as root:
/etc/default/grub

Change GRUB_TIMEOUT_STYLE=hidden to GRUB_TIMEOUT_STYLE=menu and add a decent
timeout GRUB_TIMEOUT=30

Then update grub:

$ sudo update-grub

If you can test 5.19.0-38-generic from -updates first, and then test the test
kernel, it would be best to compare the two directly. Hopefully the test kernel
resolves the issue.

Please note this package is NOT SUPPORTED by Canonical, and is for TESTING
PURPOSES ONLY. ONLY Install in a dedicated test environment.

Instructions to install (on a Jammy or Kinetic system):
1) sudo add-apt-repository ppa:mruffell/lp2007001-test
2) sudo apt update
3) sudo apt install linux-image-unsigned-5.19.0-38-generic linux-modules-5.19.0-38-generic linux-modules-extra-5.19.0-38-generic linux-headers-5.19.0-38-generic
4) sudo reboot
5) uname -rv
5.19.0-38-generic #39+TEST2007001v20230418b1-Ubuntu SMP PREEMPT_DYNAMIC Tue Apr 18

Upon boot, you should be able to see the screen as normal on your VMWare VM.

Let me know if the test kernel works.

Thanks,
Matthew

Hi Matthew,

Thanks for your test kernel (and thanks for upload the jammy hwe version of it too)

I can confirm this kernels gives me video output in VMware Fusion 13.0.1

No error on vmwgfx startup:
[ 0.000000] Linux version 5.19.0-38-generic (buildd@bos02-arm64-003) (aarch64-linux-gnu-gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #39~22.04.1+TEST2007001v20230418b1-Ubuntu SMP PREEMPT_DYNAMIC Tu (Ubuntu 5.19.0-38.39~22.04.1+TEST2007001v20230418b1-generic 5.19.17)
[...]
[ 1.284460] vmwgfx 0000:00:0f.0: [drm] Register MMIO at 0x0x000000003d000000 size is 4096 kiB
[ 1.285181] vmwgfx 0000:00:0f.0: [drm] VRAM at 0x0000000070000000 size is 131072 kiB
[ 1.285193] vmwgfx 0000:00:0f.0: [drm] Running on SVGA version 3.
[ 1.285196] vmwgfx 0000:00:0f.0: [drm] Capabilities: cursor, cursor bypass, alpha cursor, 3D, pitchlock, irq mask, traces, command buffers, command buffers 2, gbobject, dx, hp cmd queue, no bb restriction, cap2 register,
[ 1.285201] vmwgfx 0000:00:0f.0: [drm] Capabilities2: grow otable, intra surface copy, dx2, gb memsize 2, screendma reg, otable ptdepth2, non ms to ms stretchblt, cursor mob, mshint, cb max size 4mb, dx3, frame type, trace full fb, extra regs, lo staging,
[ 1.285204] vmwgfx 0000:00:0f.0: [drm] DMA map mode: Caching DMA mappings.
[ 1.285243] vmwgfx 0000:00:0f.0: [drm] Legacy memory limits: VRAM = 4096 kB, FIFO = 256 kB, surface = 524288 kB
[ 1.285259] vmwgfx 0000:00:0f.0: [drm] MOB limits: max mob size = 1048576 kB, max mob pages = 524288
[ 1.285261] vmwgfx 0000:00:0f.0: [drm] Maximum display memory size is 262144 kiB
[ 1.289628] vmwgfx 0000:00:0f.0: [drm] No GMR memory available. Graphics memory resources are very limited.
[ 1.289794] vmwgfx 0000:00:0f.0: [drm] Screen Target display unit initialized
[ 1.290031] vmwgfx 0000:00:0f.0: [drm] Using command buffers with DMA pool.
[ 1.290037] vmwgfx 0000:00:0f.0: [drm] Available shader model: SM_5_1X.
[ 1.290439] Console: switching to colour frame buffer device 128x48
[ 1.291782] [drm] Initialized vmwgfx 2.20.0 20211206 for 0000:00:0f.0 on minor 0

To double check, I reinstalled the latest 5.19 HWE kernel for 22.04 and double-checked I could reproduce the issue in the archive:
[ 0.000000] Linux version 5.19.0-40-generic (buildd@bos02-arm64-021) (aarch64-linux-gnu-gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #41~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Mar 31 16:02:33 UTC 2 (Ubuntu 5.19.0-40.41~22.04.1-generic 5.19.17)
[...]
[ 0.739889] vmwgfx 0000:00:0f.0: BAR 2: can't reserve [mem 0x70000000-0x77ffffff 64bit pref]
[ 0.739893] vmwgfx: probe of 0000:00:0f.0 failed with error -16

Thanks a lot for the fix and looking forward to seeing it in jammy-updates :-)

Hi everyone,

I have submitted the revert to the Kernel Team for SRU:

Cover letter:
https://lists.ubuntu.com/archives/kernel-team/2023-April/138908.html
Patch:
https://lists.ubuntu.com/archives/kernel-team/2023-April/138909.html

The next step is for the kernel team to review the patch, and for two senior
Kernel Team members to ack the patch. At that stage, it will be accepted into
the next SRU cycle.

You can track the dates for the 2023.05.15 SRU cycle on https://kernel.ubuntu.com/

I will write back once have have two acks from the Kernel Team.

Thanks,
Matthew

This bug is awaiting verification that the linux/5.19.0-44.45 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-kinetic' to 'verification-done-kinetic'. If the problem still exists, change the tag 'verification-needed-kinetic' to 'verification-failed-kinetic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Matthew,

I installed the 5.19.0-44.45 kernel from -proposed, and the console now appears. It does seem to solve the problem.

dmesg reports:

[ 9.808628] vmwgfx 0000:00:0f.0: [drm] Register MMIO at 0x0x000000003d000000 size is 4096 kiB
[ 9.808637] vmwgfx 0000:00:0f.0: [drm] VRAM at 0x0000000070000000 size is 131072 kiB
[ 9.808642] vmwgfx 0000:00:0f.0: [drm] Running on SVGA version 3.
[ 9.808644] vmwgfx 0000:00:0f.0: [drm] Capabilities: cursor, cursor bypass, alpha cursor, pitchlock, irq mask, traces, c
ommand buffers, command buffers 2, gbobject, dx, hp cmd queue, no bb restriction, cap2 register,
[ 9.808647] vmwgfx 0000:00:0f.0: [drm] Capabilities2: grow otable, intra surface copy, dx2, gb memsize 2, screendma reg,
 otable ptdepth2, non ms to ms stretchblt, cursor mob, mshint, cb max size 4mb, dx3, frame type, trace full fb, extra regs,
 lo staging,
[ 9.808648] vmwgfx 0000:00:0f.0: [drm] DMA map mode: Caching DMA mappings.
[ 9.808693] vmwgfx 0000:00:0f.0: [drm] Legacy memory limits: VRAM = 4096 kB, FIFO = 256 kB, surface = 524288 kB
[ 9.808693] vmwgfx 0000:00:0f.0: [drm] MOB limits: max mob size = 262144 kB, max mob pages = 196608
[ 9.808696] vmwgfx 0000:00:0f.0: [drm] Maximum display memory size is 262144 kiB
[ 9.818387] vmwgfx 0000:00:0f.0: [drm] No GMR memory available. Graphics memory resources are very limited.
[ 9.818514] vmwgfx 0000:00:0f.0: [drm] Screen Target display unit initialized
[ 9.818748] vmwgfx 0000:00:0f.0: [drm] Using command buffers with DMA pool.
[ 9.818753] vmwgfx 0000:00:0f.0: [drm] Available shader model: Legacy.
[ 9.819374] Console: switching to colour frame buffer device 128x48
[ 9.822461] [drm] Initialized vmwgfx 2.20.0 20211206 for 0000:00:0f.0 on minor 0

and /proc/iomem reports

50000000-7fffffff : PCI Bus 0000:00
  70000000-77ffffff : 0000:00:0f.0
    70000000-77ffffff : vmwgfx probe
  78000000-784fffff : PCI Bus 0000:01

Wonderful! Thank you very much Paul for testing!

This will slowly work its way through the Kernel SRU process. We should see a release to -updates the week of 5th June, as per https://kernel.ubuntu.com/, give or take a few days if any CVEs turn up.

Is it worth having a separate task for linux-hwe in jammy?

I've tested 5.19.0-42.43~22.04.1 from jammy-proposed which had a few vmwgfx changes listed in its changelog, but it didn't work; I didn't see the same revert as in 5.19.0-44.45 or a reference to that bug, so I'll open a jammy task to land this fix there too.

Its not necessary Loic, linux-hwe is a derivative of the kinetic kernel, and it will be automatically built and pushed to -proposed in due course, likely over the next few days.

It will be a part of the 5.19.0-44 HWE kernel. I'll keep an eye on it, I don't think the HWE kernel is moving to the 6.2 kernel just yet.

This bug is awaiting verification that the linux-nvidia-5.19/5.19.0-1014.14 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

I can verify that the 5.10.0-1014 kernel in -proposed does boot on a VMware Fusion VM running Jammy aarch64, and that the console appears as expected. I'm hesitant to change the tag to verification-done-jammy since I don't have anything nVidia on arm64. I can do that if my testing meets what you're looking for.

I've verified that the vmwgfx driver has claimed the fb memory:

dmesg shows:

[ 0.867225] vmwgfx 0000:00:0f.0: [drm] Register MMIO at 0x0x000000003d000000 size is 4096 kiB
[ 0.867229] vmwgfx 0000:00:0f.0: [drm] VRAM at 0x0000000070000000 size is 131072 kiB
[ 0.867234] vmwgfx 0000:00:0f.0: [drm] Running on SVGA version 3.
[ 0.867237] vmwgfx 0000:00:0f.0: [drm] Capabilities: cursor, cursor bypass, alpha cursor, pitchlock, irq mask, traces, comma
nd buffers, command buffers 2, gbobject, dx, hp cmd queue, no bb restriction, cap2 register,
[ 0.867241] vmwgfx 0000:00:0f.0: [drm] Capabilities2: grow otable, intra surface copy, dx2, gb memsize 2, screendma reg, ota
ble ptdepth2, non ms to ms stretchblt, cursor mob, mshint, cb max size 4mb, dx3, frame type, trace full fb, extra regs, lo stag
ing,
[ 0.867243] vmwgfx 0000:00:0f.0: [drm] DMA map mode: Caching DMA mappings.
[ 0.867279] vmwgfx 0000:00:0f.0: [drm] Legacy memory limits: VRAM = 4096 kB, FIFO = 256 kB, surface = 524288 kB
[ 0.867281] vmwgfx 0000:00:0f.0: [drm] MOB limits: max mob size = 262144 kB, max mob pages = 196608
[ 0.867283] vmwgfx 0000:00:0f.0: [drm] Maximum display memory size is 262144 kiB

and cat /proc/iomem shows:

50000000-7fffffff : PCI Bus 0000:00
  70000000-77ffffff : 0000:00:0f.0
    70000000-77ffffff : vmwgfx probe

Apologies for the typo - I meant the linux-nvidia-5.19/5.19.0-1014.14 kernel in -proposed.

Hi Paul,

Thanks for verifying that linux-nvidia kernel, but for the future, you don't need to worry about verifying each and every kernel series. These days, the kernel team maintains over 100 kernel variants, and it is simply impossible to check them all.

We just check the primary ones, e.g. 5.19 in kinetic, and then when they are made into their specific derivative kernels, e.g. 5.19 linux-nvidia, they contain all the patches from the primary kernel, plus some specific patches to enable whatever hardware the derivative kernel is for.

So, don't worry if there are any more from other odd kernel combinations, I will let you know if we need to do any more testing.

In other news, looking at https://kernel.ubuntu.com/, we should be on track for a release to -updates early next week, so look forward to that and getting this bug fixed for everyone.

Thanks,
Matthew

Thanks Matthew. I was erring on the side of caution as it was easy for me to at least give that variant a whirl on my trusty Jammy VM. I'll definitely take your lead on the need for future testing.

Looking forward to release next week. I'll update you on my experiences - but I suspect that it'll be a non-event.

Should I assume that the -proposed kernel will be made available to both Kinetic and Jammy HWE? I maintain a tips/techniques document for VMWare Fusion users on Apple Silicon. This issue is highlighted along with a few work-arounds. If this is fixed in both releases I can test both and remove a lot of work-around discussions.

Thanks,

Paul

Hi Paul,

Yes, if we look at https://kernel.ubuntu.com/sru/dashboards/web/kernel-stable-board.html under 2023.05.15 SRU cycle, under Jammy it has linux-hwe-5.19 5.19.0-44.45~22.04.1 sitting in -proposed, so yes, this fix will be made available to kinetic and jammy-hwe.

Saying that, the dashboard also lists 5.19.0-45 for both, but still in the preparation stage and not pushed to -proposed yet. I looked up the code, and 5.19.0-45 only has one patch ontop of 5.19.0-44, which is a fix to https://bugs.launchpad.net/bugs/2020599, fixing a regression in WPA offload.

commit 3358af2b077f8e90461a9a8941e8292b995db63b
Author: Hector Martin <email address hidden>
Date: Sat Mar 11 23:19:14 2023 +0900
wifi: cfg80211: Partial revert "wifi: cfg80211: Fix use after free for wext"
BugLink: https://bugs.launchpad.net/bugs/2020599

I think the kernel team will probably get 5.19.0-45 into -proposed as soon as they can, and this will likely be the kernel that will be released next week.

You won't need to re-verify, since its the same as 5.19.0-44 with a single patch ontop.

Thanks,
Matthew

This bug was fixed in the package linux - 5.19.0-45.46

---------------
linux (5.19.0-45.46) kinetic; urgency=medium

  * kinetic/linux: 5.19.0-45.46 -proposed tracker (LP: #2023057)

  * Kinetic update: upstream stable patchset 2023-05-23 (LP: #2020599)
    - wifi: cfg80211: Partial revert "wifi: cfg80211: Fix use after free for wext"

linux (5.19.0-44.45) kinetic; urgency=medium

  * kinetic/linux: 5.19.0-44.45 -proposed tracker (LP: #2019827)

  * Linux 5.19 amdgpu: NULL pointer on GCN2 and invalid load on GCN1
    (LP: #2018470)
    - drm/amdgpu: Fix for BO move issue

  * CVE-2023-32233
    - netfilter: nf_tables: deactivate anonymous set from preparation phase

  * CVE-2023-2612
    - SAUCE: shiftfs: prevent lock unbalance in shiftfs_create_object()

  * CVE-2023-31436
    - net: sched: sch_qfq: prevent slab-out-of-bounds in qfq_activate_agg

  * CVE-2023-1380
    - wifi: brcmfmac: slab-out-of-bounds read in brcmf_get_assoc_ies()

  * conntrack mark is not advertised via netlink (LP: #2016269)
    - netfilter: ctnetlink: revert to dumping mark regardless of event type

  * 5.19 not reporting cgroups v1 blkio.throttle.io_serviced (LP: #2016186)
    - SAUCE: blk-throttle: Fix io statistics for cgroup v1

  * [SRU] Backport request for hpwdt from upstream 6.1 to Jammy (LP: #2008751)
    - watchdog/hpwdt: Enable HP_WATCHDOG for ARM64 systems.
    - watchdog/hpwdt: Include nmi.h only if CONFIG_HPWDT_NMI_DECODING
    - [Config] Add arm64 option to CONFIG_HP_WATCHDOG

  * vmwgfx fails to reserve graphics buffer on aarch64 leading to blank display
    (LP: #2007001)
    - SAUCE: Revert "video/aperture: Disable and unregister sysfb devices via
      aperture helpers"

  * Ubuntu 22.04 raise abnormal NIC MSI-X requests with larger CPU cores (256)
    (LP: #2012335)
    - ice: Allow operation with reduced device MSI-X

  * Dell: Enable speaker mute hotkey LED indicator (LP: #2015972)
    - platform/x86: dell-laptop: Register ctl-led for speaker-mute

  * [SRU]With "Performance per Watt (DAPC)" enabled in the BIOS, Bootup time is
    taking longer than expected (LP: #2008527)
    - cpufreq: ACPI: Defer setting boost MSRs

  * [SRU][Jammy] CONFIG_PCI_MESON is not enabled (LP: #2007745)
    - [Config] arm64: Enable PCI_MESON module

  * Kinetic update: upstream stable patchset 2023-05-08 (LP: #2018948)
    - HID: asus: use spinlock to protect concurrent accesses
    - HID: asus: use spinlock to safely schedule workers
    - powerpc/mm: Rearrange if-else block to avoid clang warning
    - ARM: OMAP2+: Fix memory leak in realtime_counter_init()
    - arm64: dts: qcom: qcs404: use symbol names for PCIe resets
    - arm64: dts: qcom: msm8996-tone: Fix USB taking 6 minutes to wake up
    - arm64: dts: qcom: sm8150-kumano: Panel framebuffer is 2.5k instead of 4k
    - arm64: dts: qcom: sm6125: Reorder HSUSB PHY clocks to match bindings
    - arm64: dts: imx8m: Align SoC unique ID node unit address
    - ARM: zynq: Fix refcount leak in zynq_early_slcr_init
    - arm64: dts: mediatek: mt8183: Fix systimer 13 MHz clock description
    - arm64: dts: qcom: sdm845-db845c: fix audio codec interrupt pin name
    - arm64: dts: qcom: sc7180: correct SPMI bus addres...

Installed the released kernel now available from updates. All works as expected.
The HWE 22.10 kernel also works in 22.04.2.

The 22.04.2 ISO installer still has a 5.19.0-32 kernel, though. It exhibits the blank screen issue when choosing the "Ubuntu Server with HWE kernel" installer option. I suspect that issue will get fixed in 22.04.3, but this issue can be easily worked around.

Thanks to all of the Ubuntu crew for delivering this fix.