Annoying boot messages interfering with splash screen

Thank you for taking the time to report this issue and helping to make Ubuntu better. Examining the information you have given us, this does not appear to be a bug report so we are closing it and converting it to a question in the support tracker. We understand the difficulties you are facing, but it is better to raise problems you are having in the support tracker at https://answers.launchpad.net/ubuntu if you are uncertain if they are bugs. You can also find help with your problem in the support forum of your local Ubuntu community http://loco.ubuntu.com/ or asking at https://askubuntu.com or https://ubuntuforums.org. For help on reporting bugs, see https://help.ubuntu.com/community/ReportingBugs.

This is a bug. The 'splash' option is configured, but text messages from the kernel are being shown.

Hello,

I have spin up two VMs with the exact same specs, one using the 20.04.4 Desktop ISO and other the 22.04 Desktop ISO.

Please check the attached file "Screencast from 24-04-2022 16:39:39.webm".

GRUB config for both VMs:

GRUB_DEFAULT=saved
GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX=""
GRUB_SAVEDEFAULT=true
GRUB_DISABLE_OS_PROBER=false

As you can see, the 22.04 VM is definitely more verbose, taking splash screen out of the way.

And please remember these are VMs. On my real hardware I get even more messages that are interfering even more (just some low severity Kernel warnings that already existed before in Focal but they weren't interfering with the splash screen) (check attached file 20220422_014058.jpg).

The boot time is the same so this is definitely a low prio/cosmetic thing. But still something has changed that shouldn't.

Not sure if this is of any use, but just fyi, the fsck messages seem to come from here [1]. The USB messages are just dmesg logs.

[1] https://github.com/tytso/e2fsprogs/blob/master/e2fsck/unix.c#L445

Confirm the exact same behaviour on a fresh 22.04 install on a Dell Inspiron 16 7610.

Confirm this issue on Dell XPS 13 9310 2021

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1970069

Confirmed this problem on Lenovo Thinkpad T480s

Confirmed issue on iMac late 2017

This bug is still a thing for me.

This looks more likely to be a kernel bug than plymouth right now. It would only be a Plymouth bug if plymouthd was the one displaying the kernel log messages while it waits for DRM to start. Thereafter it switches to graphics mode.

I was wrong, the kernel has no interest in "splash" and I suspect it is plymouth displaying the unwanted messages. Perhaps we're not necessarily seeing the kernel log though. Are we just seeing systemd messages (PLY_ENABLE_SYSTEMD_INTEGRATION)? If that's it then we just need to make sure plymouthd's default mode is to not display anything (till DRM is ready with graphics).

I'm also having this on Ubuntu 23.10 on a Lenovo Slim Pro 9i. I'll see the UEFI splash, then some logs (including fsck), then the screen will go dark for a moment before the Ubuntu splash shows up.

Hello There!
Confirmed issue on HP Aero 13.

Like Ubuntu very much, but printable errors during Ubuntu loading despite `quiet splash` Grub's settings driving me mad.

Is any update for this issue?

:D I also use an HP Aero 13 and see this bug every time it boots. It's even more distracting on such a machine that skips displaying the BIOS logo.

I think the next step is to either try disabling PLY_ENABLE_SYSTEMD_INTEGRATION, or to work around Red Hat's preference for Plymouth to wait for DRM and allow it to render with fbdev immediately on boot (that would fix other bugs too).

I'm finally looking at this again. Theories tested today:

* Disabling PLY_ENABLE_SYSTEMD_INTEGRATION didn't fix anything. Seems we're dealing with kernel messages and not just systemd logging.

* Reducing device_timeout to 0.1 (seconds) in ubuntu-default-devicetimeout.patch helps a lot and hides most of the problem by not waiting in the background for DRM startup. But not all of the problem is fixed with that and presumably would be a regression of bug 1838725 on some machines.

New ideas for next week:

* The "quiet" kernel parameter is too noisy(!?). Perhaps "loglevel=0" would be better. I can't tell because I just broke the machine I was using to test.

* Would "boot_delay" be a way of hiding the early messages?

* "bgrt_disable": Why does this exist in the kernel and is it somehow interfering?

* Find out if plymouth's initramfs integration is capable of rendering a splash screen before kernel startup. If it ever was then it's sure not now.

* Go back and try focal on the same machine(s) to get a better idea of what it looked like when it was working before.

* Wait and try the proposed plymouth update for Noble that other people are working on. Upstream might have just fixed things.

The usb messages at least are because we use CONFIG_CONSOLE_LOGLEVEL_QUIET=4 (kernel default), Fedora uses 3 instead and one of the plymouth maintainers suggested we should do the same because errors messages are just difficult to avoid and we will keep hitting those type of issues otherwise ('$ sudo dmesg --level=err')

https://src.fedoraproject.org/rpms/kernel/blob/rawhide/f/kernel-x86_64-fedora.config#_1050

the fsck message is not coming from the kernel and another issue

I've opened bug #2049390 asking for our kernel default to be changed to 3 (I opened a new bug to help focusing the request and make it clear where the current report has more conversation and covers other issues than the loglevel)

#18
Instead of changing this patch can you finish the adoption for simpledrm? This will get a DRM framebuffer up and running instantly. It should help a lot.

#19
Presumably the fsck stuff is coming from systemd-fsck@.service, right?

> #18
> Instead of changing this patch can you finish the adoption for simpledrm?
> This will get a DRM framebuffer up and running instantly. It should help a lot.

Yes! I've been thinking the same. We already have bug 1965303 to track that.

> #19
> Presumably the fsck stuff is coming from systemd-fsck@.service, right?

Yes I do need to double-check if hacking out PLY_ENABLE_SYSTEMD_INTEGRATION affects the presence of those specific messages. When I said doing so didn't help in comment #18 I may have been looking at *all* messages.

I don't think the bug description is quite right here. I went back to 20.04 and no bug, but then tried 20.04.6 and the bug had already started. Reverting plymouth to the original version made no improvement. So the main difference I see between Ubuntu 20.04 (no bug) and Ubuntu 20.04.6 (bug) seems to be the kernel upgrade from 5.4 to 5.15.

We're not quite at a stage where we can close the plymouth task though, in case there is still something that can be done in plymouth to mask the new kernel messages.

It seems the BGRT logo gets displayed 3 times on boot, not 2. Why is that? You can see this more clearly when using the grub menu or a grub delay:

1. BIOS displays the BGRT logo.

2. Screen is cleared for the grub menu or delay.

3. Something displays the BGRT logo.

4. Screen is cleared and some kernel messages appear.

5. Plymouth finally takes over and displays the BGRT logo and boot animation for a split second.

6. Login screen appears.

Who or what is step 3? Is it a separate plymouth instance just for initrd?

> Who or what is step 3? Is it a separate plymouth instance just for initrd?

It's the kernel ("efifb: showing boot graphics").

A solution, but a bit scary:

1. If "splash" then tell fbcon it's not allowed to take over the framebuffer:

   fbcon=map:1

2. After plymouth is finished (or even when it's just started and taken over the framebuffer), tell fbcon it is allowed to use the framebuffer:

   $ con2fbmap 1 0

   This could also be done as a simple ioctl from plymouth itself.

Virtual terminals won't work until you do step 2.

Actually, not that scary. I think I can implement a complete solution as a single kernel patch. Just extend fbcon's deferred takeover feature to wait until a certain timeout has passed. And if "splash" is present then you can assume that should be a default of 10 seconds to wait for Plymouth's 8 second device timeout.

First draft, proof of concept, fix.

The attachment "0001-fbcon-Defer-AND-delay-console-take-over.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

The kernel patch is still in progress but I also wonder if plymouthd starts early enough to do all the console blocking? If so then it could FBIOPUT_CON2FBMAP like con2fbmap does, to umap the console from the framebuffer. But that's still racing with other kernel things which might have already dumped text onto the console. So a kernel patch is probably still the only fully reliable solution.

Here's a complete self-contained solution. Do kernel patches need to be proposed elsewhere?

If it's a SAUCE only patch, I think it should be proposed here: https://lists.ubuntu.com/mailman/listinfo/kernel-team

Added "Signed-off-by:"

I think it's fortunate the Ubuntu kernel team rejected my email formatting. They also suggest it should go upstream first. One reason against that was I originally couldn't get the upstream kernel to reproduce the bug. But today it does, so upstream patches shall go.

Here's what was sent for review upstream. Thankfully only the description changed, no code.

If anyone has the ability to test kernel patches in the meantime then please do. I'm starting a long weekend...

I'm not sure where you posted that to, I didn't see it dri-devel. But I think this is the first use of "splash" from the kernel command line within the kernel. I think it needs to be documented.

The kernel documentation explicitly mentions mailing lists for other components but not the one being modified:

FRAMEBUFFER CORE
M: Daniel Vetter <email address hidden>
S: Odd Fixes
T: git git://anongit.freedesktop.org/drm/drm-misc
F: drivers/video/fbdev/core/

This seems to be a bug in the kernel documentation.

I'm now waiting until my conversation with Daniel Vetter is concluded before formulating a plan for how to approach dri-devel.

> This seems to be a bug in the kernel documentation.

Yeah I guess so.

> I'm now waiting until my conversation with Daniel Vetter is concluded before formulating a plan for how to approach dri-devel.

👍

The userspace part of this bug is coming from our systemd patch:

  debian/fsckd-daemon-for-inter-fsckd-communication.patch

In particular where systemd-fsckd.service says:

  StandardOutput=journal+console

Is the patch still required at all, or can it be reordered to wait for Plymouth to splash first?

You can use the 'fastboot' kernel parameter to bypass the systemd-fsckd issue. So overall most systems can work around the bug in full using kernel parameters:

  quiet splash loglevel=2 fastboot

After that I've only encountered one weird laptop that refused to stay silent (even with loglevel=0).

The blanking from GRUB_TIMEOUT>0 will be made much worse by CONFIG_FB_EFI=n because usually in Ubuntu it's the efifb that restores the logo after Grub has wiped it. I think not having CONFIG_FB_EFI at the same time as using GRUB_TIMEOUT>0 explains your previous "really long time of a black screen".

I can tell you've restored GRUB_TIMEOUT=0 in the video and the one remaining screen blank at Plymouth's startup might be bug 1836858. But either way it's not this bug. Related bugs can be found at https://bugs.launchpad.net/ubuntu/+bugs?field.tag=flickerfreeboot

Patches sent (to multiple places per get_maintainer.pl):

https://lkml.org/lkml/2024/2/2/373
https://lkml.org/lkml/2024/2/2/375

Status changed to 'Confirmed' because the bug affects multiple users.

Status update:
https://lists.freedesktop.org/archives/dri-devel/2024-February/442066.html

So once merged, the fix will need to be manually enabled by:
FRAMEBUFFER_CONSOLE_DEFERRED_TAKEOVER_CONDITION = "splash"

I'm starting to suspect our patch (comment #39) isn't directly to blame for the systemd issue. Disabling or otherwise breaking systemd-fsckd.service doesn't stop fsck output from polluting the console and triggering this bug. Only skipping fsck avoids it. You can skip fsck using 'fastboot' or 'fsck.mode=skip' on the kernel command line.

The fact that fsck prints a useless success message to its stdout is the issue, but it's upstream systemd asking for that:

  https://github.com/systemd/systemd/blob/main/src/fsck/fsck.c#L336

and there doesn't seem to be any simple fsck parameter to tell it to run more quietly (systemd already uses fsck -T).

I think maybe Red Hat unwittingly works around the problem by starting the Plymouth splash earlier in initrd because they have SimpleDRM enabled.

Also focal (20.04.6) does have the bug now, per comment #23.

Noble is now shipping with SimpleDRM enabled in kernel 6.8, so the next step is comment #49 (which would also resolve bug 1869655).

It's a bit unclear to me what change we need now to take advantage of simpledrm? Comment #49 state getting plymouth in the initrd but that should already be the case today or luks encryption wouldn't be working no?
what change would trigger the include of extra drm modules?

Plymouth only gets added to the initrd when LUKS is enabled or you mark another reason for needing the framebuffer.

So the suggestion I have from comment #49 is to mark needing the framebuffer by default, and then stop including any DRM modules because simpledrm is built into the kernel.

AFAICT; these initramfs-tools package changes would cover it:

* hooks/framebuffer can be totally removed.
* scripts/init-top/framebuffer should probably stay
* conf/initramfs.conf needs FRAMEBUFFER=y added to it

I've analysed a couple of Noble systems today and don't think initramfs-tools needs improving (although it might not hurt). On both systems, plymouth-start was within 1 second of simpledrm starting (same timestamp), but the bug still occurs. The reasons are:

1. One system actually experiences the bug ("fbcon: Taking over console") _before_ "Initialized simpledrm" due to kernel error-level messages.

2. After setting loglevel=3 (bug 2049390), it's a slight improvement but the "clean" message from fsck via systemd still interrupts the boot sequence.

So it looks like we need either:

* loglevel=3 (bug 2049390) and a systemd fix for fsck; or
* The kernel patch I've already proposed upstream.

You can test if the above suggestion will work for you by adding these kernel parameters:

  loglevel=3 fsck.mode=skip

Maybe initramfs-tools does still need changing. plymouth-start.service seems to run twice, and I get the feeling only the second instance is actually succeeding in splashing on screen.

Even if FRAMEBUFFER=Y wasn't added, I think that a change to stop adding all those other drm drivers makes a lot of sense. No use doubling the initrd size for the LUKS case.

> * hooks/framebuffer can be totally removed.

By chance I was testing on a desktop that has the Nvidia driver installed (though no Nvidia card today, Intel only). Disabling hooks/framebuffer had relatively little impact. In the end it was hooks/framebuffer-nvidia that also needed disabling to remove most of the initrd bloat.

Also hooks/plymouth is equally to blame and needs modifying, as mentioned in comment #49 which I forgot.

Looking at my older 6.6 kernel however I'm reminded that some kernels may have the drivers we need in the form of tiny drm (which is the backend for simpledrm??):

  /lib/modules/6.6.0-14-generic/kernel/drivers/gpu/drm/tiny/

I don't think we'd want to exclude those from initrd.

Hopefully I'm wrong about the need for all the 'tiny' drivers and simpledrm is in a class of its own. But I'd like to better understand what the backend of simpledrm is and how much we can assume that it will just work(tm).

The backend is a framebuffer driver. For example efifb which uses the framebuffer set up by GOP in pre-boot.

If there are framebuffer drivers in tinydrm then maybe they matter for those architectures.

That might explain the strange behaviour I'm seeing where simpledrm doesn't seem to be usable until a few seconds into the boot (maybe not till i915drmfb is mentioned in the log?).

/boot/config-6.8.0-11-generic:# CONFIG_FB_EFI is not set

It was removed as part of bug 1965303. But I was hoping simpledrm magically bypassed the need for fbdev modules.

Can you double check the framebuffer FB related conf options in your kconfig against those in Fedora?

Fedora 39:

CONFIG_FB_VESA=y
CONFIG_FB_EFI=y

Ubuntu 24.04:

# CONFIG_FB_VESA is not set
# CONFIG_FB_EFI is not set

In the kernel source, DRM_SIMPLEDRM recommends SYSFB_SIMPLEFB if you want it to work on UEFI and VESA. And we do enable SYSFB_SIMPLEFB whose help text says:

> you should still keep vesafb and others enabled as fallback
> if a system framebuffer is incompatible with simplefb.

So overall I don't think SimpleDRM *needs* any legacy fbdev like FB_EFI or FB_VESA, they're just recommended as a fallback.

I'll do more testing in the coming days to verify if SimpleDRM is really working as intended. But the main concern here is that we might have broken some systems by disabling the legacy fbdev drivers. I'm not sure. Anyone testing Noble with kernel 6.8 right now should be able to tell us...

I think the delay I'm seeing is a feature of Plymouth, not a bug:

  "ignoring since we only handle SimpleDRM devices after timeout"

As much as I dislike it perpetuating bug 1869655, this feature does prevent bug 2054769 from reoccurring.

But this raises the question: is it worth putting i915, amdgpu etc in initrd just to try and solve bug 1869655? Possibly not.

Plymouth would still need patching in order to optimize the drm modules in debian/local/plymouth.hook

Seems it's fairly easy to make a working solution in initramfs-tools alone, but there's a compromise. If I only include tiny/simple framebuffer drivers in initrd then i915 starts so late that Plymouth doesn't use it, and bug 2054769 can occur on some machines. If I include all the current drm drivers in initrd then it bloats to 100MB, but at least i915 starts early enough (only just) to avoid bug 2054769.

I'm currently leaning toward only including tiny/simple framebuffer drivers and trying to solve bug 2054769 with additional heuristics in Plymouth itself (when the physical monitor dimensions are unknown under simpledrm). If nothing else, fixing this bug is more important than fixing bug 2054769 for all cases.

https://code.launchpad.net/~vanvugt/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+merge/462482

Here's a fix for plymouth/noble.

The source is in:
https://salsa.debian.org/ubuntu-dev-team/plymouth/-/commits/ubuntu/latest

We skipped ubuntu5 because that technically existed in bug 2054769 for 10 days already.

Also note this will cause some temporary initrd bloat while the initramfs-tools change is pending in comment #90.

It seems unlikely my all-in-one kernel fix will be accepted upstream. While I could propose it as an Ubuntu patch, another alternative is to just let the linux task track bug 2049390 (Fix Committed). Then most people will only need the above Plymouth patch to get a complete fix.

The remaining tasks beyond that are:

initramfs-tools: Required to keep the initrd size under control after the Plymouth fix is released.

systemd: Would still be nice to implement a redundant fix to suppress the "clean" message. That would fix bug 1914409 and when combined with loglevel=3, would also prevent this bug 1970069.

Looking at the remaining "add drm modules (only if MODULES=dep)" section: This logic should moved to hooks/framebuffer as well.

That change is in plymouth, but hooks/framebuffer is in initramfs-tools. If you're suggesting moving the code from plymouth into initramfs-tools then I kind of understand, but it's not directly related to fixing this bug.

If you are suggesting moving the logic from plymouth into initramfs-tools then please start by duplicating the code you'd like moved into initramfs-tools. That would be efficient since you have commit access to initramfs-tools, and would mean I only have to deal with plymouth here.

Daniel, please review and comment https://code.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+merge/462691 (see the description for the two possible implementations)

Sponsored plymouth 24.004.60-1ubuntu6 after dropping the complete drm modules code and correcting the changelog entries. Unsubscribing ~ubuntu-sponsors.

Thanks. I've reflected those changes in git:
https://salsa.debian.org/ubuntu-dev-team/plymouth/-/commits/ubuntu/latest/

https://salsa.debian.org/ubuntu-dev-team/plymouth/-/commit/bd9d0119fdbeb3b30c8d3caafdb8a31b23c188c4 does not reflect the removal of the complete drm section.

That commit is not meant to reflect the outcome of discussions that happened after it (comment #93). It's just an intermediate "UNRELEASED" commit reflecting intermediate development.

My intent with shared repos like this is to show what is being worked on before anything is released to the archive. The tags still accurately reflect what reached or was proposed to the archive.

This bug was fixed in the package initramfs-tools - 0.142ubuntu23

---------------
initramfs-tools (0.142ubuntu23) noble; urgency=medium

  [ Daniel van Vugt ]
  * hooks/framebuffer: Only add simple/tiny framebuffer drivers. This is to
    limit the size of initrd when FRAMEBUFFER=y is soon enabled for desktop
    installations (LP: #1970069, #1869655).

  [ Benjamin Drung ]
  * autopkgtest: Increase QEMU timeouts on arm64/armhf
  * hooks/framebuffer:
    - Move adding framebuffer drivers into auto_add_modules
    - Drop looking in $MODULESDIR/initrd/ for kernel modules
    - Support MODULES=dep in framebuffer hook

initramfs-tools (0.142ubuntu22) noble; urgency=medium

  * autopkgtest: update systemd-udevd path from /lib to /usr/lib

initramfs-tools (0.142ubuntu21) noble; urgency=medium

  [ Benjamin Drung ]
  * configure_networking:
    - Increase minimum timeout to 30 seconds
    - Fix configuring BOOTIF when using iSCSI (LP: #2056187)
    - Set interface MTU if provided by the DHCP server (LP: #2056194)
    - log sleep durations before retries
  * Copy /etc/passwd into the initramfs to allow dhcpcd running as dhcpcd user
  * Replace obsolete pkg-config build-dependency by pkgconf

  [ Dan Bungert ]
  * Restore nvdimm and dax pmem-related modules (LP: #1981385)

 -- Benjamin Drung <email address hidden> Thu, 21 Mar 2024 10:57:54 +0100

This bug was fixed in the package plymouth - 24.004.60-1ubuntu6

---------------
plymouth (24.004.60-1ubuntu6) noble; urgency=medium

  * Cherry-pick upstream fixes for consistent default scale selection
    that matches what Mutter chooses on the login screen (LP: #2054769)
  * plymouth.hook: Stop automatically re-installing DRM kernel modules in
    initrd. initramfs-tools already does this for us in hooks/framebuffer and
    should be the authority for which drivers to include.
  * Install a /usr/share/initramfs-tools/conf-hooks.d/plymouth with
    FRAMEBUFFER=y to make it clear to initramfs-tools that we always want
    framebuffer support when Plymouth is installed. (LP: #1970069)

 -- Daniel van Vugt <email address hidden> Thu, 21 Mar 2024 14:46:56 +0100

In case anyone is wondering, you won't start to see an improvement until you have at least:

linux >= 6.8.0-20.20 and
plymouth >= 24.004.60-1ubuntu6

I've installed the 20240410 iso on a couple of machines and this bug is as good as fixed.

The systemd task would be nice to complete but that's more for the sake of bug 1914409. Right now I have more important bugs to focus on...