gnome-shell freezes on notification arrival (fixed upstream)

Thanks for the bug report.

If you would like the bug fixed in 18.04 then please click the above link "Nominate for series".

Hi Daniel, thanks for your prompt reply.
Good to see that this problem is fixed in future versions of Ubuntu.
Indeed it would be nice to see it in 18.04, I'm not a member of the project therefore I can't do "Nominate for series" but I hope someone does nominate it as the issue seems relatively small and improves performance/avoid a crash.

You should see the "Nominate for series" link above "Bug Description". Please click it if you can.

I can't. If I click it I get "You do not have permission to nominate this bug." since I'm not a maintainer of this package.

I'm not sure this fix is complete. What about, for example: https://gitlab.gnome.org/GNOME/gnome-shell/-/commit/19e084036a1dc3c5da010c3e7a42ce67df317797? As far as I understand, it is exactly (and only) the fsync() call that we're trying to move off the main thread, and this commit suggests to me that the fix proposed in this SRU won't actually do that.

Please could you take a look, and also for anything else relevant in upstream's branches? We risk regression every time we land an SRU, so if this commit or others need to go in also, then it'd be better for us to SRU a fix for this only once.

Given the age of this bug I suggest it's most helpful to make each series' fix the same. If the symptoms, or similar symptoms persist then I would track that in a new bug so that we can make additional fixes the same in each series.

Disco and Eoan are EOL though, and Focal looks like it already has upstream commit 19e0840 included. So isn't it your proposed Bionic upload that would be different? Why not look into making that the same as what is in Focal?

Regardless, I'd like you to please at least consider the possibility that cherry-picking only 86a00b6 from upstream is an incomplete fix to this bug and look into further upstream changes to that file since that commit to see if 19e0840 or others are relevant. If after understanding the relevant changes you conclude that there is still a benefit to SRU this as-is, can explain the trade-off in not also taking the other relevant changes and can make the case for 19e0840 being correct on its own, then I'll consider that.

As it stands it appears to me that 19e0840 contains an oversight that was since corrected upstream, but your SRU doesn't include the correction. Unless you change my mind, I don't consider this acceptable for SRU.

Sounds reasonable. Marco?

Yes, I wanted to keep the changes minimal initially, as the reporter mentioned that that commit was enough, but I can reiterate with the whole change that is in fact complete with the whole set.

An upload of gnome-shell to bionic-proposed has been rejected from the upload queue for the following reason: "Fix appears incomplete, and no progress in a month. Please see https://bugs.launchpad.net/ubuntu/+source/gnome-shell/+bug/1838152/comments/5 and subsequent discussion.".

I've prepared a new package with the whole set of fixes: https://launchpad.net/ubuntu/bionic/+queue?queue_state=1&queue_text=gnome-shell

Hello Luca, or anyone else affected,

Accepted gnome-shell into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/gnome-shell/3.28.4-0ubuntu18.04.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hi Luka/@zapduke,

Can you test/verify bionic-proposed?

Thanks!

Hi Mauricio,
unfortunately I've upgraded to Focal Fossa since then and the bug is not present in this version of Ubuntu.

Performing verification for Bionic.

I want to say straight up, I haven't been able to reproduce the problem, I have tried many mechanisms, but I can not reproduce the 20 second shell hang and crash, as documented in https://github.com/GSConnect/gnome-shell-extension-gsconnect/issues/549#issuecomment-515679156

I first tried to reproduce the problem using gnome-shell 3.28.4-0ubuntu18.04.3 from -updates.

I have tried the following in both KVM and on physical hardware of a Lenovo Thinkpad X1 Laptop.

I installed the "TeaTime" gnome-shell extension version 28 from https://extensions.gnome.org/extension/604/teatime/ and used it to send multiple notifications that arrive in 5 second delays. The hang never occured.

I tried creating a draft email in evolution as suggested in https://gitlab.gnome.org/GNOME/gnome-shell/-/issues/470, and ctrl-clicked the link to spawn firefox with a notification. I did this several times, no hangs. I then tried the calendar feature in Evolution to send a notification on a new event in 5 minutes time. I did this multiple times, and never got any hangs. https://bugzilla.redhat.com/show_bug.cgi?id=1575281

I installed the gsconnect gnome-shell extension https://extensions.gnome.org/extension/1319/gsconnect/ and paired it to my android phone running the kdeconnect app.
In the settings menu on the android app, there is a "send ping" button. Press that and a notification is spawned in gnome-shell. I clicked this many times, and tried clearing notifications and being in various apps, or trying to race notifications and still could not get a hang.

Reading the commits, the gnome-shell hang is caused by fsync() being called from a main thread on persistent gnome-shell state files. I found the gnome-shell pid, and attached strace like so:

$ sudo strace -e fsync -p <PID>

On 3.28.4-0ubuntu18.04.3 each time I pressed "Send Ping" from kdeconnect, a line is printed with:

fsync(38) 0

suggesting that fsync is called every time a notification comes in.

I then enabled -proposed and installed gnome-shell 3.28.4-0ubuntu18.04.7

I retried all my initial tests, and everything worked and did not hang.

When it came to running strace against gnome-shell to view fsync calls, I noticed that fsync was no longer being called each time I pressed "Send Ping", meaning it is no longer happening per notification arrival. I clicked the clock icon to see all notifications, and a large amount of fsync calls appeared in strace output all at once, suggesting that they were put on pause and batched all at once.

I can see the behaviour change that 86a00b6 and 19e0840 has had on fsync behaviour, so the commits are doing what they say.

In my testing, I did not see any harm caused by these commits, as notifications still arrived as usual.

Again, I could not reproduce the original report of a 20 second hang, but the new package in -proposed appears to do no harm from the testing I have undertaken.

I talked to Marco about this bug, and we agreed it is okay to mark as verified. So, with some hesitation, I will mark this bug as verified.

Ok, this feels like an enough verification. Releasing!

This bug was fixed in the package gnome-shell - 3.28.4-0ubuntu18.04.7

---------------
gnome-shell (3.28.4-0ubuntu18.04.7) bionic; urgency=medium

  [ Matthew Ruffell ]
  * d/p/authPrompt-Properly-get-oVirt-service-name.patch,
    d/p/authPrompt-set-value-of-beginRequestType-to-DONT_PROVIDE_.patch,
    d/p/gdm-Introduce-vmware-credential-manager-for-pre-authentic.patch,
    d/p/gdm-Refactor-oVirt-to-a-generic-CredentialManager-interfa.patch:
    - Enable support for VMware Horizon SSO to ensure compatibility
      with the Horizon Agent (LP: #1886592)

  [ Marco Trevisan (Treviño) ]
  * debian/patches: Rename VMWare OVirt patches to use Gbp-Pq format

gnome-shell (3.28.4-0ubuntu18.04.6) bionic; urgency=medium

  - d/p/util-cache-local-GTimeZone.patch
    d/p/ShellAppCache-add-cache-to-help-keep-I-O-off-main-thread.patch,
    d/p/appDisplay-use-ShellAppCache-to-translate-folder-names.patch,
    d/p/appDisplay-use-ShellAppCache-to-access-GAppInfo.patch,
    d/p/Revert-appDisplay-use-ShellAppCache-to-access-GAppInfo.patch,
    d/p/Revert-appDisplay-use-ShellAppCache-to-translate-folder-n.patch,
    d/p/Revert-ShellAppCache-add-cache-to-help-keep-I-O-off-main-.patch,
    d/p/Revert-util-cache-local-GTimeZone.patch:
    + Include all the upstream patches to fix notification freezes
      (LP: #1838152).
  - d/p/series: Reorder so that upstream fixes come first

gnome-shell (3.28.4-0ubuntu18.04.5) bionic; urgency=medium

  * d/p/keyboardManager_apply-added-input-sources-instantly.patch:
    - Renamed to keyboardManager-Avoid-idempotent-calls-to-meta_backend_se.patch
    - Fixed DEP-3 tagging, and update them as per upstream cherry-pick
  * d/p/shell-global-Make-saving-of-persistent-state-asynchronous.patch:
    - Make saving of persistent state asynchronous (LP: #1838152)

gnome-shell (3.28.4-0ubuntu18.04.4) bionic; urgency=medium

  * d/p/keyboardManager_apply-added-input-sources-instantly.patch:
    - Make added input sources work instantly without relogin/reboot
      or other tricks (LP: #1890875).

 -- Marco Trevisan (Treviño) <email address hidden> Wed, 16 Dec 2020 01:24:41 +0100

The verification of the Stable Release Update for gnome-shell has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.