Flatpak related refs are not installed when using GNOME Software on Ubuntu 18.04

> This issue should be already fixed in GNOME Software upstream so I believe that this is a downstream Ubuntu issue.

Why do you think this?

Upstream issue (but I think this is an Ubuntu issue, not upstream one): https://gitlab.gnome.org/GNOME/gnome-software/issues/324

@jbicha: It should be fixed in: https://gitlab.gnome.org/GNOME/gnome-software/merge_requests/2

> It should be fixed in: https://gitlab.gnome.org/GNOME/gnome-software/merge_requests/2

That particular commit is already in gnome-software 3.27.92.

So, I have just tested this on the latest Fedora Rawhide and experienced the same issue as on Ubuntu 18.04 so it looks like that this is really an upstream issue.

https://gitlab.gnome.org/GNOME/gnome-software/merge_requests/48

Here is a Merge Request that fixes the issue. It will hopefully be merged into the stable (gnome-3-28) branch in upcoming weeks.

The issue was fixed in the current master (gnome-3-30) branch. It still needs to be fixed in the older branch that Ubuntu 18.04 uses.

https://gitlab.gnome.org/GNOME/gnome-software/merge_requests/58

Here is the actual Merge Request for the gnome-3-28 that is still waiting to be merged.

The issue is unfortunately still not fully fixed in master (see https://gitlab.gnome.org/GNOME/gnome-software/issues/235#note_280405) and is not fixed at all in the older branch (!58 was still not merged).

Curiously I cannot find "GNOME Web" when searching flathub from my cosmic VM. However I installed org.gnome.Eolie from GNOME Software and observed that it is started with Adwaita and an English locale, whereas the system theme is Ambiance and locale is French.
When installing from the CLI ("flatpak install flathub org.gnome.Eolie"), the theme is Ambiance as expected. The language remains English, maybe a separate issue?

@osomon, hey o/ :-)

GNOME Web looks like it has been archived on Flathub https://github.com/flathub/org.gnome.Epiphany and removed from the store for now. I believe this was due to security issues, which will hopefully be solved with the new freedesktop runtime.

Eolie, looks like there are other people stating that the languages aren't working, so might just be that the translations aren't available for the app. (probably discuss with upstream)

Yes runtime extensions are not installed automatically with gnome software 3.28. Eg themes, extra drivers, codecs etc. This works with the CLI as it had extra logic to deal with automatically installing such extensions. Now with newer versions of libflatpak this logic is included as a "FlatpakTransaction API", GNOME Software 3.30 has a new flatpak plugin and will make use of this - resolving the issue for cosmic onwards.

However, there was also code submitted upstream which added this logic to the old GNOME Software flatpak plugin to install these extensions automatically - resolving the issue for 3.28 users. If this is approved and lands into a 3.28 point release, I'd be happy to help go through the SRU process to include such a change :-) (ping me on IRC if you want to discuss anything).

Great, thanks Andrew for the detailed heads-up!
That sounds like a good SRU candidate if it lands in a 3.28 point release indeed. Can you point me to the upstream branch that has those changes?

@osomon, sure :-)

This was the original PR against master https://gitlab.gnome.org/GNOME/gnome-software/merge_requests/48 but i believe that has been superseded by https://gitlab.gnome.org/GNOME/gnome-software/merge_requests/58 for the stable series 3.28. (jrocha from Endless was working on the code, he hangs around in #gnome-software on GIMPnet/GNOME IRC - i'm sure he'd be happy to answer any questions/help).

As as noted in the earlier comments, this is the bug that is tracking it upstream https://gitlab.gnome.org/GNOME/gnome-software/issues/235

The question is if the MR for gnome-3-28 will be ever merged. If you want Flatpak support in GS to be working properly on Ubuntu 18.04, please show your support and let Richard Hughes (or another GS dev) know. Either via comment in the MR (https://gitlab.gnome.org/GNOME/gnome-software/merge_requests/58) or by contacting the dev directly. Thanks!

The code from MR !58 should be working without any issues and is already used in Endless OS.

The other possible solution is to create a downstream patch for Ubuntu 18.04 using the code from MR !58.

So the MR !58 got closed and upstream doesn't seem to be interested in maintaining the Flatpak code in the gnome-3-28 branch anymore. They are focusing on the new code that is using the FlatpakTransaction API (and most likely won't be backported into the older GS branch) instead.

Here's a patch from the MR commits that can be applied downstream and fixes this issue.

@ahayzen Are you interested in doing a SRU for the Ubuntu 18.04 GS with this patch applied?

@Asciiwolf, Personally I would prefer such a large change (229 additions and 37 deletions across 28 files) to be approved by upstream - especially as this is going into a LTS release. As it is not always clear what other behaviour this may change or break.

I understand that this is the patch that Endless are using in their distro, but I would need to understand how the flatpak plugin works before I would be able to make a decision.

Ultimately to get this change into Ubuntu it would need to be approved by Robert Ancell, so maybe Robert is the one who can advise if we should require upstream approval or if it is OK to distro patch. As this is quite an important fix for trying to use flatpak with a GUI in bionic.

I am happy to help with the SRU process and submitting the patches to the Ubuntu gnome-software git etc, I just feel I don't have enough knowledge of gnome-software's code to state if this is an OK change to carry.

(... or maybe we just need to wait until gnome-software is a flatpak/snap/appimage ;-) then we can magically get 3.30 :-) )

(Also if this does get submitted I have a test plan for flatpak itself that includes UI tests for gnome-software, so this would help in doing the QA part)

@ahayzen I fully agree, let's wait for Robert's opinion about this. Also, please ignore the previous patch, the commits are in wrong order and even if I fix the order, it fails to apply for some reason. I will send the individual patches.

Here are the correct patches that apply without any issues on the latest gnome-3-28 branch.

The attachment "gs-add-related-refs-support.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

Any progress?

By the way, GS developer told me that they didn't want to merge it because they don't have enough people and time to further maintain it. So if someone from Canonical would be willing to maintain it in the gnome-3-28 branch, there should not be any problem having the patches upstream.

(https://gitlab.gnome.org/GNOME/gnome-software/merge_requests/58#note_328627)

@AsciiWolf - if you want to propose this in 18.04 the best way is to do a MR to the ubuntu-3-28 branch on https://gitlab.gnome.org/Community/Ubuntu/gnome-software - that's where we get the patches for the .deb.

A brief look at the changes seems like it should be fine. I'd drop the test changes unless they break the build otherwise they'll make the patch enormous.

@Robert Ancell Done. See: https://gitlab.gnome.org/Community/Ubuntu/gnome-software/merge_requests/8

I believe this issue is resolved in cosmic and is only an issue in bionic. Normally such issues are marked as "Fix Released" for "ubuntu" and then proposed to the series bionic. I don't have the permissions to do this, Robert do you normally work with this workflow ?

Also I will test the normal flatpak manual test plan (which includes gnome-software UI tests) against this change once it is in -proposed to confirm this doesn't regress anything. And I am happy to help with any parts of the SRU process :-)

@Robert, will you be generating a debdiff for the gnome-3-28 branch when it's ready? (I see you are making other changes and I'm not familiar how you are taking the git repo and turning the changes into debian patches).

Also I see that 3.28.1-0ubuntu4.18.04.4 [0] is currently stuck [1] in bionic-proposed as bug 1756379 has not been verified. I guess that this should be done before pushing the next release through :-)

0 - https://launchpad.net/ubuntu/+source/gnome-software/3.28.1-0ubuntu4.18.04.4
1 - http://people.canonical.com/~ubuntu-archive/pending-sru.html

@Andrew

The debian patches come from the ubuntu-3-28 branch [1] and are committed to the packaging branch [2]. Yeah, it's a bit complicated...

[1] https://gitlab.gnome.org/Community/Ubuntu/gnome-software/tree/ubuntu-3-28
[2] https://code.launchpad.net/~ubuntu-desktop/ubuntu/+source/gnome-software/+git/gnome-software/+ref/ubuntu/bionic

Hello AsciiWolf, or anyone else affected,

Accepted gnome-software into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/gnome-software/3.28.1-0ubuntu4.18.04.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

The good news is that this now correctly installs themes when installing an app.

Unfortunately for me test "2.4 Update an app in GNOME Software" from the flatpak test plan [0], fails for me with the package from proposed.

After reverting to an older commit hash, then refreshing gnome-software it shows the update. But when I click "Update" the UI shows progress, hides the item and then it reappears again. Furthermore using the CLI you can see that the package hasn't been updated. So this change appears to be breaking an update.

When running from the console I get "01:38:32:0760 Gs tried overwriting org.gnome.Recipes.desktop key flatpak::Commit from 6d5f7c35ff1f3fd449b2789324843a89140257bb3cdca468f83a65162b177fa0 to 1ac37481845c65857a64e8c0680bae137b0b56faaf6f11f3f0d415a3205bc1c2" and if i run in verbose mode I get this https://pastebin.ubuntu.com/p/r8pWhpmcG7/

Note flipping back to 3.28.1-0ubuntu4.18.04.4 it works again.

Therefore I am going to have to mark this as verification-failed-bionic for now.

$ apt-cache policy gnome-software-plugin-flatpak
gnome-software-plugin-flatpak:
  Installed: 3.28.1-0ubuntu4.18.04.5
  Candidate: 3.28.1-0ubuntu4.18.04.5
  Version table:
 *** 3.28.1-0ubuntu4.18.04.5 500
        500 http://gb.archive.ubuntu.com/ubuntu bionic-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     3.28.1-0ubuntu4.18.04.4 500
        500 http://gb.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages
     3.28.1-0ubuntu4 500
        500 http://gb.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages

0 - https://wiki.ubuntu.com/Process/Merges/TestPlan/flatpak

You're right. I can confirm that the runtime related refs are now installed without any issue, however the new code seems to have problems with downgrades. :-( Unfortunately, I am not a good in C so I cannot fix it myself and Joaquim Rocha doesn't work on GNOME Software anymore.

Other possible solution to the related refs issue is to switch to the new GNOME Software 3.30.* version in Bionic instead of using an older one. This solution is recommended by GS devs because the new GS has many fixes (not only the Flatpak ones).

It would be great if some GS dev from Canonical could fix the current patches.

Proper Flatpak support is really important to end-users as Flatpak is becoming a standard in cross-distro application distribution.

I've re-uploaded the SRU with these changes removed. If someone can re-propose fixed patches and/or additional patches to resolve this then we can consider it for another SRU.

Hello AsciiWolf, or anyone else affected,

Accepted gnome-software into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/gnome-software/3.28.1-0ubuntu4.18.04.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Marking as verification-done; the change that regressed has been reverted, so this bug isn't actually fixed, but it also hasn't introduced the downgrade problem.

This bug was fixed in the package gnome-software - 3.28.1-0ubuntu4.18.04.8

---------------
gnome-software (3.28.1-0ubuntu4.18.04.8) bionic; urgency=medium

  * Re-released to generate correct .changes file

gnome-software (3.28.1-0ubuntu4.18.04.7) bionic; urgency=medium

  * debian/rules:
  * debian/patches/0001-packagekit-Add-configure-option-to-use-autoremove-fe.patch:
    - Build with PackageKit autoremove support (LP: #1552792)
  * debian/control:
  * debian/patches/0023-snap-Don-t-treat-auth-cancellation-as-an-error.patch:
    - Stop cancelling snapd authorization triggers error notification
      (LP: #1785240)

gnome-software (3.28.1-0ubuntu4.18.04.6) bionic; urgency=medium

  * debian/patches/0023-Revert-Revert-flatpak-Use-list-of-related-apps-for-i.patch:
  * debian/patches/0024-flatpak-Check-if-a-related-app-is-installed-even-whe.patch:
  * debian/patches/0025-flatpak-Include-related-refs-to-the-app-s-runtime-on.patch:
    - Disable the Flatpak changes - SRU testing showed they broke downgrades

gnome-software (3.28.1-0ubuntu4.18.04.5) bionic; urgency=medium

  * debian/patches/0023-Revert-Revert-flatpak-Use-list-of-related-apps-for-i.patch:
  * debian/patches/0024-flatpak-Check-if-a-related-app-is-installed-even-whe.patch:
  * debian/patches/0025-flatpak-Include-related-refs-to-the-app-s-runtime-on.patch:
    - Pull related flatpak refs (LP: #1754864)
  * debian/patches/0003-Sort-snaps-before-other-apps.patch:
    - Fix snap search result ordering (LP: #1798228)
  * debian/patches/0001-Set-reboot-notifications-as-urgent-to-avoid-them-tim.patch:
    - Stop reboot notification from timing out (LP: #1719797)
  * debian/patches/0001-lib-Allow-opening-CAB-files-that-resolve-more-than-o.patch:
    - Support composite CAB files (LP: #1798470)

 -- Robert Ancell <email address hidden> Wed, 21 Nov 2018 12:29:27 +1300

The verification of the Stable Release Update for gnome-software has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Was this issue really fixed? I thought the patches were reverted.

Settings as wontfix for bionic atm, it doesn't sound like anyone is working on fixing in for that serie, that can be changed back if someone cares enough about the issue and works on a SRU though

Thanks, for the record, I previously discussed this issue with upstream and about them including such a change in a point release. But they didn't want to commit to making and supporting major changes to the old flatpak plugin (from gnome-software 3.30 the plugin was rewritten and the issue doesn't occur). I personally don't have any knowledge of the gnome-software code base, so would not feel confident in fixing the code and ensuring it doesn't break anything else.

As stated before, if anyone with knowledge is able to either get a merge approved by upstream for the 3.28 series or has enough knowledge of the code base to be confident in their changes, I would be happy in helping with the SRU verification part of the process.

This bug was mentioned in the Ubuntu 18.04.2 release notes as "fixed", however this issue was not fixed in Ubuntu 18.04 and probably won't be.

As far as I know, Joaquim Rocha now works again on GNOME Software at Endless, so we could ask him to provide a better patches, but I don't think the patches will be good enough to include them in Ubuntu (because, unfortunately, even the upstream Flatpak plugin quality was bad in the pre-3.30 GNOME Software).