ceph-osd fails to start with ProtectClock=true

Bug #1925347 reported by James Page
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Fix Released
Undecided
Unassigned
Wallaby
Fix Released
Undecided
Unassigned
Xena
Fix Released
Undecided
Unassigned
ceph (Ubuntu)
Fix Released
Critical
James Page
Hirsute
Fix Released
Critical
James Page

Bug Description

[Impact]
ceph-osd daemon is unable to start on fresh installs or post upgrade

[Test Case]
Deploy ceph with OSD units
ceph-osd will fail to start

[Regression Risk]
Reverts ceph to its pre-pacific configuration for this option.

[Original Bug Report]
Ceph Pacific
Ubuntu 20.04 (but 21.04 has the same change)

Upstream pull: https://github.com/ceph/ceph/pull/40845
Upstream issue: https://tracker.ceph.com/issues/50347

Upstream enabled ProtectClock=true as part of a change to reduce permissions needed for daemons - however this has the side effect of disabling access to block devices, which is needed for the ceph-osd daemon (at least).

CVE References

James Page (james-page)
description: updated
Changed in ceph (Ubuntu Hirsute):
status: New → In Progress
importance: Undecided → Critical
assignee: nobody → James Page (james-page)
Revision history for this message
James Page (james-page) wrote :

Test packages in:

  https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3535

for any impacted Hirsute users (however don't expect to many of those and the backport to the Wallaby UCA has picked this change up and a transient patch for the backporter).

Revision history for this message
Timo Aaltonen (tjaalton) wrote : Please test proposed package

Hello James, or anyone else affected,

Accepted ceph into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/16.2.1-0ubuntu0.21.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in ceph (Ubuntu Hirsute):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-hirsute
Revision history for this message
Aurelien Lourot (aurelien-lourot) wrote :

This fixed the bug for me, thanks! To reproduce/validate I deploy Ceph Pacific on Hirsute s390x machines using this Juju bundle. [0]

If I leave the software source as `distro`, I hit the bug, i.e. ceph-osd fails to start with:
bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (1) Operation not permitted

If I then change the software source to `distro-proposed`, this ends up installing ceph-osd 16.2.1-0ubuntu0.21.04.1 and Ceph works as expected.

[0] https://github.com/openstack-charmers/openstack-bundles/blob/master/development/ceph-base-hirsute-pacific/bundle.yaml

tags: added: verification-done-hirsute
removed: verification-needed-hirsute
Revision history for this message
James Page (james-page) wrote :

Confirmed - package in proposed resolves the issue and allows the ceph-osd daemons to start correctly - thanks for verifying Aurelien.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ceph - 16.2.1-0ubuntu1

---------------
ceph (16.2.1-0ubuntu1) impish; urgency=medium

  [ Chris MacNaughton ]
  * d/ceph-base.install: Remove ceph-deploy man page installation
    (LP: #1892448).

  [ James Page ]
  * SECURITY UPDATE: New upstream point release (LP: #1925322):
    - CVE-2021-20288
  * d/rules: remove temporary build objects after install to avoid
    running out of disk space during package builds.
  * d/p/bug1925347.patch: Cherry pick fix to revert ProtectClock
    permissions change in systemd configurations which prevents the
    ceph-osd process from starting (LP: #1925347).

 -- James Page <email address hidden> Tue, 04 May 2021 19:21:24 +0100

Changed in ceph (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ceph - 16.2.1-0ubuntu0.21.04.1

---------------
ceph (16.2.1-0ubuntu0.21.04.1) hirsute-proposed; urgency=medium

  * SECURITY UPDATE: New upstream point release (LP: #1925322):
    - CVE-2021-20288
  * d/rules: remove temporary build objects after install to avoid
    running out of disk space during package builds.
  * d/p/bug1925347.patch: Cherry pick fix to revert ProtectClock
    permissions change in systemd configurations which prevents the
    ceph-osd process from starting (LP: #1925347).

 -- James Page <email address hidden> Thu, 22 Apr 2021 10:21:35 +0100

Changed in ceph (Ubuntu Hirsute):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for ceph has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.