unable to enable ceph-mgr dashboard module

Bug #1967139 reported by Aji Muhammad Arya Revaldi
50
This bug affects 8 people
Affects Status Importance Assigned to Milestone
ceph (Ubuntu)
Invalid
Undecided
Unassigned
Jammy
Invalid
Undecided
Unassigned
python-cheroot (Ubuntu)
Fix Released
Undecided
Unassigned
Jammy
Fix Released
High
Unassigned

Bug Description

[Impact]

The PEP443 in-compliant version reported by python-cheroot causes Ceph's dashboard module to be unavailable on Jammy.

[Test Plan]

$ ceph mgr module enable dashboard
Error ENOENT: module 'dashboard' reports that it cannot run on the active manager daemon: invalid version number '8.5.2+ds1' (pass --force to force enablement)

With this change, the above command should execute successfully and enable the ceph dashboard

[Where problems could occur]

Issues could occur with version detection in depending code as the version would now be reporting a PEP443 compatible version.

[Other Info]

---- Original Report ----

# lsb_release -rd
Description: Ubuntu Jammy Jellyfish (development branch)
Release: 22.04

# apt-cache policy ceph
ceph:
  Installed: 17.1.0-0ubuntu3
  Candidate: 17.1.0-0ubuntu3
  Version table:
 *** 17.1.0-0ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status

# What expected to happen?
Enable to activate ceph-mgr dashboard module

# What happened instead?
My cluster status:
# ceph -s
  cluster:
    id: 52dcb872-a44a-11ec-a9d5-00155d1a624f
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum aa-ceph01,aa-ceph02,aa-ceph03 (age 17m)
    mgr: aa-ceph03(active, since 3m), standbys: aa-ceph01, aa-ceph02
    osd: 9 osds: 9 up (since 16m), 9 in (since 16m)

  data:
    pools: 1 pools, 1 pgs
    objects: 2 objects, 449 KiB
    usage: 185 MiB used, 270 GiB / 270 GiB avail
    pgs: 1 active+clean

It shows following error:

# ceph mgr module enable dashboard
Error ENOENT: module 'dashboard' reports that it cannot run on the active manager daemon: invalid version number '8.5.2+ds1' (pass --force to force enablement)

# some logs collected from ceph-mgr
2022-03-30T13:45:11.706+0000 7fb965cf2dc0 -1 mgr[py] Module not found: 'dashboard'
2022-03-30T13:45:11.706+0000 7fb965cf2dc0 -1 mgr[py] Traceback (most recent call last):
  File "/usr/share/ceph/mgr/dashboard/__init__.py", line 52, in <module>
    from .module import Module, StandbyModule # noqa: F401
  File "/usr/share/ceph/mgr/dashboard/module.py", line 49, in <module>
    patch_cherrypy(cherrypy.__version__)
  File "/usr/share/ceph/mgr/dashboard/cherrypy_backports.py", line 197, in patch_cherrypy
    accept_socket_error_0(v)
  File "/usr/share/ceph/mgr/dashboard/cherrypy_backports.py", line 124, in accept_socket_error_0
    if v < StrictVersion("9.0.0") or cheroot_version < StrictVersion("6.5.5"):
  File "/lib/python3.10/distutils/version.py", line 64, in __gt__
    c = self._cmp(other)
  File "/lib/python3.10/distutils/version.py", line 168, in _cmp
    other = StrictVersion(other)
  File "/lib/python3.10/distutils/version.py", line 40, in __init__
    self.parse(vstring)
  File "/lib/python3.10/distutils/version.py", line 137, in parse
    raise ValueError("invalid version number '%s'" % vstring)
ValueError: invalid version number '8.5.2+ds1'

2022-03-30T13:45:11.706+0000 7fb965cf2dc0 -1 mgr[py] Class not found in module 'dashboard'
2022-03-30T13:45:11.706+0000 7fb965cf2dc0 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory
2022-03-30T13:45:11.706+0000 7fb965cf2dc0 1 mgr[py] Loading python module 'progress'
2022-03-30T13:45:11.838+0000 7fb965cf2dc0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member
2022-03-30T13:45:11.838+0000 7fb965cf2dc0 1 mgr[py] Loading python module 'localpool'
2022-03-30T13:45:11.958+0000 7fb965cf2dc0 1 mgr[py] Loading python module 'osd_perf_query'
2022-03-30T13:45:12.134+0000 7fb965cf2dc0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member
2022-03-30T13:45:12.134+0000 7fb965cf2dc0 1 mgr[py] Loading python module 'telemetry'
2022-03-30T13:45:12.442+0000 7fb965cf2dc0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member
2022-03-30T13:45:12.442+0000 7fb965cf2dc0 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ceph (Ubuntu):
status: New → Confirmed
Revision history for this message
Heinrich Schuchardt (xypron) wrote :

Package python3-cheroot has version 8.5.2+ds1-1ubuntu.

src/pybind/mgr/dashboard/cherrypy_backports.py:124:
from distutils.version import StrictVersion
if v < StrictVersion("9.0.0") or cheroot_version < StrictVersion("6.5.5"):

>>> StrictVersion("8.5.2+ds1-1ubuntu")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.10/distutils/version.py", line 40, in __init__
    self.parse(vstring)
  File "/usr/lib/python3.10/distutils/version.py", line 137, in parse
    raise ValueError("invalid version number '%s'" % vstring)
ValueError: invalid version number '8.5.2+ds1-1ubuntu'

The regular expression to check version numbers is
r'(?x)^(\d+)\.(\d+)(\.(\d+))?([ab](\d+))?$'

Allowable package versions are defined in PEP440 (https://peps.python.org/pep-0440/) as
[N!]N(.N)*[{a|b|rc}N][.postN][.devN]

So both the test in the deprecated distutils package as well as the version number reported by package python3-cheroot is wrong.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in python-cheroot (Ubuntu):
status: New → Confirmed
Revision history for this message
Heinrich Schuchardt (xypron) wrote :
Revision history for this message
Heinrich Schuchardt (xypron) wrote :

This line needs to be corrected in python-cheroot:

cheroot/__init__.py:13:
__version__ = pkg_resources.get_distribution('cheroot').version

Revision history for this message
Heinrich Schuchardt (xypron) wrote (last edit ):
Revision history for this message
Heinrich Schuchardt (xypron) wrote :

Debian published python3-cheroot (8.6.0+ds1-3) in the sid repository to resolve the issue.

Revision history for this message
Heinrich Schuchardt (xypron) wrote :

A sync request has been created: LP #1973737

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-cheroot - 8.6.0+ds1-3ubuntu1

---------------
python-cheroot (8.6.0+ds1-3ubuntu1) kinetic; urgency=medium

  * Merge from Debian unstable (LP: #1967139), remaining changes
    + d/rules: Enable test suite, skip small subset of tests
      that rely on unpackaged dependencies, unset proxy
      variable to support SSL tests.
    + d/rules: Remove dh_auto_test no-op override.
  * dropped:
    + d/p/0*.patch: dropped as included in new upstream
  * added:
    + d/p/remove_pypytools_dependency.patch to work around test dependency

python-cheroot (8.6.0+ds1-3) unstable; urgency=medium

  * Add a patch to declare a PEP440-compatible version (Closes: #1010894).

python-cheroot (8.6.0+ds1-2) unstable; urgency=medium

  * Drop excess comma from Maintainer field.

python-cheroot (8.6.0+ds1-1) unstable; urgency=medium

  * Fix d/watch (again).
  * New upstream release.

python-cheroot (8.5.2+ds1-3) unstable; urgency=medium

  [ Debian Janitor ]
  * Remove constraints unnecessary since buster:
    + Build-Depends: Drop versioned constraint on dh-python.

  [ Sandro Tosi ]
  * debian/control
    - use proper team name and email address

python-cheroot (8.5.2+ds1-2) unstable; urgency=medium

  * Fix d/watch.
  * Bump standards-version to 4.6.0.

 -- Heinrich Schuchardt <email address hidden> Wed, 18 May 2022 10:47:40 +0000

Changed in python-cheroot (Ubuntu):
status: Confirmed → Fix Released
description: updated
Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

This debdiff cherry-picks Debian's fix for the specific issue impacting usage at Jammy

tags: added: patch
James Page (james-page)
Changed in python-cheroot (Ubuntu Jammy):
importance: Undecided → High
Changed in ceph (Ubuntu):
status: Confirmed → Invalid
Changed in ceph (Ubuntu Jammy):
status: New → Invalid
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in python-cheroot (Ubuntu Jammy):
status: New → Confirmed
Revision history for this message
Steve Langasek (vorlon) wrote : Please test proposed package

Hello Aji, or anyone else affected,

Accepted python-cheroot into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-cheroot/8.5.2+ds1-1ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in python-cheroot (Ubuntu Jammy):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-jammy
Revision history for this message
Vern Hart (vern) wrote :

Hi Steve,

I tried this out and there were no python updates after configuring the jammy-proposed repo.

Then I realized, jammy doesn't have python-cheroot, it has python3-cheroot. But there are no updates for that package:

$ apt-cache policy python-cheroot
N: Unable to locate package python-cheroot
$ apt-cache policy python3-cheroot
python3-cheroot:
  Installed: 8.5.2+ds1-1ubuntu3
  Candidate: 8.5.2+ds1-1ubuntu3
  Version table:
 *** 8.5.2+ds1-1ubuntu3 500
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status

Just to confirm I have the proposed repo configured, I see there's an update to the ua tools available:

$ apt-cache policy ubuntu-advantage-tools
ubuntu-advantage-tools:
  Installed: 27.9~22.04.1
  Candidate: 27.10.1~22.04.1
  Version table:
     27.10.1~22.04.1 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
 *** 27.9~22.04.1 500
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     27.7~22.04.1 500
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 Packages

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (python-cheroot/8.5.2+ds1-1ubuntu3.1)

All autopkgtests for the newly accepted python-cheroot (8.5.2+ds1-1ubuntu3.1) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

translate-toolkit/3.6.0-2 (ppc64el, arm64, armhf, s390x, amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#python-cheroot

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

I've confirmed that the dashboard does deploy correctly on Jammy with this update enabled:

# apt-cache policy python3-cheroot
python3-cheroot:
  Installed: 8.5.2+ds1-1ubuntu3.1
  Candidate: 8.5.2+ds1-1ubuntu3.1
  Version table:
 *** 8.5.2+ds1-1ubuntu3.1 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     8.5.2+ds1-1ubuntu3 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu jammy/main amd64 Packages

Looking at the autopkgtest failures, it looks like it may have been from the testbed being out of date

tags: added: verification-done verification-done-jammy
removed: verification-needed verification-needed-jammy
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Thank you for the validation. I have run two of the arches with migration-reference to see if those are failing in the updates pocket already. We can then hint them if those are existing failures.

Did you manage to take a look at those already?

Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

I hadn't looked at the existing versions in -updates; I did look through the autopkgtest failures and they seem to be due to pyparsing being pulled from proposed (https://bugs.launchpad.net/ubuntu/+source/translate-toolkit/+bug/1969132)

Revision history for this message
Eric Gelinas (egelinas) wrote :

Fixed with the -proposed package. The issue happened with a bare-metal cluster running ceph (focal, pacific) where I updated the hosts to jammy.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-cheroot - 8.5.2+ds1-1ubuntu3.1

---------------
python-cheroot (8.5.2+ds1-1ubuntu3.1) jammy; urgency=medium

  * d/p/pep440_version_number.patch: Cherry-pick Debian patch to produce
    pep440 compliant versions (LP: #1967139).

 -- Chris MacNaughton <email address hidden> Fri, 22 Jul 2022 16:45:15 +0100

Changed in python-cheroot (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for python-cheroot has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Alan Baghumian (alanbach) wrote :

Hello Ceph Wizards!

I was just deploying Ceph dashboard on my Pacific and Quincy clusters, Quincy went through just fine, however the deployment on Pacific failed with this similar issue:

2023-07-22T16:34:38.664+0000 7f7241644440 -1 mgr[py] Class not found in module 'dashboard'
2023-07-22T16:34:38.664+0000 7f7241644440 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory
2023-07-22T16:34:38.664+0000 7f7241644440 1 mgr[py] Loading python module 'devicehealth'
2023-07-22T16:34:38.744+0000 7f7241644440 1 mgr[py] Loading python module 'influx'

root@juju-b096f0-104-lxd-0:/var/log/ceph# apt-cache policy python3-cheroot
python3-cheroot:
  Installed: (none)
  Candidate: 8.3.0+ds-1
  Version table:
     8.3.0+ds-1 500
        500 http://lds.int.hrizn.cloud/repository/standalone/ubuntu focal/universe amd64 Packages

$ rmadison python3-cheroot

 python3-cheroot | 8.3.0+ds-1 | focal/universe | all
 python3-cheroot | 8.5.2+ds1-1ubuntu3 | jammy | all
 python3-cheroot | 8.5.2+ds1-1ubuntu3.1 | jammy-updates | all

Do we have a plan to fix this for Pacific?

Best,
Alan

Revision history for this message
Alan Baghumian (alanbach) wrote :

Looking deeper into this, on the Focal/Pacific machine:

$ dpkg -l | grep dash
ii ceph-mgr-dashboard 15.2.17-0ubuntu0.20.04.4 all dashboard module for ceph-mgr

While compared to Jammy/Quincy:

$ dpkg -l | grep dashboard
ii ceph-mgr-dashboard 17.2.5-0ubuntu0.22.04.3 all dashboard module for ceph-mgr

This seems to be a package discrepancy issue, the Pacific packages are available on UCA.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.