bootstrap_gnocchi error log: ImportError: No module named 'rados' nor 'cradox'

Found the issue that the gnocchi deb package on Ubuntu will install gnocchi-api & gnocchi-upgrade which using python3 as python execution application.
To fix this, need to build the image which installed require python packages in Python3 method. But it caused another issue from apache2 that port 8041 already in use after launch gnocchi-api and keep restarting container.

So there're two approaches to fix:
1. Build the image with require packages in python3, then check why default port has already in use.
2. Change gnocchi binary to use python2 or virtual env.

Is this something that you can fix Eddie?

Hi Mark,

Sorry but I'm too busy to keep trying fix this issue in present. I believe it caused by the Ubuntu repository has updated gnocchi binary that required python3 dependency in Bionic.
I think using python3 packages is the best way to fix, but I still have no clue and wonder know why apache2 said the port already in use during gnocchi deployment. And I can't access into container to see what's going on because it keep restarting.

Do you have any possibilities about port conflict issue? It's really hard to think this happen caused by install python3 dependencies instead of python2's.

Eddie, sometimes the distro packages install their own config files which include the port, and conflict with those generated by kolla-ansible. Could you look inside the container image for that?

Hi Mark,

I done the comparison between Docker Hub binary image and the binary image built by Kolla-build. Both gnocchi-api images has already got gnocchi-api.conf at /etc/apache2/sites-available inside, and this config file already set default port number of gnocchi.

Perhaps that's a root cause but still need to do the test based on re-build image. I'll try to add the command to remove this file in the Dockerfile, the rebuild is necessary because python3 dependencies is known required.

I finally tested the gnocchi-api image with gnocchi-api.conf deleted. The result is clear. The gnocchi-api has successfully running after its bootstrap. And no error occur during ceilometer deployment.

For modification, I do the code comparison between rocky and stein. Let it install python3 packages when building the rocky image, and delete the file I said in gnocchi-api Dockerfile. Also I modified wsgi-gnocchi.conf.j2 in Kolla-ansible so that to make sure it running with python3 based.

One more thing. I also found the aodh-api has a same issue about port conflict. And found that the distro package also create its own aodh-api conf file in the same direction.

Since the images from Docker Hub may all built by Kolla-build, I think these issue need to be fix. I'll upload my diff text to Pastebin.

But I have another question. Since we already found port conflict issue on gnocchi-api binary and aodh-api binary, is that better to replace the distro config rather than delete? Like just let wsgi-gnocchi.conf to replace /etc/apache2/sites-available/gnocchi-api.conf. If so, we can just change the dest for wsgi-gnocchi.conf in gnocchi-api.json.j2

Here's my modification. Already using diff to comparison.

Kolla gnocchi deploy fix (rocky) - wsgi-gnocchi -> https://pastebin.com/Sbjz0bkb
Kolla gnocchi deploy fix (rocky) - Dockerfile-gnocchi_api -> https://pastebin.com/4Z9rvbDn
Kolla gnocchi deploy fix (rocky) - Dockerfile-gnocchi_api -> https://pastebin.com/kpzx4Tj8
Kolla gnocchi deploy fix (rocky) - kolla-build macros -> https://pastebin.com/M6gV0W1L

Thanks for investigating Eddie, will you be pushing patches with these fixes?

About this:

> But I have another question. Since we already found port conflict issue on gnocchi-api binary and aodh-api binary, is that better to replace the distro config rather than delete? Like just let wsgi-gnocchi.conf to replace /etc/apache2/sites-available/gnocchi-api.conf. If so, we can just change the dest for wsgi-gnocchi.conf in gnocchi-api.json.j2

the config files might be named differently in different distros, so makes sense to just start with nothing, agree?

> The config files might be named differently in different distros, so makes sense to just start with nothing, agree?

Hmm, sounds make sense. Let's keep the deletion instead of replacement.

I will push. But I'm not sure where to push and how. Will appreciate tell me how to.

Related fix proposed to branch: master
Review: https://review.opendev.org/670832

Reviewed: https://review.opendev.org/670425
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=330fc3698df0bddeea0e56f6c75623ff4dd88414
Submitter: Zuul
Branch: stable/rocky

commit 330fc3698df0bddeea0e56f6c75623ff4dd88414
Author: Eddie Yen <email address hidden>
Date: Fri Jul 12 01:29:10 2019 +0000

    Change to use Python3 dependencies for gnocchi on Ubuntu binary.

    Since Ubuntu Bionic distro has updated gnocchi binary that require
    python3, we need to install its python dependences based on python3.
    Especially 'cradox', this module is necessary for gnocchi to access Ceph.
    Wrong package version install will cause error during container
    bootstrapping.

    Related-Bug: 1832306

    Change-Id: I639564ce35d72ad9d16c191620a53481ef5920db

Reviewed: https://review.opendev.org/670832
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=5eb780d0b0d12186c57b828320c3421b98b97840
Submitter: Zuul
Branch: master

commit 5eb780d0b0d12186c57b828320c3421b98b97840
Author: Eddie Yen <email address hidden>
Date: Mon Jul 15 13:20:39 2019 +0000

    Delete default gnocchi & aodh wsgi conf in Ubuntu binary

    Ubuntu distro package will create default wsgi conf in apache
    during gnocchi-api and aodh-api installation. We need to delete
    them since we only use the conf that pushed by kolla-ansible.

    Change-Id: Ib31e733eba12deebdf469c306dfba4191a1b89f5
    Related-Bug: 1832306
    Related-Bug: 1836520

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/676616

Reviewed: https://review.opendev.org/676616
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=cc6694dbc8907bad5085f9129fccd5b682e72b66
Submitter: Zuul
Branch: stable/stein

commit cc6694dbc8907bad5085f9129fccd5b682e72b66
Author: Eddie Yen <email address hidden>
Date: Mon Jul 15 13:20:39 2019 +0000

    Delete default gnocchi & aodh wsgi conf in Ubuntu binary

    Ubuntu distro package will create default wsgi conf in apache
    during gnocchi-api and aodh-api installation. We need to delete
    them since we only use the conf that pushed by kolla-ansible.

    Change-Id: Ib31e733eba12deebdf469c306dfba4191a1b89f5
    Related-Bug: 1832306
    Related-Bug: 1836520
    (cherry picked from commit 5eb780d0b0d12186c57b828320c3421b98b97840)

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/679617

Reviewed: https://review.opendev.org/679617
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=69e46f78aff21ecad5f7793fb9aa79b56da87747
Submitter: Zuul
Branch: stable/rocky

commit 69e46f78aff21ecad5f7793fb9aa79b56da87747
Author: Eddie Yen <email address hidden>
Date: Mon Jul 15 13:20:39 2019 +0000

    Delete default gnocchi & aodh wsgi conf in Ubuntu binary

    Ubuntu distro package will create default wsgi conf in apache
    during gnocchi-api and aodh-api installation. We need to delete
    them since we only use the conf that pushed by kolla-ansible.

    Change-Id: Ib31e733eba12deebdf469c306dfba4191a1b89f5
    Related-Bug: 1832306
    Related-Bug: 1836520
    (cherry picked from commit 5eb780d0b0d12186c57b828320c3421b98b97840)
    (cherry picked from commit cc6694dbc8907bad5085f9129fccd5b682e72b66)