ceilometer error because gnocchiclient > 3.0 for stable/newton

Jesse will try to reproduce this in his environment.

Hi, some more information

Ceilometer and gnocchi from branch stable/newton (14.1.1)

Two things:

FIRST: bug from oslo.messaging using ssl

logs:
ceilometer-collector.log
2017-03-09 11:30:23.549 14913 INFO oslo.messaging._drivers.impl_rabbit [-] [3647df9b-a940-44c7-90c3-07b96fe08276] Reconnected to AMQP server on 172.29.236.8:5671 via [amqp] clientwith port 43884.
2017-03-09 11:30:23.656 14913 ERROR oslo_messaging.notify.listener [-] Fail to ack/requeue message.
2017-03-09 11:30:23.656 14913 ERROR oslo_messaging.notify.listener Traceback (most recent call last):
2017-03-09 11:30:23.656 14913 ERROR oslo_messaging.notify.listener File "/openstack/venvs/ceilometer-14.1.1/lib/python2.7/site-packages/oslo_messaging/notify/listener.py", line 177, in _process_incoming
2017-03-09 11:30:23.656 14913 ERROR oslo_messaging.notify.listener m.requeue()
2017-03-09 11:30:23.656 14913 ERROR oslo_messaging.notify.listener File "/openstack/venvs/ceilometer-14.1.1/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 129, in requeue
2017-03-09 11:30:23.656 14913 ERROR oslo_messaging.notify.listener self.message.requeue()
2017-03-09 11:30:23.656 14913 ERROR oslo_messaging.notify.listener File "/openstack/venvs/ceilometer-14.1.1/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 255, in requeue
2017-03-09 11:30:23.656 14913 ERROR oslo_messaging.notify.listener self._raw_message.requeue()
2017-03-09 11:30:23.656 14913 ERROR oslo_messaging.notify.listener File "/openstack/venvs/ceilometer-14.1.1/lib/python2.7/site-packages/kombu/message.py", line 135, in requeue
2017-03-09 11:30:23.656 14913 ERROR oslo_messaging.notify.listener self.channel.basic_reject(self.delivery_tag, requeue=True)
2017-03-09 11:30:23.656 14913 ERROR oslo_messaging.notify.listener File "/openstack/venvs/ceilometer-14.1.1/lib/python2.7/site-packages/amqp/channel.py", line 2334, in basic_reject
2017-03-09 11:30:23.656 14913 ERROR oslo_messaging.notify.listener self._send_method((60, 90), args)
2017-03-09 11:30:23.656 14913 ERROR oslo_messaging.notify.listener File "/openstack/venvs/ceilometer-14.1.1/lib/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method
2017-03-09 11:30:23.656 14913 ERROR oslo_messaging.notify.listener self.channel_id, method_sig, args, content,
2017-03-09 11:30:23.656 14913 ERROR oslo_messaging.notify.listener File "/openstack/venvs/ceilometer-14.1.1/lib/python2.7/site-packages/amqp/method_framing.py", line 221, in write_method
2017-03-09 11:30:23.656 14913 ERROR oslo_messaging.notify.listener write_frame(1, channel, payload)
2017-03-09 11:30:23.656 14913 ERROR oslo_messaging.notify.listener File "/openstack/venvs/ceilometer-14.1.1/lib/python2.7/site-packages/amqp/transport.py", line 182, in write_frame
2017-03-09 11:30:23.656 14913 ERROR oslo_messaging.notify.listener frame_type, channel, size, payload, 0xce,
2017-03-09 11:30:23.656 14913 ERROR oslo_messaging.notify.listener File "/openstack/venvs/ceilometer-14.1.1/lib/python2.7/site-packages/amqp/transport.py", line 254, in _write
2017-03-09 11:30:23.656 14913 ERROR oslo_messaging.notify.listener n = write(s...

For the ceilometer / gnocchi bug, i solved my issue.

The idea was to respect the constraint defined in the setup.cfg of the ceilometer package (ceilometer-7.0.2-py2-none-any.whl) that is:

[extras]
gnocchi =
    gnocchiclient>=2.2.0,<3.0.0 # Apache-2.0

For that, i defined an additionnal upper constraints to OpenStack's upper constraints in the ansible role repo_build.

In default/main.yml, i add:

repo_build_upper_constraints_overrides: [gnocchiclient===2.8.2]

I rebuild the repo, launch the playbook os_ceilometer and os_gnocchi.
gnocchiclient is now 2.8.2 and no more 3.1.1.

Ceilometer-collector is now functionnal and gnocchi have metrics.

I'm going to take a look at this to see if this is reproduce-able in the stable/newton branch.

I've confirmed that this is still the case in the latest newton tag, 14.2.0.

My guess is that this has something to do with the way the repo_build role builds these wheels. It's not picking up the requirements inside the setup.cfg file.

I think it only picks up requirements inside the global-requirements.txt file in the requirements project, and the requirements.txt file inside each project. A full list of files that are read as part of the py_pkgs lookup can be found here:

https://github.com/openstack/openstack-ansible-plugins/blob/stable/newton/lookup/py_pkgs.py#L46-L52

We could either convince ceilometer to change their requirements file(which I highly doubt they will do, since the gnocchi client doesn't need to be a requirement for ceilometer deployments not using it), or change our lookup plugin to include setup.cfg as a requirements file type.

Or we could append "gnocchiclient===2.8.2" to repo_build_upper_constraints_overrides whenever gnocchi is deployed.

A patch for this has been proposed here:
https://review.openstack.org/#/c/453842/

This is blocked by https://review.openstack.org/#/c/456323/2

This is ready for review here: https://review.openstack.org/#/c/453842/

Reviewed: https://review.openstack.org/453842
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_ceilometer/commit/?id=f8995cf0843663677d9b876dd06d21956e9271b3
Submitter: Jenkins
Branch: stable/newton

commit f8995cf0843663677d9b876dd06d21956e9271b3
Author: Miguel Alex Cantu <email address hidden>
Date: Wed Apr 5 19:33:40 2017 +0000

    Pin gnocchiclient to comply with optional deps

    Ceilometer pins gnocchiclient in setup.cfg to >=2.2.0,<3.0.0,
    since the pk_pkgs lookup plugin does not introspect optional
    dependencies, we must also add the pin here.

    Closes-Bug: #1670632

    Change-Id: I305fb5138182b16b59205dfbc21da32a94b42019

This issue was fixed in the openstack/openstack-ansible-os_ceilometer 14.2.2 release.

Unfortunately the patch did not resolve this problem. The workaround at this stage is to implement the following in your /etc/openstack_deploy/user_variables.yml file:

repo_build_upper_constraints_overrides:
 - gnocchiclient<3.0.0

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/462074

Reviewed: https://review.openstack.org/462074
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_ceilometer/commit/?id=2cee5acc0fe90fbe67143ca86ba78344c2106626
Submitter: Jenkins
Branch: stable/newton

commit 2cee5acc0fe90fbe67143ca86ba78344c2106626
Author: Jesse Pretorius <email address hidden>
Date: Wed May 3 11:30:15 2017 +0100

    Add known issue for gnocchiclient build failure

    Closes-Bug: #1670632
    Change-Id: I125d66e5d2f10918094c937095d8b219ae7f9735

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/466660

Reviewed: https://review.openstack.org/466660
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=3c47742c47d226d75e7865f0429b7f2493d096ea
Submitter: Jenkins
Branch: stable/newton

commit 3c47742c47d226d75e7865f0429b7f2493d096ea
Author: Jesse Pretorius <email address hidden>
Date: Mon May 22 10:59:59 2017 +0100

    Add gnocchiclient constraints to global pins

    In order to ensure that the correct version of
    gnocchiclient is built on the repo server, the
    global pin is set. This does away with the need
    for a deployer to set the upper constraint
    override themselves.

    Closes-Bug: #1670632
    Change-Id: Id7751934aaab36558115748819e2aadeb483a75e

This issue was fixed in the openstack/openstack-ansible 14.2.4 release.

This issue was fixed in the openstack/openstack-ansible-os_ceilometer 14.2.4 release.