amqp 2.0 breaks the kombu driver

Related bug report: https://bugs.launchpad.net/nova/+bug/1586979

We hit this in a fresh Bifrost deployment this morning, and, so far as I can tell, it's going to affect every project that uses oslo.messaging (eg, Nova, Keystone, Ironic, ...) downstream, if someone installs that project with "pip install -r requirements.txt", because oslo.messaging specifies "amqp>=1.4.0".

It does not affect project gates because the gate environment uses upper-constraints, which currently pins "amqp===1.4.9".

Upstream kombu issue @ https://github.com/celery/kombu/issues/595

This should also be blocked in global requirements @ https://github.com/openstack/requirements/commit/14d25f43b724a96329c8db599ce2e84bad641099

This impacts all projects. Can we find out status? A lot of folks use the project requirements.txt file to install dependencies. Is this no longer a viable solution and we should use the global requirements files?

Where is this documented?

We need an update.

So https://github.com/openstack/oslo.messaging/blob/master/requirements.txt#L36 now has 'amqp<2.0,>=1.4.0' that should solve the install issue currently, as far as the kombu underlying issue; that is @ https://github.com/celery/kombu/issues/595 and seeing more folks get on that issue might be helpful (I'm not a kombu author, ha).

I don't think it's their issue tbh.

Adding my previous from the other bug I filed:

Master has the following:
https://github.com/openstack/oslo.messaging/blob/master/requirements.txt#L36

Mitaka has the following
https://github.com/openstack/oslo.messaging/blob/stable/mitaka/requirements.txt#L36

So we need to update the requirements in mitaka.

I got it. The patch for Mitaka now is here:
https://review.openstack.org/#/c/323555/

Nice!

Should we not use the requirements.txt file located in the projects and use the global-requirements moving forward? I can include that in our build process so we don't run into this kind of stuff anymore.

Ah... I reviewed the doc. The projects will be updated accordingly. Still a welcome sight :)

I marked the other bug as duplicate

keystone is also affected by this issue, i documented it here: https://bugs.launchpad.net/keystone/+bug/1587239

but i think we need a new release of oslo.messaging since we don't depend on amqp directly

new release request of oslo.messaging in https://review.openstack.org/323755

AMQP 2.0.1 is being released as well, but all they're doing is marking the message as "deprecated" and still allowing things to function.

I believe this bug should be revived now that kombu 4.0.0 with amqp 2.x support is out. I just installed amqp 2.1.1 and kombu 4.0.0 into oslo.messaging py27 tox env and executed tests. I got 6 failures: http://paste.fedoraproject.org/467518/23153147

As a first step, I proposed to uncap amqp in g-r: https://review.openstack.org/#/c/392245/

As a note, using Kombu 4 and amqp 2 I encountered the following:

Failed to consume message from queue: 'NoneType' object has no attribute 'getsockname'

I am using a standard listener with the threading executor but my guess is that oslo or whoever handles reconnects skips this and never reconnects again. This happens after some time and I believe it's due to socket timeouts or other similar issues.
Unfortunately I have no trace for now, I'm still trying to reproduce this.

And here is stack trace to help in dealing with errors:

NOTE that I am running amqp 2.0 and Kombu 4.0.2 with oslo_messaging.
This seems to be a small issue in oslo_messaging not dealing with this type of error.
It's the only issue I've encountered so far after running this for some time.

ERROR 2017-03-02 02:00:36,652 excutils:262 wrapper Unexpected exception occurred 60 time(s)... retrying.
Traceback (most recent call last):
  File "[..]/virtualenv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 250, in wrapper
    return infunc(*args, **kwargs)
  File "[..]/virtualenv/lib/python2.7/site-packages/oslo_messaging/_drivers/base.py", line 302, in _runner
    batch_size=self.batch_size, batch_timeout=self.batch_timeout)
  File "[..]/virtualenv/lib/python2.7/site-packages/oslo_messaging/_drivers/base.py", line 52, in wrapper
    msg = func(in_self, timeout=timeout_watch.leftover(True))
  File "[..]/virtualenv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 211, in poll
    self.conn.consume(timeout=timeout)
  File "[..]/virtualenv/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1098, in consume
    error_callback=_error_callback)
  File "[..]/virtualenv/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 817, in ensure
    ret, channel = autoretry_method()
  File "[..]/virtualenv/lib/python2.7/site-packages/kombu/connection.py", line 507, in _ensured
    errback and errback(exc, 0)
  File "[..]/virtualenv/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 750, in on_error
    info.update(self._get_connection_info())
  File "[..]/virtualenv/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1176, in _get_connection_info
    client_port = self.channel.connection.sock.getsockname()[1]
AttributeError: 'NoneType' object has no attribute 'getsockname'
ERROR 2017-03-02 02:00:37,658 impl_rabbit:1063 _error_callback Failed to consume message from queue: 'NoneType' object has no attribute 'getsockname'

oslo.messaging has required amqp >= 2.1 and kombu >= 4.0 since pike.
Is anyone affected by this?
Moving to incomplete. If no more activity this bug will be expired in 60 days.

[Expired for oslo.messaging because there has been no activity for 60 days.]