oslo.messaging

  1. Bug #1545002
  2. Comment #2

Comment 2 for bug 1545002

Revision history for this message
Matt Riedemann (mriedem) wrote : Re: Grenade Failure - ip floating remove

The failure in n-api is here:

http://logs.openstack.org/12/278012/3/check/gate-grenade-dsvm/ac87c30/logs/new/screen-n-api.txt.gz?level=TRACE#_2016-02-12_10_32_05_378

That timeout is at 10:32:05. Around the same time in the n-net logs we have this:

http://logs.openstack.org/12/278012/3/check/gate-grenade-dsvm/ac87c30/logs/new/screen-n-net.txt.gz#_2016-02-12_10_31_05_380

2016-02-12 10:31:05.380 DEBUG oslo_concurrency.lockutils [req-b1f17c4c-b814-4a21-9209-13db7915370f cinder_grenade cinder_grenade] Lock "172.24.5.2" acquired by "nova.network.floating_ips.do_disassociate" :: waited 0.000s inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:270
2016-02-12 10:32:06.595 22979 DEBUG oslo.messaging._drivers.impl_rabbit [-] Received recoverable error from kombu: on_error /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:654
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 436, in _ensured
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs)
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 508, in __call__
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0]
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 704, in execute_method
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit method()
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1050, in _publish
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit producer.publish(msg, expiration=timeout)
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 172, in publish
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit routing_key, mandatory, immediate, exchange, declare)
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 188, in _publish
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit mandatory=mandatory, immediate=immediate,
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 2132, in basic_publish_confirm
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit self.wait([(60, 80), (60, 120)])
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 67, in wait
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit self.channel_id, allowed_methods, timeout)
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 241, in _wait_method
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit channel, method_sig, args, content = read_timeout(timeout)
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 330, in read_timeout
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit return self.method_reader.read_method()
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 189, in read_method
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit raise m
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit IOError: Socket closed
2016-02-12 10:32:06.595 22979 ERROR oslo.messaging._drivers.impl_rabbit
2016-02-12 10:32:06.597 22979 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server 127.0.0.1:5672 closed the connection. Check login credentials: Socket closed