terminate_instance(): RuntimeError: Second simultaneous read on fileno 16 detected

Bug #1229475 reported by Joe Gordon
44
This bug affects 13 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
High
Unassigned
python-neutronclient
Fix Released
High
Davanum Srinivas (DIMS)

Bug Description

http://logs.openstack.org/59/47659/6/check/gate-tempest-devstack-vm-neutron/fc83d44/logs/screen-n-cpu.txt.gz#_2013-09-23_18_25_06_484

2013-09-23 18:25:06.484 ERROR nova.openstack.common.rpc.amqp [req-3a80ff5a-817b-4eb8-be67-7f180bef8a6e demo demo] Exception during message handling
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp **args)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 354, in decorated_function
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/exception.py", line 90, in wrapped
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp payload)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/exception.py", line 73, in wrapped
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 244, in decorated_function
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp pass
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 230, in decorated_function
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 295, in decorated_function
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 272, in decorated_function
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 259, in decorated_function
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 1793, in terminate_instance
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp do_terminate_instance(instance, bdms)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/lockutils.py", line 246, in inner
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp return f(*args, **kwargs)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 1785, in do_terminate_instance
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp reservations=reservations)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/hooks.py", line 105, in inner
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp rv = f(*args, **kwargs)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 1758, in _delete_instance
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp user_id=user_id)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 1727, in _delete_instance
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp self.conductor_api.instance_info_cache_delete(context, db_inst)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/conductor/api.py", line 107, in instance_info_cache_delete
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp return self._manager.instance_info_cache_delete(context, instance)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/conductor/rpcapi.py", line 312, in instance_info_cache_delete
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp cctxt.call(context, 'instance_info_cache_delete', instance=instance_p)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/rpcclient.py", line 85, in call
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp return self._invoke(self.proxy.call, ctxt, method, **kwargs)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/rpcclient.py", line 63, in _invoke
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp return cast_or_call(ctxt, msg, **self.kwargs)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/proxy.py", line 126, in call
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp result = rpc.call(context, real_topic, msg, timeout)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/__init__.py", line 139, in call
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp return _get_impl().call(CONF, context, topic, msg, timeout)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/impl_kombu.py", line 816, in call
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp rpc_amqp.get_connection_pool(conf, Connection))
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/amqp.py", line 572, in call
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp rv = multicall(conf, context, topic, msg, timeout, connection_pool)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/amqp.py", line 565, in multicall
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp with ConnectionContext(conf, connection_pool) as conn:
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/amqp.py", line 122, in __init__
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp self.connection = connection_pool.get()
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/eventlet/pools.py", line 119, in get
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp created = self.create()
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/amqp.py", line 76, in create
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp return self.connection_cls(self.conf)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/impl_kombu.py", line 465, in __init__
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp self.reconnect()
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/impl_kombu.py", line 535, in reconnect
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp self._connect(params)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/impl_kombu.py", line 511, in _connect
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp self.connection.connect()
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 246, in connect
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp return self.connection
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 761, in connection
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp self._connection = self._establish_connection()
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 720, in _establish_connection
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp conn = self.transport.establish_connection()
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 115, in establish_connection
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp conn = self.Connection(**opts)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 142, in __init__
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp (10, 10), # start
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 69, in wait
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp self.channel_id, allowed_methods)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 204, in _wait_method
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp self.method_reader.read_method()
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 192, in read_method
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp raise m
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp RuntimeError: Second simultaneous read on fileno 16 detected. Unless you really know what you're doing, make sure that only one greenthread can read any particular socket. Consider using a pools.Pool. If you do know what you're doing and want to disable this error, call eventlet.debug.hub_prevent_multiple_readers(False)
2013-09-23 18:25:06.484 3476 TRACE nova.openstack.common.rpc.amqp

Revision history for this message
Joe Gordon (jogo) wrote :
Download full text (7.7 KiB)

http://logs.openstack.org/48/47948/1/check/gate-tempest-devstack-vm-neutron/6e1f600/logs/screen-n-cpu.txt.gz#_2013-09-24_01_33_24_904

Another case:

 2013-09-24 01:33:24.904 ERROR nova.openstack.common.rpc.amqp [req-337bcd98-3d47-452c-a269-0629725d3dce demo demo] Exception during message handling
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp **args)
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 354, in decorated_function
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/exception.py", line 90, in wrapped
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp payload)
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/exception.py", line 73, in wrapped
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 244, in decorated_function
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp pass
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 230, in decorated_function
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 295, in decorated_function
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 272, in decorated_function
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 259, in decorated_function
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 1793, in terminate_instance
2013-09-24 01:33:24.904 4483 TRACE nova.openstack.common.rpc.amqp do_terminate_instance(instance, bdms)
2013-09-24 01:33:24.904 4483 T...

Read more...

Revision history for this message
Joe Gordon (jogo) wrote :

logstash query: @message:"Second simultaneous read on fileno" AND @fields.build_status:"FAILURE"

Mark McLoughlin (markmc)
Changed in oslo:
milestone: none → havana-rc1
Revision history for this message
Joe Gordon (jogo) wrote :

Mark, I an not sure what the actual bug is here. The first stacktrace was from rpc, but the second is from neutronclient.

Here is a third case of it: http://logs.openstack.org/87/44787/13/check/gate-tempest-devstack-vm-neutron/11d697f/logs/screen-n-cpu.txt.gz

Also until we get elastic-recheck fully running this one is hard to find for the untrained eye. But logstash shows its been hit 12 times in the last 24 hours.

Revision history for this message
Mark McLoughlin (markmc) wrote :

I think we're seeing this often enough that we should be concerned. It's in terminate_instance() every time, so it's not completely random. Putting it on the havana-rc1 milestone to raise its visibility.

Definitely looks like we're not cleaning up a closed fd somehow - e.g. a greenthread closes an fd but keeps the handle around and incorrectly read()/poll()s on it? Then rpc/neutronclient create a new connection, re-uses this fd and we see an error that suggests the issue is rpc/neutronclient rather than whatever other thing screwed up.

Changed in python-neutronclient:
status: New → Incomplete
Changed in oslo:
status: New → Incomplete
milestone: havana-rc1 → none
Changed in nova:
status: New → Triaged
importance: Undecided → High
milestone: none → havana-rc1
summary: - From oslo rpc: RuntimeError: Second simultaneous read on fileno 16
- detected
+ terminate_instance(): RuntimeError: Second simultaneous read on fileno
+ 16 detected
Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

Looking at logstash - I see 214 hits for @message:"Second simultaneous read" and can break it into 3 buckets.

100 Hits - @message:"Second simultaneous read" AND @fields.build_name:"gate-tempest-devstack-vm-neutron" AND @fields.filename:"logs/screen-n-cpu.txt" AND @fields.module:"nova.openstack.common.rpc.amqp"
 90 Hits - @message:"Second simultaneous read" AND @fields.build_name:"gate-tempest-devstack-vm-neutron" AND @fields.filename:"logs/screen-n-cpu.txt" AND @fields.module:"nova.network.neutronv2"
 14 Hits - @message:"Second simultaneous read" AND @fields.build_name:"gate-tempest-devstack-vm-neutron" AND @fields.filename:"logs/screen-n-cpu.txt" AND (NOT @fields.module:"nova.openstack.common.rpc.amqp") AND (NOT @fields.module:"nova.network.neutronv2")

Notes:
* All the hits from "gate-tempest-devstack-vm-neutron" jenkins job
* All issues happen in the "logs/screen-n-cpu.txt"
* First hit was on 2013-09-12T09:19:53.000

Still looking...

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

14 hits for @message:"Second simultaneous write"

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-neutronclient (master)

Fix proposed to branch: master
Review: https://review.openstack.org/48552

Changed in python-neutronclient:
assignee: nobody → Davanum Srinivas (DIMS) (dims-v)
status: Incomplete → In Progress
Changed in python-neutronclient:
assignee: Davanum Srinivas (DIMS) (dims-v) → nobody
status: In Progress → Confirmed
Revision history for this message
Joe Gordon (jogo) wrote :

Looks like this is still happening. And is happening during passing tempest jobs:

134 hits in the last 24 hours. and happened 34 times during passing tempest jobs.

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

@Joe, Managed to capture some tracebacks in http://logs.openstack.org/11/49011/4/check/check-tempest-devstack-vm-neutron/0ab22df/logs/screen-n-cpu.txt.gz

First call for fd 16 is here - http://paste.openstack.org/show/47754/
Second call to fd 16 is here - http://paste.openstack.org/show/47755/

And then we end up with
2013-10-01 12:34:13.962 3428 TRACE nova.openstack.common.rpc.amqp RuntimeError: My Second simultaneous read on fileno 16 detected. Unless you really know what you're doing, make sure that only one greenthread can read any particular socket. Consider using a pools.Pool. If you do know what you're doing and want to disable this error, call eventlet.debug.hub_prevent_multiple_readers(False)

Revision history for this message
Russell Bryant (russellb) wrote :

After some more looking at the run dims linked in comment #9, it's clear that the error comes from neutronclient getting a bogus fd. That fd (16) is actively in use by rpc. neutronclient barfs on it because it ends up reading data from rabbitmq.

http://logs.openstack.org/11/49011/4/check/check-tempest-devstack-vm-neutron/0ab22df/logs/screen-n-cpu.txt.gz#_2013-10-01_12_34_23_604

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

This reading from bad fd may be the problem behind the other bug as well - https://bugs.launchpad.net/tempest/+bug/1223161

Changed in python-neutronclient:
assignee: nobody → Davanum Srinivas (DIMS) (dims-v)
status: Confirmed → In Progress
Revision history for this message
Thierry Carrez (ttx) wrote :

No need for a nova patch

Changed in nova:
status: Triaged → In Progress
status: In Progress → Invalid
no longer affects: oslo
Revision history for this message
Russell Bryant (russellb) wrote :

Marking Incomplete for Nova, because I don't think we have found the root cause yet, so it still may be nova's fault.

Changed in nova:
milestone: havana-rc1 → none
status: Invalid → Incomplete
tags: added: havana-rc-potential
Changed in python-neutronclient:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-neutronclient (master)

Reviewed: https://review.openstack.org/49210
Committed: http://github.com/openstack/python-neutronclient/commit/e6a9f79066943b75e2c60b0165533d5886cc9cd8
Submitter: Jeremy Stanley (<email address hidden>)
Branch: master

commit e6a9f79066943b75e2c60b0165533d5886cc9cd8
Author: Davanum Srinivas <email address hidden>
Date: Tue Oct 1 12:27:32 2013 -0400

    Forcibly clear connection pool after every request

    Neutron client is ending up with bad socket descriptors
    which causes the behavior including trying to read from
    the socket that amqp currently holds etc. A temporary
    solution is to clear the connection pool so we get a
    fresh socket built for every request.

    Closes-Bug: 1229475

    Change-Id: I43b1fe535bc5bd1fc726c97a293bf7912cd7ef1e

Changed in python-neutronclient:
status: In Progress → Fix Committed
Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :
Changed in python-neutronclient:
status: Fix Committed → In Progress
Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :
Changed in python-neutronclient:
status: In Progress → Fix Committed
Changed in nova:
status: Incomplete → Invalid
Thierry Carrez (ttx)
tags: removed: havana-rc-potential
Akihiro Motoki (amotoki)
Changed in python-neutronclient:
milestone: none → 2.3.0-2.3.4
Akihiro Motoki (amotoki)
Changed in python-neutronclient:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.