Comment 5 for bug 1811515

Looking at some failures in logstash:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22OSError%3A%20%5BErrno%209%5D%20Bad%20file%20descriptor%5C%22%20AND%20build_status%3A%5C%22FAILURE%5C%22

It appears this started around Jan 9. Looking at changes to neutron and upper-constraints for dependent package versions we test with, these stand out given there are privsep errors in the neutron l3 agent logs:

https://github.com/openstack/neutron/commit/c6d358d4c6926638fe9d5194e3da112c2750c6a4

https://github.com/openstack/requirements/commit/6b45c47e53b8820b68ff78eaec8062c4fdf05a56#diff-0bdd949ed8a7fdd4f95240bd951779c8

In fact, the upper-constraints change depends on the neutron change, so I'm guessing those are the roots of the problem and probably require some more debugging. We do see privsep failures in the l3 agent log:

http://logs.openstack.org/43/628943/2/check/tempest-full/07fe3fe/controller/logs/screen-q-l3.txt.gz#_Jan_13_17_57_16_887623

Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[139657795965328]: 255 {{(pid=5905) _process_cmd /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:452}}
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: Traceback (most recent call last):
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py", line 447, in _process_cmd
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: ret = func(*f_args, **f_kwargs)
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/priv_context.py", line 216, in _wrap
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: return func(*args, **kwargs)
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/opt/stack/neutron/neutron/privileged/agent/linux/ip_lib.py", line 331, in set_link_attribute
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: return _run_iproute_link("set", device, namespace, **attributes)
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/opt/stack/neutron/neutron/privileged/agent/linux/ip_lib.py", line 194, in _run_iproute_link
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: return ip.link(command, index=idx, **kwargs)
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/usr/local/lib/python2.7/dist-packages/pyroute2/iproute/linux.py", line 1115, in link
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: msg_flags=msg_flags)
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/usr/local/lib/python2.7/dist-packages/pyroute2/netlink/nlsocket.py", line 352, in nlm_request
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: return tuple(self._genlm_request(*argv, **kwarg))
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/usr/local/lib/python2.7/dist-packages/pyroute2/netlink/nlsocket.py", line 836, in nlm_request
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: callback=callback):
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/usr/local/lib/python2.7/dist-packages/pyroute2/netlink/nlsocket.py", line 355, in get
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: return tuple(self._genlm_get(*argv, **kwarg))
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/usr/local/lib/python2.7/dist-packages/pyroute2/netlink/nlsocket.py", line 752, in get
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: callback)
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/usr/local/lib/python2.7/dist-packages/pyroute2/netlink/nlsocket.py", line 174, in parse
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: msg.decode()
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/usr/local/lib/python2.7/dist-packages/pyroute2/netlink/__init__.py", line 981, in decode
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: self.unregister_clean_cb()
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/usr/local/lib/python2.7/dist-packages/pyroute2/netlink/__init__.py", line 751, in unregister_clean_cb
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: del clean_cbs[seq]
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: KeyError: 255
Jan 13 17:57:16.889489 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: DEBUG oslo.privsep.daemon [-] privsep: reply[139657795965328]: (5, 'exceptions.KeyError', (255,)) {{(pid=5905) _call_back /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:473}}