Okay, and pserv.log.1.gz also has the Too many open files problem, starting with the familiar find_mac_via_arp() issue:
2015-01-03 10:07:12+0000 [-] Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/tftp/protocol.py", line 67, in _startSession context, self.backend.get_reader, datagram.filename)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
return func(*args,**kw)
File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 62, in wrapper
return maybeDeferred(func, *args, **kwargs)
--- <exception caught here> ---
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 139, in maybeDeferred
result = f(*args, **kw)
File "/usr/lib/python2.7/dist-packages/provisioningserver/pserv_services/tftp.py", line 216, in get_reader mac_address = get_remote_mac()
File "/usr/lib/python2.7/dist-packages/provisioningserver/boot/__init__.py", line 106, in get_remote_mac
return find_mac_via_arp(remote_host)
File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/network.py", line 101, in find_mac_via_arp
output = call_and_check(['ip', 'neigh'], env={'LC_ALL': 'C'})
File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/shell.py", line 142, in call_and_check
process = Popen(command, *args, stdout=PIPE, stderr=PIPE, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__ errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1215, in _execute_child errpipe_read, errpipe_write = self.pipe_cloexec()
File "/usr/lib/python2.7/subprocess.py", line 1167, in pipe_cloexec
r, w = os.pipe() exceptions.OSError: [Errno 24] Too many open files
After a few of those, a while later, we get a few other crashes caused by fd exhaustion (in check_lease_changes(), and find_ip_via_arp()) and then we go into the OOPS generation storm which lasts
Okay, and pserv.log.1.gz also has the Too many open files problem, starting with the familiar find_mac_via_arp() issue:
2015-01-03 10:07:12+0000 [-] Unhandled Error python2. 7/dist- packages/ tftp/protocol. py", line 67, in _startSession
context, self.backend. get_reader, datagram.filename) python2. 7/dist- packages/ twisted/ python/ context. py", line 118, in callWithContext text(). callWithContext (ctx, func, *args, **kw) python2. 7/dist- packages/ twisted/ python/ context. py", line 81, in callWithContext python2. 7/dist- packages/ provisioningser ver/utils/ twisted. py", line 62, in wrapper python2. 7/dist- packages/ twisted/ internet/ defer.py" , line 139, in maybeDeferred python2. 7/dist- packages/ provisioningser ver/pserv_ services/ tftp.py" , line 216, in get_reader
mac_ address = get_remote_mac() python2. 7/dist- packages/ provisioningser ver/boot/ __init_ _.py", line 106, in get_remote_mac via_arp( remote_ host) python2. 7/dist- packages/ provisioningser ver/utils/ network. py", line 101, in find_mac_via_arp check([ 'ip', 'neigh'], env={'LC_ALL': 'C'}) python2. 7/dist- packages/ provisioningser ver/utils/ shell.py" , line 142, in call_and_check python2. 7/subprocess. py", line 710, in __init__
errread, errwrite) python2. 7/subprocess. py", line 1215, in _execute_child
errpipe_ read, errpipe_write = self.pipe_cloexec() python2. 7/subprocess. py", line 1167, in pipe_cloexec
exceptions. OSError: [Errno 24] Too many open files
Traceback (most recent call last):
File "/usr/lib/
File "/usr/lib/
return self.currentCon
File "/usr/lib/
return func(*args,**kw)
File "/usr/lib/
return maybeDeferred(func, *args, **kwargs)
--- <exception caught here> ---
File "/usr/lib/
result = f(*args, **kw)
File "/usr/lib/
File "/usr/lib/
return find_mac_
File "/usr/lib/
output = call_and_
File "/usr/lib/
process = Popen(command, *args, stdout=PIPE, stderr=PIPE, **kwargs)
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
r, w = os.pipe()
After a few of those, a while later, we get a few other crashes caused by fd exhaustion (in check_lease_ changes( ), and find_ip_via_arp()) and then we go into the OOPS generation storm which lasts