Comment 11 for bug 1405998

Revision history for this message
Christian Reis (kiko) wrote :

Okay, and pserv.log.1.gz also has the Too many open files problem, starting with the familiar find_mac_via_arp() issue:

2015-01-03 10:07:12+0000 [-] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib/python2.7/dist-packages/tftp/protocol.py", line 67, in _startSession
            context, self.backend.get_reader, datagram.filename)
          File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
            return func(*args,**kw)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 62, in wrapper
            return maybeDeferred(func, *args, **kwargs)
        --- <exception caught here> ---
          File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 139, in maybeDeferred
            result = f(*args, **kw)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/pserv_services/tftp.py", line 216, in get_reader
            mac_address = get_remote_mac()
          File "/usr/lib/python2.7/dist-packages/provisioningserver/boot/__init__.py", line 106, in get_remote_mac
            return find_mac_via_arp(remote_host)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/network.py", line 101, in find_mac_via_arp
            output = call_and_check(['ip', 'neigh'], env={'LC_ALL': 'C'})
          File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/shell.py", line 142, in call_and_check
            process = Popen(command, *args, stdout=PIPE, stderr=PIPE, **kwargs)
          File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
            errread, errwrite)
          File "/usr/lib/python2.7/subprocess.py", line 1215, in _execute_child
            errpipe_read, errpipe_write = self.pipe_cloexec()
          File "/usr/lib/python2.7/subprocess.py", line 1167, in pipe_cloexec
            r, w = os.pipe()
        exceptions.OSError: [Errno 24] Too many open files

After a few of those, a while later, we get a few other crashes caused by fd exhaustion (in check_lease_changes(), and find_ip_via_arp()) and then we go into the OOPS generation storm which lasts