isc-dhcp-server fails to start on second & further attempts with 'Can't open /var/lib/dhcp/dhcpd.leases for append'

Try disabling apparmour. That fixed it for me. see also bug 1540672.

Status changed to 'Confirmed' because the bug affects multiple users.

This bug was fixed in the package isc-dhcp - 4.3.3-5ubuntu8

---------------
isc-dhcp (4.3.3-5ubuntu8) xenial; urgency=medium

  * debian/apparmor/sbin.dhclient: include the openssl abstraction
    (LP: #1543794)

 -- Jamie Strandboge <email address hidden> Thu, 25 Feb 2016 08:32:37 -0600

Yes, isc-dhcp - 4.3.3-5ubuntu8 fixes the issue with the dhclient stuff that caused this:

Feb 26 14:05:21 DOUG-64 kernel: [ 17.518717] audit: type=1400 audit(1456524321.220:10): apparmor="DENIED" operation="open" profile="/sbin/dhclient" name="/etc/ssl/openssl.cnf" pid=483 comm="dhclient" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

However, that is not what this bug report is about. This bug report is about these ones:

Feb 27 07:53:07 DOUG-64 kernel: [ 21.350331] audit: type=1400 audit(1456588387.171:12): apparmor="ALLOWED" operation="capable" profile="/usr/sbin/dhcpd" pid=1248 comm="dhcpd" capability=1 capname="dac_override"
Feb 27 07:53:07 DOUG-64 kernel: [ 21.466959] audit: type=1400 audit(1456588387.287:13): apparmor="ALLOWED" operation="capable" profile="/usr/sbin/dhcpd" pid=1248 comm="dhcpd" capability=3 capname="fowner"

Note: the above are "ALLOWED" because I have it set to complain mode.

which were fixed, but have now returned.
setting back to confirmed.

isc-dhcp 4.3.3-5ubuntu8 appears to fix bug #1549736, but not this issue.

This bug was fixed in the package isc-dhcp - 4.3.3-5ubuntu9

---------------
isc-dhcp (4.3.3-5ubuntu9) xenial; urgency=medium

  * debian/isc-dhcp-server.isc-dhcp-server{,6}.service: adjust permissions
    for most recent PARANOIA commits to avoid granting 'capability
    dac_override' in the AppArmor profile on second and subsequent starts
    (LP: #1543794):
    - /var/lib/dhcp is root:dhcpd and 775
    - /var/lib/dhcp/dhcpd{,6}.leases is root:dhcpd and 664
    - don't chown /run/dhcp-server to dhcpd:dhcpd since the pid file is now
      created as root:root and the pid file was not being created before. If
      this needs to be reintroduced in the future, use root:dhcpd and 775,
      like with the leases file.

 -- Jamie Strandboge <email address hidden> Tue, 01 Mar 2016 08:19:43 -0600

The problem still exists, for 4.3.3-5ubuntu12.1 (Ubuntu 16.04).

How to reproduce (using /etc/dhcp/dhcpd.conf installed from the package itself, without changes):

1. Starting with a fresh install:

nornetpp@bjoernson:~$ sudo rm -rf /var/lib/dhcp /run/dhcp-server
nornetpp@bjoernson:~$ sudo apt-get install --reinstall isc-dhcp-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 0 B/412 kB of archives.
After this operation, 0 B of additional disk space will be used.
Preconfiguring packages ...
(Reading database ... 105366 files and directories currently installed.)
Preparing to unpack .../isc-dhcp-server_4.3.3-5ubuntu12.1_amd64.deb ...
Unpacking isc-dhcp-server (4.3.3-5ubuntu12.1) over (4.3.3-5ubuntu12.1) ...
Processing triggers for systemd (229-4ubuntu7) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up isc-dhcp-server (4.3.3-5ubuntu12.1) ...
insserv: warning: script 'screen-cleanup' missing LSB tags and overrides
insserv: Default-Start undefined, assuming empty start runlevel(s) for script `screen-cleanup'
insserv: Default-Stop undefined, assuming empty stop runlevel(s) for script `screen-cleanup'
isc-dhcp-server start/running, process 24418

nornetpp@bjoernson:~$ apt-show-versions | grep isc-dhcp-server
isc-dhcp-server:amd64/xenial-updates 4.3.3-5ubuntu12.1 uptodate
isc-dhcp-server:i386 not installed

2. Starting the service fails:

nornetpp@bjoernson:~$ sudo service isc-dhcp-server start
isc-dhcp-server start/running, process 24553
nornetpp@bjoernson:~$ sleep 3
nornetpp@bjoernson:~$ sudo service isc-dhcp-server status
isc-dhcp-server stop/waiting
nornetpp@bjoernson:~$ tail -n20 /var/log/syslog.1
Aug 3 11:23:35 bjoernson dhcpd[24599]: process and the information we find helpful for debugging..
Aug 3 11:23:35 bjoernson dhcpd[24599]:
Aug 3 11:23:35 bjoernson dhcpd[24599]: exiting.
Aug 3 11:23:35 bjoernson dhcpd[24603]: Internet Systems Consortium DHCP Server 4.3.3
Aug 3 11:23:35 bjoernson dhcpd[24603]: Copyright 2004-2015 Internet Systems Consortium.
Aug 3 11:23:35 bjoernson dhcpd[24603]: All rights reserved.
Aug 3 11:23:35 bjoernson dhcpd[24603]: For info, please visit https://www.isc.org/software/dhcp/
Aug 3 11:23:35 bjoernson dhcpd[24604]: Internet Systems Consortium DHCP Server 4.3.3
Aug 3 11:23:35 bjoernson dhcpd[24604]: Copyright 2004-2015 Internet Systems Consortium.
Aug 3 11:23:35 bjoernson dhcpd[24604]: All rights reserved.
Aug 3 11:23:35 bjoernson dhcpd[24604]: For info, please visit https://www.isc.org/software/dhcp/
Aug 3 11:23:35 bjoernson dhcpd[24604]: Can't open /var/lib/dhcp/dhcpd.leases for append.
Aug 3 11:23:35 bjoernson dhcpd[24604]:
Aug 3 11:23:35 bjoernson dhcpd[24604]: If you think you have received this message due to a bug rather
Aug 3 11:23:35 bjoernson dhcpd[24604]: than a configuration issue please read the section on submitting
Aug 3 11:23:35 bjoernson dhcpd[24604]: bugs on either our web page at www.isc.org or in the README file
Aug 3 11:23:35 bjoernson dhcpd[24604]: before submitting a bug. ...

I got around it by disabling apparmor for dhcpd:

sudo ln -s /etc/apparmor.d/usr.sbin.dhcpd /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.dhcpd

This is the workaround that I used after upgrading to 16.04:

sudo ln -s /etc/apparmor.d/usr.sbin.dhcpd /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.dhcpd

The output in comment #8 shows that this system is running upstart despite being 16.04. Upstart is not supported as init on 16.04 except during the upgrade. This bug appears to have been fixed for the systemd unit, but not for the upstart job. A simple reboot to systemd should suffice to fix the problem on 16.04.

I have the same issue on Ubuntu 16.04, removing apparmor confs helped.

Issue still exists with fresh installation of 16.04 server.

rolly@head:~$ apt-show-versions | grep isc-dhcp-server
isc-dhcp-server:amd64/xenial-updates 4.3.3-5ubuntu12.6 uptodate

it works with the solution:
sudo ln -s /etc/apparmor.d/usr.sbin.dhcpd /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.dhcpd

I am running dhcpd on infiniband, ipoib on ib0, here is the working output:
rolly@head:~$ sudo service isc-dhcp-server status
[sudo] password for rolly:
● isc-dhcp-server.service - ISC DHCP IPv4 server
   Loaded: loaded (/lib/systemd/system/isc-dhcp-server.service; disabled; vendor preset: enabled)
   Active: active (running) since Mon 2017-01-09 22:05:34 HKT; 19min ago
     Docs: man:dhcpd(8)
 Main PID: 5173 (dhcpd)
    Tasks: 1
   Memory: 9.0M
      CPU: 25ms
   CGroup: /system.slice/isc-dhcp-server.service
           └─5173 dhcpd -user dhcpd -group dhcpd -f -4 -pf /run/dhcp-server/dhcpd.pid -cf /etc/dhcp/dhcpd.conf ib0

Jan 09 22:05:34 head dhcpd[5173]: All rights reserved.
Jan 09 22:05:34 head dhcpd[5173]: For info, please visit https://www.isc.org/software/dhcp/
Jan 09 22:05:34 head dhcpd[5173]: Wrote 0 leases to leases file.
Jan 09 22:05:34 head dhcpd[5173]: Listening on LPF/ib0//192.168.0.0/24
Jan 09 22:05:34 head sh[5173]: Listening on LPF/ib0//192.168.0.0/24
Jan 09 22:05:34 head sh[5173]: Sending on LPF/ib0//192.168.0.0/24
Jan 09 22:05:34 head sh[5173]: Sending on Socket/fallback/fallback-net
Jan 09 22:05:34 head dhcpd[5173]: Sending on LPF/ib0//192.168.0.0/24
Jan 09 22:05:34 head dhcpd[5173]: Sending on Socket/fallback/fallback-net
Jan 09 22:05:34 head dhcpd[5173]: Server starting service.

Rolly, please file a new bug against isc-dhcp and make sure to include the AppArmor DENIED lines from your dmesg or auditd logs. Thanks!