maas startup can leave lock file

On 16 October 2012 13:40, John A Meinel <email address hidden> wrote:
> Public bug reported:
>
> Maas takes a proper atomic lock during startup, but if the startup
> crashes, it does not properly release the lock file. (It only seems to
> release the lock when startup is successful.)
>
> The current workaround is to just manually rm /run/locks/maas*.lock
> That is a little dangerous because you might accidentally delete an active lock.

The lock is taken with setlock, which:

       ... opens fn for writing (creating it if it does not exist),
       obtains an exclusive lock on it, and runs child.

Where "runs" equals "execs".

This means that when setlock complains that there's still a lock held
that's because there *is* a process with an open file-handle to the
lock file. Use `fuser -v /run/lock/maas.*` to find out what.

> (It only seems to release the lock when startup is successful.)

This might be a bug; the lock is meant to be held the entire time that
the service is running.

With `make run` or `make services/.../@run`, setlock is used to invoke
the service's "run" script. When @start is used, setlock is used to
invoke the "supervise" process.

It might be that start_up is currently spinning in a process because of failing to connect to rabbitmq, etc. However, the lock file will exist after reboot.

The presence of the file doesn't matter so much because the lock is
taken with lockf/flock. The file ought to not be there after a reboot
though:

  $ mount | fgrep /run/lock
  none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/16/2012 5:40 PM, Gavin Panella wrote:
> The presence of the file doesn't matter so much because the lock
> is taken with lockf/flock. The file ought to not be there after a
> reboot though:
>
> $ mount | fgrep /run/lock none on /run/lock type tmpfs
> (rw,noexec,nosuid,nodev,size=5242880)
>
It is taken with a FileLock which creates a different file and
hardlinks it to the lock location.

It isn't open O_EXCL. That is a different way to do it, but not how
the startup.lock file is taken.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (Cygwin)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iEYEARECAAYFAlB9bM8ACgkQJdeBCYSNAAO6oQCfVYJwK/DYQFm+wq1pItkACF++
8LoAn0LW29Wcw4VlhCVqdrXiQ++sd0WL
=f/Oa
-----END PGP SIGNATURE-----

Oops, I've been talking about the wrong thing (development services
locks). You can safely ignore most/all of what I've said. Sorry.

It looks like this may be caused by apache shutdown. It sends several SIGTERM messages, but if the process does not shut down, it forces it with SIGKILL. And once you have a stable maasserver.start_up.lock file, then restarting apache causes all threads to continue to block waiting for the lock to be released.

The current workaround is to shutdown apache completely, then delete the file manually, and then start apache again.