authserver performance problems

James Troup <email address hidden> writes:

> The authserver seems to be taking a long time to process requests.
> Two test requests by Andrew took 45s and over a minute. The
> authserver is apparently using 99% CPU and processing an awful lot
> of requests.

This appears to be a problem with log rotation. The authserver is
setup to rotate logfiles once they reach 1Mb big. Because every wiki
page view hits the authserver, that size limit is reached very fast
and there was over 30000 rotated logfiles. Andrew's moved them out of
the way into an 'archived' folder and the authserver performance has
returned to something more normal/reasonable.

(However, logging is currently broken - possibly until the authserver
is restarted)

--
James

The authserver has been restarted since the last launchpad rollout, and is now producing (and rotating) logfiles again. It produced ~10 logfiles in the first two hours (so 5/hour), and then ~20 in the third hour. If the high demand keeps up, that's 20*24 = 480 logfiles per day. I guess that means the rotation will start hurting noticeably within a week. (that's roughly 200000 authserver requests served in 3 hours, btw).

Unfortunately, twistd doesn't give us any easy way to vary the default logging behaviour, so we'll need to workaround this in the short term.

Workarounds that spring to mind:
  * symlink authserver.log to /dev/null
  * a daily cronjob to move the days logs to archived/yyyy-mm-dd/
  * a regular cronjob to just delete logs.

I think the cronjob to archive the logs is probably the best approach for now.

James, what do you think?

Andrew Bennetts <email address hidden> writes:

> Workarounds that spring to mind:
> * symlink authserver.log to /dev/null
> * a daily cronjob to move the days logs to archived/yyyy-mm-dd/
> * a regular cronjob to just delete logs.
>
> I think the cronjob to archive the logs is probably the best approach
> for now.

I agree.

--
James

A daily cronjob is now in place to move rotated logs out of the way.

The cronjob seems to have taken care of the problem. Thanks!