[2.2] maas.log rotation is daily and stored for one week only

Bug #1677005 reported by Mike Pontillo
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Medium
Unassigned

Bug Description

When looking for the root cause of a bug I found, I saw that only 7 days of MAAS logs are retained.

This makes it difficult to triage the root cause of issues that could have occurred longer ago than one week.

We should consider a less aggressive schedule, especially for `maas.log`. (Maybe it's fine if regiond and rackd logs are deleted more frequently, but we could still lose important information by being too aggressive here.)

It seems to me that, at a minimum, maas.log should rotate in a way more similar to dpkg.log:

# ls -la /var/log/dpkg.log*
-rw-r--r-- 1 root root 231389 Mar 28 10:37 /var/log/dpkg.log
-rw-r--r-- 1 root root 332437 Feb 28 03:50 /var/log/dpkg.log.1
-rw-r--r-- 1 root root 141406 Jun 1 2016 /var/log/dpkg.log.10.gz
-rw-r--r-- 1 root root 22345 Jan 31 10:43 /var/log/dpkg.log.2.gz
-rw-r--r-- 1 root root 22599 Dec 23 19:25 /var/log/dpkg.log.3.gz
-rw-r--r-- 1 root root 29989 Dec 1 08:34 /var/log/dpkg.log.4.gz
-rw-r--r-- 1 root root 3753 Oct 28 01:11 /var/log/dpkg.log.5.gz
-rw-r--r-- 1 root root 15470 Sep 29 15:07 /var/log/dpkg.log.6.gz
-rw-r--r-- 1 root root 32667 Aug 30 2016 /var/log/dpkg.log.7.gz
-rw-r--r-- 1 root root 7422 Jul 29 2016 /var/log/dpkg.log.8.gz
-rw-r--r-- 1 root root 23754 Jul 1 2016 /var/log/dpkg.log.9.gz

If I look in dpkg.log, I can see that it contains multiple days worth of logging.

Related branches

Revision history for this message
Andres Rodriguez (andreserl) wrote :

I'm fine with that, the difference with dpkg logging vs MAAS logging is that MAAS logging is super verbose in comparison to dpkg's. This means MAAS has WAY more data.

Changed in maas:
importance: Undecided → Wishlist
status: New → Triaged
milestone: none → next
Revision history for this message
Andres Rodriguez (andreserl) wrote :

Actually, marking this as incomplete? What do you mean by we are being too aggressive? I've checked the log rotation configuration and we logrotate once a week.

Looking and long standing MAAS configurations, I can see logs older than 1 week, so we are not removing older logs and we are keeping logs from weeks ago.

Changed in maas:
status: Triaged → Incomplete
importance: Wishlist → Undecided
milestone: next → 2.2.0
summary: - [2.2] MAAS log rotation is too aggressive
+ [2.2] maas.log rotation is daily and stored for one week only
Changed in maas:
importance: Undecided → Medium
status: Incomplete → Triaged
Revision history for this message
Mike Pontillo (mpontillo) wrote :

I think maas.log is on the order of dpkg.log's verboseness, but I agree if we're talking about regiond.log or rackd.log. We tend to log important information in maas.log, so we should err on the side of not deleting old maas.log files.

What I observed is that maas.log is rotated daily and deleted after 1 week, as follows:

# ls -la /var/log/maas/maas*
-rw-r--r-- 1 syslog syslog 42656 Mar 28 19:10 /var/log/maas/maas.log
-rw-r--r-- 1 syslog syslog 114338 Mar 28 06:20 /var/log/maas/maas.log.1
-rw-r--r-- 1 syslog syslog 2946 Mar 27 06:25 /var/log/maas/maas.log.2.gz
-rw-r--r-- 1 syslog syslog 2884 Mar 26 06:14 /var/log/maas/maas.log.3.gz
-rw-r--r-- 1 syslog syslog 5526 Mar 25 06:14 /var/log/maas/maas.log.4.gz
-rw-r--r-- 1 syslog syslog 3000 Mar 24 06:18 /var/log/maas/maas.log.5.gz
-rw-r--r-- 1 syslog syslog 3350 Mar 23 06:19 /var/log/maas/maas.log.6.gz
-rw-r--r-- 1 syslog syslog 3085 Mar 22 06:19 /var/log/maas/maas.log.7.gz

This MAAS was installed longer than one week ago and upgraded to 2.2 on March 18th.

Changed in maas:
status: Triaged → New
status: New → Triaged
Changed in maas:
status: Triaged → Fix Committed
Revision history for this message
Gavin Panella (allenap) wrote :

With the probable exception of maas.log, all logs messages are also
stored by journald. We should simply *stop* logging to regiond.log and
rackd.log. Leave a message in rackd.log and regiond.log like so:

  Use `journalctl -u maas-rackd.service` to view the rack logs.

or:

  Use `journalctl -u maas-regiond*.service` to view the region logs.

This would mean we're not logging the bulk of MAAS's output _twice_ and
make it reasonable to keep more history of maas.log around.

FWIW, this has been the situation for *years*. Every time I mention it
the silence is deafening. What's blocking us from just doing this?

Revision history for this message
Mike Pontillo (mpontillo) wrote :

Gavin, I think you are absolutely right.

I think there is some sense of fear to moving to journalctl for all logging. Personally, I can accept it rationally. But the idea that /var/log isn't where the logs go flies in the face of decades of traditional UNIX administration experience, expectations, and that "everything is a file" sense of simplicity.

We will move to the 21st century one of these days. But that is a separate bug report. ;-)

Of course, given our apparent emotional attachment to these log files, we should still rotate them correctly if we're going to rotate them!

Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.