logs are not logrotated

Bug #1078213 reported by Zygmunt Krynicki
102
This bug affects 14 people
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
Medium
Unassigned
juju-core
Fix Released
High
Wayne Witzel III

Bug Description

I've been running a small juju cluster for a week now.

There were some connection issues in the data center I'm using and juju had issues connecting to zookeeper. As a result /var/log/juju/juju-machine-agent.log quickly grew to 200M

Overall on the four machines that make up my installation all of juju logs, combined, were over 2GB in size. With the size of the instances this is quickly becoming an issue. I don't imagine how I would manage a large juju cluster without scripting around this issue. I think that juju must support logrotate automatically out of the box.

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: juju 0.5+bzr531-0ubuntu1.3
ProcVersionSignature: User Name 3.2.0-32.51-virtual 3.2.30
Uname: Linux 3.2.0-32-virtual x86_64
ApportVersion: 2.0.1-0ubuntu14
Architecture: amd64
Date: Tue Nov 13 09:09:01 2012
Ec2AMI: ami-000000bf
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
PackageArchitecture: all
ProcEnviron:
 TERM=xterm-256color
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: juju
UpgradeStatus: No upgrade log present (probably fresh install)

Related branches

Revision history for this message
Zygmunt Krynicki (zyga) wrote :
description: updated
Revision history for this message
James Page (james-page) wrote :

I've seen big log files in some of my long running deployments as well; Marking 'Triaged' and 'Medium' and raising a task for juju-core as well.

Changed in juju (Ubuntu):
status: New → Triaged
importance: Undecided → Medium
summary: - juju-machine-agent.log is not logrotated
+ juju-machine-agent.log/logs are not logrotated
Changed in juju-core:
milestone: none → 2.0
Revision history for this message
William Reade (fwereade) wrote : Re: juju-machine-agent.log/logs are not logrotated

We clearly should rotate logs, but the urgency has gone down since we made some changes to what we log in the course of initial juju-core scale testing.

Changed in juju-core:
status: New → Triaged
importance: Undecided → Medium
Tim Penhey (thumper)
tags: added: logging
Curtis Hovey (sinzui)
tags: removed: ec2-images
Curtis Hovey (sinzui)
Changed in juju-core:
importance: Medium → High
Curtis Hovey (sinzui)
tags: added: canonical-webops canonistack
Tom Haddon (mthaddon)
tags: added: canonical-is
removed: canonical-webops
Changed in juju-core:
importance: High → Medium
Curtis Hovey (sinzui)
summary: - juju-machine-agent.log/logs are not logrotated
+ logs are not logrotated
Mark Ramm (mark-ramm)
Changed in juju-core:
importance: Medium → High
milestone: none → 1.21-alpha1
Revision history for this message
Ante Karamatić (ivoks) wrote :

$ du -hs /var/log/juju/
6.9G /var/log/juju/

System installed on 2014-06-02.

Ian Booth (wallyworld)
Changed in juju-core:
assignee: nobody → Nate Finch (natefinch)
status: Triaged → In Progress
Nate Finch (natefinch)
Changed in juju-core:
assignee: Nate Finch (natefinch) → Wayne Witzel III (wwitzel3)
Curtis Hovey (sinzui)
Changed in juju-core:
milestone: 1.21-alpha1 → 1.21-alpha2
Revision history for this message
Haw Loeung (hloeung) wrote :

ubuntu@juju-prod-XX-YYY-machine-0:/var/log/juju$ ls -la
total 19790064
drwxr-xr-x 2 syslog adm 4096 Oct 2 07:33 .
drwxr-xr-x 11 root root 4096 Oct 2 06:38 ..
-rw-r--r-- 1 syslog adm 11719171973 Oct 2 07:33 all-machines.log.1
-rw------- 1 syslog syslog 879 Aug 18 20:17 ca-cert.pem
-rw-r--r-- 1 root root 1757 Oct 2 07:33 machine-0.log
-rw-r--r-- 1 root root 8545817300 Oct 2 07:32 machine-0.log.1
-rw------- 1 syslog syslog 830 Aug 18 20:17 rsyslog-cert.pem
-rw------- 1 syslog syslog 887 Aug 18 20:17 rsyslog-key.pem
u

machines:
  "0":
    agent-state: started
    agent-version: 1.18.4.1
    dns-name: XXX.XXX.XXX.XXX
    instance-id: 542c922e-a61d-4062-8efe-54eed8b9b7b8
    instance-state: ACTIVE
    series: precise
    hardware: arch=amd64 cpu-cores=2 mem=4096M root-disk=51200M

Revision history for this message
Haw Loeung (hloeung) wrote :

Err, it's .1 because I manually rotated them

Revision history for this message
Chris Stratford (chris-gondolin) wrote : Re: [Bug 1078213] Re: logs are not logrotated

As an interim solution:

juju deploy cs:~chris-gondolin/trusty/logrotate
juju set logrotate='logfiles={"juju":{"path":"/var/log/juju/*.log"}}'

On 02/10/14 08:38, Haw Loeung wrote:
> ubuntu@juju-prod-XX-YYY-machine-0:/var/log/juju$ ls -la
> total 19790064
> drwxr-xr-x 2 syslog adm 4096 Oct 2 07:33 .
> drwxr-xr-x 11 root root 4096 Oct 2 06:38 ..
> -rw-r--r-- 1 syslog adm 11719171973 Oct 2 07:33 all-machines.log.1
> -rw------- 1 syslog syslog 879 Aug 18 20:17 ca-cert.pem
> -rw-r--r-- 1 root root 1757 Oct 2 07:33 machine-0.log
> -rw-r--r-- 1 root root 8545817300 Oct 2 07:32 machine-0.log.1
> -rw------- 1 syslog syslog 830 Aug 18 20:17 rsyslog-cert.pem
> -rw------- 1 syslog syslog 887 Aug 18 20:17 rsyslog-key.pem

Revision history for this message
Wayne Witzel III (wwitzel3) wrote :

Thanks for the Can you provide the contents of the logrotate.conf that Juju generates? There was a bug that prevents Juju from properly rotating log files that was fixed in master (https://github.com/juju/juju/commit/d4328ef1d825679d67aee76f2e6709ce4faddaf6) and I want to make sure that this isn't the same issue.

Curtis Hovey (sinzui)
Changed in juju-core:
milestone: 1.21-alpha2 → 1.21-alpha3
Curtis Hovey (sinzui)
Changed in juju-core:
milestone: 1.21-alpha3 → 1.21-beta1
Changed in juju-core:
status: In Progress → Fix Committed
Curtis Hovey (sinzui)
Changed in juju-core:
status: Fix Committed → Fix Released
Xav Paice (xavpaice)
tags: added: canonical-bootstack
Changed in juju (Ubuntu):
status: Triaged → New
affects: juju (Ubuntu) → juju
Revision history for this message
Xav Paice (xavpaice) wrote :

Turns out that Juju 2 now rotates logs by size, 300MB, internally (and tar.gz's them). we can close this bug.

Changed in juju:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.