graceful full disk handling. syslog.0 and kern.log.0 grow huge

Thank you for taking the time to report this bug and helping to make Ubuntu better. You reported this bug a while ago and there hasn't been any activity in recently. We were wondering if this is still an issue for you? Thanks in advance.

As I mentioned in the description, I deleted the problem files and all was well. So no, it is not still an issue for me. But I am concerned that this can just happen for no apparent reason. Does Ubuntu have some way of limiting the size of log files? 7Gb should be plenty big enough for /. I am not keen to leave clients with machines running an OS that can just fail to start at random due to something so apparently trivial.

this happened to me recently too, syslog.0 kernlog.0 and messages.0 where all 13.4 GB each, i just deleted the files when i realised what had happened so it's not a major problem and i'll keep watcging them to see if it happenes again. I'm running ;

ubuntu gutsy
2.6.22-14-generic x86_64 GNU/Linux

I'm hoping the following new tool in Hardy may prevent future occurrences of this bug:

https://wiki.ubuntu.com/HardyFullDiskHandling

It also happened to me. I deleted it, but the problem is, after few reboot, it grow huge again.

I confirm this bug. I'm running Ubuntu Hardy 8.04 kernel 2.6.24-19 and both syslog.0 and kern.log.0 take up 1,3 GB each.

I also confirm the bug. My syslog.0 and kern.log.0 are taking 1.2gb each.
I'm running Ubuntu Hardy 8.04 with kernel 2.6.24-19 as well.

Are your systems running cron? Have you, at any time while running hardy, enabled, then disabled your root account?

If the latter is the case, this is a side effect of a bug which has been fixed in intrepid (and should be SRU'd to hardy).

> Are your systems running cron? Have you, at any time while running hardy, enabled, then disabled your root account?

Personally - no, and no. I haven't had this problem since I upgraded from 7.04 - either in 7.10 or in any of the subsequent releases, although it appears that other people have.

I've just recently upgraded to Ibex and started having this problem, except so far it hasn't gone away.

I upgraded a few days ago, and two days ago my computer suddenly went under massive load and froze up. I eventually managed to get to a tty to see a kernel logging error message being repeated many times over. I had no real control so I did a hard reset, X failed due to no HDD space, etc. I then found the following log files:

-rw-r----- 1 syslog adm 2147483647 kern.log.0
-rw-r----- 1 syslog adm 2147483647 messages.0
-rw-r----- 1 syslog adm 2147483647 syslog.0
-rw-r----- 1 syslog adm 1523269632 kern.log
-rw-r----- 1 syslog adm 1523269632 syslog
-rw-r----- 1 syslog adm 1428373504 messages

I removed them and all was well again, except just now the problem repeated itself. So far all I have done to investigate or resolve the problem is remove the log files. I'm not aware of altering any cron jobs or any other behaviour that could have caused this, and it appears to have only begun to occur since installing Ibex.

I can post later with any information requested.

Um, in the last ten minutes or so it appears that 500mb of disk space has been used on logging ... I'll check back later today.

I also have this issue with syslog.0 and daemon.log.0

-rw-r----- 1 syslog adm 16224438 2008-11-10 19:18 daemon.log
-rw-r----- 1 syslog adm 17169385 2008-11-10 19:18 syslog
-rw-r----- 1 syslog adm 2329075136 2008-10-29 19:25 daemon.log.0
-rw-r----- 1 syslog adm 2329626084 2008-10-29 19:25 syslog.0

Here is my Hardy version :
Linux condor 2.6.24-21-generic #1 SMP Tue Oct 21 23:09:30 UTC 2008 x86_64 GNU/Linux

I continue to have this problem. I think it is related to putting my laptop on standby. So far the standby mode has been far more reliable than the previous Ubuntu version (Hardy LTS would often not wake up at all), but on some startups I get flooded with messages at the console ("bad scheduling from the idle thread!"), and that appears to be flooding the logs.

This issue is happening right now. When relatively idle and running top, the top [user:commands usage] are { root:dd 61%, klog:klogd 42%, syslog:syslogd 25% } ... which could be expected from the rate at which the logs fill up.

I'm running a Dell M1330 with Intel(R) Core(TM)2 Duo Processor T9300 - 2.5GHz, 6MB Cache, 800 MHz FSB.

I'll provide lspci output and the last 50 lines of the three log files that have been giving me difficulty. Hopefully this information will be useful to someone in identifying this issue. Again, I'm available to provide any other information necessary on request.

~$ lspci
00:00.0 Host bridge: Intel Corporation Mobile PM965/GM965/GL960 Memory Controller Hub (rev 0c)
00:01.0 PCI bridge: Intel Corporation Mobile PM965/GM965/GL960 PCI Express Root Port (rev 0c)
00:1a.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #4 (rev 02)
00:1a.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #5 (rev 02)
00:1a.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Controller #2 (rev 02)
00:1b.0 Audio device: Intel Corporation 82801H (ICH8 Family) HD Audio Controller (rev 02)
00:1c.0 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 1 (rev 02)
00:1c.1 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 2 (rev 02)
00:1c.3 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 4 (rev 02)
00:1c.5 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 6 (rev 02)
00:1d.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #1 (rev 02)
00:1d.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #2 (rev 02)
00:1d.2 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #3 (rev 02)
00:1d.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Controller #1 (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev f2)
00:1f.0 ISA bridge: Intel Corporation 82801HEM (ICH8M) LPC Interface Controller (rev 02)
00:1f.1 IDE interface: Intel Corporation 82801HBM/HEM (ICH8M/ICH8M-E) IDE Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801HBM/HEM (ICH8M/ICH8M-E) SATA AHCI Controller (rev 02)
00:1f.3 SMBus: Intel Corporation 82801H (ICH8 Family) SMBus Controller (rev 02)
01:00.0 VGA compatible controller: nVidia Corporation GeForce 8400M GS (rev a1)
03:01.0 FireWire (IEEE 1394): Ricoh Co Ltd R5C832 IEEE 1394 Controller (rev 05)
03:01.1 SD Host controller: Ricoh Co Ltd R5C822 SD/SDIO/MMC/MS/MSPro Host Adapter (rev 22)
03:01.2 System peripheral: Ricoh Co Ltd R5C843 MMC Host Controller (rev 12)
03:01.3 System peripheral: Ricoh Co Ltd R5C592 Memory Stick Bus Host Adapter (rev 12)
03:01.4 System peripheral: Ricoh Co Ltd xD-Picture Card Controller (rev ff)
09:00.0 E...

I recently upgraded to Intrepid on my Acer laptop, and I came across a similar problem:

$ ls -lah syslog*
-rw-r----- 1 syslog adm 58 2008-11-25 10:37 syslog
-rw-r----- 1 syslog adm 242K 2008-11-25 10:36 syslog.0
-rw-r----- 1 syslog adm 762M 2008-11-24 02:00 syslog.1.gz
-rw-r----- 1 syslog adm 20K 2008-11-20 11:02 syslog.2.gz
-rw-r----- 1 syslog adm 20K 2008-11-19 10:38 syslog.3.gz
-rw-r----- 1 syslog adm 187M 2008-11-18 09:54 syslog.4.gz
-rw-r----- 1 syslog adm 19K 2008-11-17 09:19 syslog.5.gz
-rw-r----- 1 syslog adm 21K 2008-11-14 10:13 syslog.6.gz

As you can see, syslog.1.gz and syslog.4.gz are still very large - even after gzipping. Before syslog.1.gz was zipped, it took up 15GB. I only noticed because gzip was using up 70% of my processor for 20 minutes..

I just noticed that it is also the case with kern.log:

$ ls -lah kern.log*
-rw-r----- 1 syslog adm 178K 2008-11-25 10:49 kern.log
-rw-r----- 1 syslog adm 15G 2008-11-24 01:05 kern.log.0
-rw-r----- 1 syslog adm 118M 2008-11-18 09:40 kern.log.1.gz
-rw-r----- 1 syslog adm 69M 2008-11-17 11:13 kern.log.2.gz
-rw-r----- 1 syslog adm 15K 2008-11-10 22:44 kern.log.3.gz

I think the cause of these large files might also have been waking up with a thread scheduling problem:

Nov 20 19:46:08 tatooine kernel: [28959.216716] Call Trace:
Nov 20 19:46:08 tatooine kernel: [28959.216718] [<ffffffff8023e06d>] dequeue_task_idle+0x2d/0x40
Nov 20 19:46:08 tatooine kernel: [28959.216721] [<ffffffff8023c436>] dequeue_task+0x96/0xe0
Nov 20 19:46:08 tatooine kernel: [28959.216723] [<ffffffff8023c4d3>] deactivate_task+0x23/0x30
Nov 20 19:46:08 tatooine kernel: [28959.216726] [<ffffffff80500685>] thread_return+0x108/0x3c3
Nov 20 19:46:08 tatooine kernel: [28959.216728] [<ffffffff8026d12e>] ? sched_clock_idle_wakeup_event+0xe/0x10
Nov 20 19:46:08 tatooine kernel: [28959.216731] [<ffffffff80210ecd>] cpu_idle+0xad/0x110
Nov 20 19:46:08 tatooine kernel: [28959.216734] [<ffffffff804fd735>] start_secondary+0x97/0xc2
Nov 20 19:46:08 tatooine kernel: [28959.216735]
Nov 20 19:46:08 tatooine kernel: [28959.216823] bad: scheduling from the idle thread!
Nov 20 19:46:08 tatooine kernel: [28959.216826] Pid: 0, comm: swapper Tainted: P 2.6.27-7-generic #1
Nov 20 19:46:08 tatooine kernel: [28959.216827]
Nov 20 19:46:08 tatooine kernel: [28959.216828] Call Trace:
Nov 20 19:46:08 tatooine kernel: [28959.216830] [<ffffffff8023e06d>] dequeue_task_idle+0x2d/0x40
Nov 20 19:46:08 tatooine kernel: [28959.216832] [<ffffffff8023c436>] dequeue_task+0x96/0xe0
Nov 20 19:46:08 tatooine kernel: [28959.216835] [<ffffffff8023c4d3>] deactivate_task+0x23/0x30
Nov 20 19:46:08 tatooine kernel: [28959.216837] [<ffffffff80500685>] thread_return+0x108/0x3c3
Nov 20 19:46:08 tatooine kernel: [28959.216840] [<ffffffff8026d12e>] ? sched_clock_idle_wakeup_event+0xe/0x10
Nov 20 19:46:08 tatooine kernel: [28959.216843] [<ffffffff80210ecd>] cpu_idle+0xad/0x110
...etc

Here I am with the very same problem on
2.6.27-7-generic #1 SMP Tue Nov 4 19:33:06 UTC 2008 x86_64 GNU/Linux
                                                               %CPU
4816 root 20 0 8204 620 500 R 61 0.0 528:55.46 dd
 4818 klog 20 0 6540 3248 460 S 40 0.1 358:39.04 klogd
 4761 syslog 20 0 12364 792 596 S 9 0.0 75:23.64 syslogd

ad several gigas of kern.log.0 syslog.0 and messages.0

is there a fix for this bug? or a limit one can set to the size of log files (at least).
Thanks

my messages file is plenty of this
Nov 26 20:48:44 roberto-desktop kernel: [ 9614.884512] ------------[ cut here ]------------
Nov 26 20:48:44 roberto-desktop kernel: [ 9614.884513] WARNING: at /build/buildd/linux-2.6.27/arch/x86/kernel/hpet.c:286 hpet_legacy_next_event+0x73/0x80()
Nov 26 20:48:44 roberto-desktop kernel: [ 9614.884515] Modules linked in: usb_storage libusual i915 drm rfcomm bridge stp bnep sco l2cap bluetooth ppdev autofs4 acpi_cpufreq cpufreq_conservative cpufreq_stats cpufreq_userspace cpufreq_ondemand freq_table cpufreq_powersave pci_slot container sbs sbshc battery ipv6 iptable_filter ip_tables x_tables ac parport_pc lp parport snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event psmouse pcspkr evdev serio_raw snd_seq snd_timer snd_seq_device snd soundcore shpchp pci_hotplug snd_page_alloc video output tpm_infineon tpm tpm_bios wmi button intel_agp ext3 jbd mbcache sr_mod sd_mod cdrom crc_t10dif sg usbhid hid ata_generic ahci pata_acpi ehci_hcd uhci_hcd libata usbcore scsi_mod dock e1000e thermal processor fan fbcon tileblit font bitblit softcursor fuse
Nov 26 20:48:44 roberto-desktop kernel: [ 9614.884549] Pid: 0, comm: swapper Tainted: G W 2.6.27-7-generic #1
Nov 26 20:48:44 roberto-desktop kernel: [ 9614.884550]
Nov 26 20:48:44 roberto-desktop kernel: [ 9614.884550] Call Trace:
Nov 26 20:48:44 roberto-desktop kernel: [ 9614.884552] [<ffffffff8024e9b4>] warn_on_slowpath+0x64/0x90
Nov 26 20:48:44 roberto-desktop kernel: [ 9614.884554] [<ffffffff80245f40>] ? check_preempt_curr_idle+0x10/0x20
Nov 26 20:48:44 roberto-desktop kernel: [ 9614.884556] [<ffffffff803a27ba>] ? __next_cpu+0x1a/0x30
Nov 26 20:48:44 roberto-desktop kernel: [ 9614.884557] [<ffffffff8024535c>] ? find_busiest_group+0x1dc/0x970
Nov 26 20:48:44 roberto-desktop kernel: [ 9614.884559] [<ffffffff802481d7>] ? balance_...

I have the exact same messages as Foice in the messages log.

My logs:
-rw-r----- 1 syslog adm 4.2G 2008-12-05 10:51 syslog
-rw-r----- 1 syslog adm 39G 2008-12-05 08:11 syslog.0
-rw-r----- 1 syslog adm 1.7G 2008-12-04 07:16 syslog.1.gz
-rw-r----- 1 syslog adm 1.7G 2008-12-03 07:40 syslog.2.gz
-rw-r----- 1 syslog adm 1.7G 2008-12-02 07:49 syslog.3.gz
-rw-r----- 1 syslog adm 1.7G 2008-12-01 07:19 syslog.4.gz
-rw-r----- 1 syslog adm 1.7G 2008-11-30 07:05 syslog.5.gz
-rw-r----- 1 syslog adm 1.1G 2008-11-29 06:48 syslog.6.gz

-rw-r----- 1 syslog adm 4.2G 2008-12-05 10:54 kern.log

syslog.0 is huge!!!!

I forgot to post my Ubuntu specs

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=8.10
DISTRIB_CODENAME=intrepid
DISTRIB_DESCRIPTION="Ubuntu 8.10"

I solved! it was an ACPI problem, thus nothing to be changed or set in linux. I had to disable all ACPI features of the BIOS.
I hope it helps.

Is anyone still experiencing this problem with Jaunty?

I've just experienced this after upgrading to Ubuntu Karmic. My syslog and kern.log were each 2.5 GiB, and my messages was 1 GiB. Deleted the files but don't know what caused them to get so large or how to fix it.

I am running Karmic on an old e-machines M51105 with no CPU throttling. When the CPU gets hot, it generates many messages in rapid succession, which in turn, fill up the log files. Should I change the temperature triggers, or is there a way to reduce the frequency of message generation?

Hello all,

I'm running Ubuntu 9.10 and experiencing the same problem. I'm testing the solution of Foice : all ACPI "disabled" in the BIOS.
I will let you know.

After 2 hours running without any ACPI option enabled, the kern.log and messages don't seems to grow any more. So it could be a solution. But perhaps dev-people need information about it, in order to solve it in future release. I will research first the version of my motherboard and Bios. If you say me what to give as other informations I could post it.

here is an example of how far this can and does go.... 20GB of logs

This is a problem in on my Karmic server. In my case, the logs are filled with the following error message:

reset low speed USB device using uhci_hcd and address 17

I only have one USB device so I know exactly what it's complaining about, but I don't want to unplug my mouse either! Please help!

I also have the same problem on lucid 64-bit. Kern.log, syslog.1 and messages are all over 2GB each.

Same problem. I run Ubuntu 10.4 Netbook edition on my new eee pc 1018P. After one month i was not able to get passed the logon screen. From the command line I realized my 250GB disk was packed. After reinstall I now can now see syslog and messages grow a few GBs in just an hour. This scares me.

Is this just a problem with log files or are the log files indicating a more seroius hardware/software issue?

MIkael

This is still a problem for me on 10.04. I have a 16G root partition and my Ubuntu install takes up ~4.5G on this partition. I'm using new drivers for a TV tuner, and it sometimes causes kern.log ans syslog to fill the remaining 11G on the partition in a matter of hours. The system then becomes unstable and will not boot into X when it starts up.

Regardless of how bad my drivers might be, the kernel should not allow the root partition to fill with logs, especially when its 11G of the same two lines. This need to be fixed, but in the meantime i'm also looking for a workaround that makes the partition usable again. Right now, I'm manually running logrotate and then deleting the large files whenever this happens

I've been lucky enough to never experience this bug on my machines, but I'd like to provide my 2 cents about what could cause the issue.
The log rotation system seems to be configured to rotate once a day:

-rw-r----- 1 syslog adm 125K 2011-09-13 17:15 syslog
-rw-r----- 1 syslog adm 86K 2011-09-13 07:40 syslog.1
-rw-r----- 1 syslog adm 3,7K 2011-09-12 07:45 syslog.2.gz
-rw-r----- 1 syslog adm 3,9K 2011-09-11 08:05 syslog.3.gz
-rw-r----- 1 syslog adm 5,0K 2011-09-10 07:35 syslog.4.gz
-rw-r----- 1 syslog adm 4,5K 2011-09-09 08:00 syslog.5.gz
-rw-r----- 1 syslog adm 4,5K 2011-09-08 08:05 syslog.6.gz
-rw-r----- 1 syslog adm 6,9K 2011-09-07 08:00 syslog.7.gz

(this is from my personal machine)

If the rotation takes place once a day regardless of size, the logfile can grow very big if there are many processes which generate a lot of "noise".

HTH

I just installed Xubuntu 11.10 fresh last night and then installed all the updates. Today I got an error that I did not have enough free space so I ran the disk usage analyzer and I have a syslog.1 and kern.log that are both 7 GB each, not sure how this happened but it could really mess things up for a noob who has no idea about this stuff. I am deleting the logs and hopefully it doesn't happen again.

The majority of both log files is this:

Mar 5 17:55:49 dv1000 kernel: [ 6741.464112] [drm:intel_prepare_page_flip] *ERROR* Prepared flip multiple times
Mar 5 17:55:49 dv1000 kernel: [ 6741.464127] [drm:intel_prepare_page_flip] *ERROR* Prepared flip multiple times
Mar 5 17:55:49 dv1000 kernel: [ 6741.464142] [drm:intel_prepare_page_flip] *ERROR* Prepared flip multiple times

Cross linking a bug that has a solution for me - https://bugs.launchpad.net/ubuntu/+source/linux/+bug/765813 ([drm:intel_prepare_page_flip] *ERROR* Prepared flip multiple times every frame with intel graphics)

I also have this problem running Xubuntu 12.04.1

Log files have bloated to 92% of my 20G partition in a few days.

I may also try the solution from Foice : Disable the ACPI in the BIOS.
but first I am going to try viewing the files to see what all the messages are.

Fixed.. by running

sudo add-apt-repository ppa:glasen/intel-driver
sudo apt-get update
sudo apt-get install xserver-xorg-video-intel

updated driver seems to be doing the job!

kubuntu 11.10 , Soluzione momentanea , non definitiva ...Serve ad azzerare i valori ingranditi di kern.log e syslog .

france1@france1:~$ sudo -i ...comando
[sudo] password for france1: ...inserire password
root@france1:~# cd /var/log ...comando
root@france1:/var/log# ls -lhS ...comando controllo valori(se sono alti ,procedere cos'ì)
root@france1:/var/log# :>kern.log ...comando
root@france1:/var/log# :>syslog ...comando
root@france1:/var/log#
root@france1:/var/log# ls -lhS ...comando controllo valori

RISOLTO:
Il problema mi era nato quando un giorno ho installato un "salvaschermo Open GL. All'inizio non capivo che fosse questa la causa perchè insieme avevo fatto altre piccole modifiche al pc. Cosa c'è di strano ad installare un "Salvaschermo Open GL"? Dovete fare attenzione a che tipo di "scheda video" avete. Il mio è un pc datato,con lui anche la scheda incriminata, quindi di conseguenza non supporta questa configurazione. SOLUZIONE ? Semplicemente ho disinstallato il Salvaschermo colpevole. Spero di essere stato di aiuto.

Kubuntu 14.04 still hangs on boot when the disk is full.

My disk was full and the result was that after the normal login the system simply hangs. After switching to the console and removing some files I could restart and login.

I think, that if the disk is full, kubuntu should popup with a message and (more important) switch to the console. And should create a warning if the disk space is getting low.

Still happening on 20.04.

I think we should limit this bug to :

- Create a mechanism that would prevent rsyslogd to fillup disk.
- This would make log turn as soon as there is only XX GB left

I also have the same problem. Here is my Computer description.

https://paste.ubuntu.com/p/cMB7PkjsZs/

I've asked about the problem on askubuntu, which explains my problem clearly.

https://askubuntu.com/questions/1265849/my-system-filling-gigabytes-of-log-files-within-minutes