Improve default journal rolling

Bug #828224 reported by Derek_
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenVista/GT.M Integration
Fix Released
Low
Derek_

Bug Description

The current default schedule, set in backups.conf and journals.conf, is to backup and purge journals daily, and to keep 3 backup files and 3 journal files.

These are not good defaults, because a production database will typically go through more than one journal file in a day, and the journal files should extend back as far as the oldest backup.

Another problem is that journal files are typically being rolled at their maximum size of 4 GB, which is awkwardly large, e.g. for moving or extracting. And since the journals roll due to size, they roll depending on database activity rather than a timed schedule, so predicting how many files need to be kept for the associated daily backups is unreliable.

A much better configuration is to set a cron job to roll the journals hourly, which rolls them long before they reach 4 GB. This ensures 24 journal files per day plus an additional file rolled at the time of the backup. Then num_journals can be reliably set by multiplying the number of daily backups by 25. Another advantage of the hourly rolling is that the journals are then divided in a clear chronology, so for example, to see what happened at a particular time you can just extract the journal file for that hour, and you can see how journal growth varies by the time of day.

To implement this by default, an hourly cron job for ovswitchjournals needs to be added to /etc/cron.d/openvista by ovinstanceadd, and num_journals needs to be set to 75 in /usr/share/openvista/journals.conf.

Derek_ (derek-name)
description: updated
description: updated
description: updated
Changed in openvista-gtm-integration:
status: New → Confirmed
Revision history for this message
bhaskar (bhaskar) wrote :

Odds and ends that may be useful...

Rolling the journals hourly guarantees at least 24 journal files a day, but it could be more if in one hour the application generates enough data to roll the journal file.

You can set a maximum journal file size of less than 4GB, if 4GB journal files are too unwieldy.

When a journal file is rolled, the former journal file is renamed with a timestamp.

Revision history for this message
Derek_ (derek-name) wrote :

Thanks. Right now the hourly rolling would keep all of our customers' journal files at much less than 4 GB, and multiple files for an hour would probably indicate something going wrong. I do like that the journals would remain hourly up to that limit.

If we see the journals for an instance legitimately getting closer to 4 GB, or even exceeding 2 GB, within an hour, then I would probably adjust the timing to half-hour periods. That is part of my reason for adding a cron job line to /etc/cron.d/openvista rather than adding a script in /etc/cron.hourly/.

Revision history for this message
Jon Tai (jontai) wrote :

The original rationale for keeping the defaults low was to remain compatible with test/demo systems. A 1G database with 3 backups times 3 databases is already 9G of backups. If the test/demo system doesn't have enough disk space, it will fail about 3 days after initial installation, even if the user has not added any data (or even logged in!). To make matters worse, test/demo systems are usually partitioned with everything on a shared "/" filesystem, and 2 of the 3 days could be a weekend.

The assumption was that these defaults would be changed on production systems with hundreds of GB of disk available for backups.

Revision history for this message
Derek_ (derek-name) wrote :

I don't plan to change the default number of backup files retained, and that will still be something to scale for the actual production system.

This change will primarily affect how the journal data is divided. It will change how much data is retained by making it accurately match the number of days needed. In test systems, there should be very little journaling, so there would probably just be 75 very small files for the 3 days retained.

This will raise the limit on how much data could be written before any automatic purging occurs. Potentially, you could have nearly 300 GB of journal files before any purging occurs. Whereas the original defaults would always purge down to below 12 GB each day.

Derek_ (derek-name)
Changed in openvista-gtm-integration:
status: Confirmed → Fix Committed
Derek_ (derek-name)
Changed in openvista-gtm-integration:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.