Note: This web site is only kept up to date for OSG Software 1.2 (VDT 2.0.0). If you are looking for information for the most recent release, the RPM-based OSG Software 3.0, please see the OSG documentation web site

Thoughts on log rotation

How logrotate is configured; data loss

See Ticket #2866 for some past work.

We use logrotate with the copytruncate option. To rotate a log file, it does:

  1. Copy entire log file
  2. Truncate log file

This is not an atomic operation, so more logging data might be written to the log file between steps 1 and 2, and we'll never see it. It's gone for good. This isn't a big case, but it certainly can cause confusion when troubleshooting.

How could we solve this data loss? Well, it's a two-step process. First, we turn off copytruncate in logrotate, and it will simply rename files. Second, we fix all of the logging that we break. Step 2 is harder.

One possibility for a fix is cronolog. I (Alain) have no direct experience with it, I just found it when researching the problem. It can fix logs like Tomcat's catalina.out. Catalina.out is created (by catalina.sh) by redirecting stdout and stderr to a file. If we merely rename the file, we haven't done a log rotation. So instead we can pipe to cronolog which will change what files it rotates to over time. This is a nice, simple solution, except that we now have an extra program. What if it dies? I'm not sure, so I haven't pursued this solution.

This page suggests a Tomcat specific change to get it to redirect webapps stdout and stderr to different files that can be rotated. I don't faully understand the details, but it's a possible approach.

The Tomcat FAQ has some suggestions for rotating catalina.out. Mostly they boil down to "don't let webapps write to stdout!" and "use cronolog".

Note: We wrap the invocation of logrotate in a program called vdt-rotate-logs. This wrapper passes two needed parameters: the state file (so we're distinct from the system's state file) and the configuration file (so we're distinctd from the system's configuration file). Both are kept in $VDT_LOCAITON.

State of log rotation in VDT 1.8.1

Software Logs Comments
Apache apache/logs/access_log
apache/logs/error_log
apache/logs/mod_jk.log
apache/logs/ssl_request_log
Rotation by logrotate
BDII bdii/var/bdii.log
bdii/var/bdii-fwd.log
Rotation by logrotate, but configured by bdii startup script, system-wide logrotate. Since almost no VDT customers use the BDII, I'm not concerned about changing it.
Bestman vdt-app-data/bestman/logs/* Rotation done internally by Bestman (according to Alex Sim)
CEmon tomcat/v5/logs/glite-ce-monitor.log Rotation done internally by log4j
Condor condor/local./log Rotation done internally by Condor
EDG mkgridmap edg/log/edg-mkgridmap.log Rotation by logrotate. Log provided by VDT wrapper script, not edg-mkgridmap
Globus Container globus/var/container.log
globus/var/container-real.log
container.log: only startup messages, recreated each startup so it doesn't grow indefinitely
container-real.log, rotated by log4j in globus/container-log4j.properties. Ticket #1937
Globus GridFTP globus/var/log/gridftp.log
globus/var/log/gridftp-auth.log
Rotation done by logrotate
Globus RLS Everything is in syslog, no local logfiles http://www.globus.org/toolkit/data/rls/rls_faq.html#logfile
GUMS 1.2 tomcat/v55/logs/gums* Rotation done internally by log4j, not logrotate
jClarens jclarens.log jclarens_rendezvous.log jclarens_traffic.log Rotated internally by log4j
MonaLisa Lots of logs. I'm confused. Maybe it rotates.
PRIMA globus/var/globus-gatekeeper.log Same as gatekeeper
PRIMA-GT4 globus/var/container-real.log Same as gatekeepe and containerr
Squid squid/var/logs/access.log
squid/var/logs/cache.log
Rotated by logrotate
Tomcat tomcat/v5/logs/catalina.out
tomcat/v55/logs/catalina.out
Rotated by logrotate
VOMS glite/log/voms.VO Rotated by logrotate
VOMS-Admin tomcat/v5/logs/voms-admin.VO.log Rotated by logrotate, could be done by log4j!
VOMRS Rotated internally by log4j (see in vomrs/var/etc/vomrs_<VO>: log4j.properties and log4j_webui.properties)
vdt-update-certs vdt/var/log/vdt-update-certs.log
vdt/var/log/vdt-update-certs.log.system_output
Done by logrotate, but configured on the fly by vdt-update-certs because the log file location can change.

Messing with logrotate

If you want to mess with logrotate:

  1. Run vdt/bin/vdt-rotate-logs
    Note that it keeps the state in $VDT_LOCATION
  2. Edit the date for the log file you care about in vdt/var/logrotate.state
  3. Re-run vdt-rotate-logs and watch the file get rotated