See Ticket #2866 for some past work.
We use logrotate with the copytruncate option. To rotate a log file, it does:
This is not an atomic operation, so more logging data might be written to the log file between steps 1 and 2, and we'll never see it. It's gone for good. This isn't a big case, but it certainly can cause confusion when troubleshooting.
How could we solve this data loss? Well, it's a two-step process. First, we turn off copytruncate in logrotate, and it will simply rename files. Second, we fix all of the logging that we break. Step 2 is harder.
One possibility for a fix is cronolog. I (Alain) have no direct experience with it, I just found it when researching the problem. It can fix logs like Tomcat's catalina.out. Catalina.out is created (by catalina.sh) by redirecting stdout and stderr to a file. If we merely rename the file, we haven't done a log rotation. So instead we can pipe to cronolog which will change what files it rotates to over time. This is a nice, simple solution, except that we now have an extra program. What if it dies? I'm not sure, so I haven't pursued this solution.
This page suggests a Tomcat specific change to get it to redirect webapps stdout and stderr to different files that can be rotated. I don't faully understand the details, but it's a possible approach.
The Tomcat FAQ has some suggestions for rotating catalina.out. Mostly they boil down to "don't let webapps write to stdout!" and "use cronolog".
Note: We wrap the invocation of logrotate in a program called vdt-rotate-logs. This wrapper passes two needed parameters: the state file (so we're distinct from the system's state file) and the configuration file (so we're distinctd from the system's configuration file). Both are kept in $VDT_LOCAITON.
| Software | Logs | Comments |
| Apache |
apache/logs/access_log apache/logs/error_log apache/logs/mod_jk.log apache/logs/ssl_request_log |
Rotation by logrotate |
| BDII |
bdii/var/bdii.log bdii/var/bdii-fwd.log |
Rotation by logrotate, but configured by bdii startup script, system-wide logrotate. Since almost no VDT customers use the BDII, I'm not concerned about changing it. |
| Bestman | vdt-app-data/bestman/logs/* | Rotation done internally by Bestman (according to Alex Sim) |
| CEmon | tomcat/v5/logs/glite-ce-monitor.log | Rotation done internally by log4j |
| Condor |
condor/local. |
Rotation done internally by Condor |
| EDG mkgridmap | edg/log/edg-mkgridmap.log | Rotation by logrotate. Log provided by VDT wrapper script, not edg-mkgridmap |
| Globus Container |
globus/var/container.log globus/var/container-real.log |
container.log: only startup messages, recreated each startup so
it doesn't grow indefinitely container-real.log, rotated by log4j in globus/container-log4j.properties. Ticket #1937 |
| Globus GridFTP |
globus/var/log/gridftp.log globus/var/log/gridftp-auth.log | Rotation done by logrotate |
| Globus RLS | Everything is in syslog, no local logfiles | http://www.globus.org/toolkit/data/rls/rls_faq.html#logfile |
| GUMS 1.2 | tomcat/v55/logs/gums* | Rotation done internally by log4j, not logrotate |
| jClarens | jclarens.log jclarens_rendezvous.log jclarens_traffic.log | Rotated internally by log4j |
| MonaLisa | Lots of logs. | I'm confused. Maybe it rotates. |
| PRIMA | globus/var/globus-gatekeeper.log | Same as gatekeeper |
| PRIMA-GT4 | globus/var/container-real.log | Same as gatekeepe and containerr |
| Squid |
squid/var/logs/access.log squid/var/logs/cache.log |
Rotated by logrotate |
| Tomcat |
tomcat/v5/logs/catalina.out tomcat/v55/logs/catalina.out |
Rotated by logrotate |
| VOMS | glite/log/voms.VO | Rotated by logrotate |
| VOMS-Admin | tomcat/v5/logs/voms-admin.VO.log | Rotated by logrotate, could be done by log4j! |
| VOMRS | Rotated internally by log4j (see in vomrs/var/etc/vomrs_<VO>: log4j.properties and log4j_webui.properties) | |
| vdt-update-certs |
vdt/var/log/vdt-update-certs.log vdt/var/log/vdt-update-certs.log.system_output |
Done by logrotate, but configured on the fly by vdt-update-certs because the log file location can change. |
If you want to mess with logrotate: