See Ticket #2866 for some past work.
We use logrotate with the copytruncate option. To rotate a log file, it does:
This is not an atomic operation, so more logging data might be written to the log file between steps 1 and 2, and we'll never see it. It's gone for good. This isn't a big case, but it certainly can cause confusion when troubleshooting.
How could we solve this data loss? Well, it's a two-step process. First, we turn off copytruncate in logrotate, and it will simply rename files. Second, we fix all of the logging that we break. Step 2 is harder.
One possibility for a fix is cronolog. I (Alain) have no direct experience with it, I just found it when researching the problem. It can fix logs like Tomcat's catalina.out. Catalina.out is created (by catalina.sh) by redirecting stdout and stderr to a file. If we merely rename the file, we haven't done a log rotation. So instead we can pipe to cronolog which will change what files it rotates to over time. This is a nice, simple solution, except that we now have an extra program. What if it dies? I'm not sure, so I haven't pursued this solution.
This page suggests a Tomcat specific change to get it to redirect webapps stdout and stderr to different files that can be rotated. I don't faully understand the details, but it's a possible approach.
The Tomcat FAQ has some suggestions for rotating catalina.out. Mostly they boil down to "don't let webapps write to stdout!" and "use cronolog".
Note: We wrap the invocation of logrotate in a program called vdt-rotate-logs. This wrapper passes two needed parameters: the state file (so we're distinct from the system's state file) and the configuration file (so we're distinctd from the system's configuration file). Both are kept in $VDT_LOCAITON.
|Rotation by logrotate|
|Rotation by logrotate, but configured by bdii startup script, system-wide logrotate. Since almost no VDT customers use the BDII, I'm not concerned about changing it.|
|Bestman||vdt-app-data/bestman/logs/*||Rotation done internally by Bestman (according to Alex Sim)|
|CEmon||tomcat/v5/logs/glite-ce-monitor.log||Rotation done internally by log4j|
||Rotation done internally by Condor|
|EDG mkgridmap||edg/log/edg-mkgridmap.log||Rotation by logrotate. Log provided by VDT wrapper script, not edg-mkgridmap|
container.log: only startup messages, recreated each startup so
it doesn't grow indefinitely
container-real.log, rotated by log4j in globus/container-log4j.properties. Ticket #1937
|Rotation done by logrotate|
|Globus RLS||Everything is in syslog, no local logfiles||http://www.globus.org/toolkit/data/rls/rls_faq.html#logfile|
|GUMS 1.2||tomcat/v55/logs/gums*||Rotation done internally by log4j, not logrotate|
|jClarens||jclarens.log jclarens_rendezvous.log jclarens_traffic.log||Rotated internally by log4j|
|MonaLisa||Lots of logs.||I'm confused. Maybe it rotates.|
|PRIMA||globus/var/globus-gatekeeper.log||Same as gatekeeper|
|PRIMA-GT4||globus/var/container-real.log||Same as gatekeepe and containerr|
|Rotated by logrotate|
|Rotated by logrotate|
|VOMS||glite/log/voms.VO||Rotated by logrotate|
|VOMS-Admin||tomcat/v5/logs/voms-admin.VO.log||Rotated by logrotate, could be done by log4j!|
|VOMRS||Rotated internally by log4j (see in vomrs/var/etc/vomrs_<VO>: log4j.properties and log4j_webui.properties)|
|Done by logrotate, but configured on the fly by vdt-update-certs because the log file location can change.|
If you want to mess with logrotate: