suggestion for automatic log file rotation

[Andrew Ford <an...@icarus.demon.co.uk>]

I know that this is too late now for 1.2, but I have been asked by a
couple of my clients whether there was any way of automatically
logging errors and accesses to log files with names that include
elements of the date and time.  My idea is that if the log file name
does not start with a "|", and looks like an strftime string, then
strftime should be used to format the file name and a new file should
be used when strftime would generate a different string.  For example,
today the directive:

  ErrorLog "logs/errors-%Y%m%d.log"

would log to "logs/errors-19961123.log" and at midnight the file would
be closed and the new file "logs/errors-19961124.log" would be used.
This functionality would obviate the need for the external
"rotatelogs" program.

The approach I am considering is on parsing the TransferLog or
ErrorLog directive to check for strftime % specifiers and to note the
smallest time element used.  For each separate log file there should
be two new fields; one giving the next time at which a new log file
should be opened and the other storing the interval between log file
changes.  When logging a message the current time is first compared
with the next time field and if the current time is later than that
time then the current log file is closed, a new log file opened and a
new next time calculated.  The message is then written to the file.  
(the next time field should be initialized to zero so that the first
message written causes the log file to be opened).

There are a number of potential problems:

* strftime strings could generate file names containing spaces (user's
  problem).

* generated log files could require new directories to be created
  (again the user's problem -- if s/he wants logs stored in
  directories by month or year, then s/he will have to ensure that the
  directories are created in time).

* the period between the start times of successive log files is not
  necessarily constant (this just has to be taken into account when the
  smallest strftime specifier refers to the month or year, although
  leap seconds and changes to and from daylight saving time could
  throw things out).

Looking at the code the main implementation problem seems to be coping
with multiple virtual hosts that use the same log file.

My question is what do people think of the suggestion?

Andrew
-- 
Andrew Ford                             Email:  andrew@icarus.demon.co.uk
Independent Software Consultant         WWW:    http://www.nhbs.co.uk/aford/
"Brittany", Wells Road,                 Tel:    +44 1452 770836
Eastcombe, Stroud, GL6 7EE, GB          Fax:    +44 1452 770835

Re: suggestion for automatic log file rotation

[Nathan Neulinger <nn...@umr.edu>]

>My question is what do people think of the suggestion?
>
>you need to take care of cases where people would want to use
>date stamped filenames but they don't rotate their logs wach day.
>
>e.g. there's a problem if someone uses this feature to name their
>logfiles but only rotates them once a week/month/etc, or even at an hour
>other than midnight. If the server is restarted on a different day, it'll
>insist on opening new logs when the intention was that it continued with
>the old ones.
>
>Renaming the logfiles when you're done with them is simple to do.

I personally don't see the big deal about processing the logs with another
utility.

I have something set up for all of our servers at our site that
periodically (hourly in some cases) shuffles the logs... Our setup
processes the logs into:

/some/dir/www.server.name:port/access-YYYY-MMM
/some/dir/www.server.name:port/error-YYYY-MMM

Those files are automatically gzipped, the log processor handles either case.

That central dir is then automatically processed into html files for the
stats. The data files can be removed with the stats html remaining.

There is no need to kill -HUP or do anything to the server, all you need to
do is null out the file. The next write by the server will pick up where it
left off. Basically, it just creates a file with a hole in it. This is not
really a problem, as the log processor will just skip over the nulls. (And
with some minor effort can be made to ignore the holes alltogether.)

It works well for us anyway.

-- Nathan

------------------------------------------------------------
Nathan Neulinger                  Univ. of Missouri - Rolla
EMail: nneul@umr.edu                  Computing Services
WWW: http://www.umr.edu/~nneul      SysAdmin: rollanet.org