You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Dirk-Willem van Gulik <di...@webweaving.org> on 2002/08/21 16:52:28 UTC

Re: Log rotation/Logging to a database?


On Wed, 21 Aug 2002, Jacob Coby wrote:

> Yikes.  Last I checked, Apache has a 25mb limit on log files, maybe that was
> an artificial limit in my config?  All i know is that apache puked once the
> logs got that big :)  That was my kick in the ass to get logrotations going.

There is no limit in apache, nor will apache puke on large limits; it is
in fact not even aware of the size of the logfile... except of course for
those limites imposed by your operating system.

Modern unixes, such as solaris or freebsd, handle well gigabytes no
trouble - even when using a mundane system like UFS.

Another cause can be fragemtation, running out of inode's or using a type
of file system which is inherently not suitable for the write/append only
style writing of a log file. But those are getting increasingly rare.

(Having said that; I've seen some vendors, Mandrake was one I think, who
ship with a default /var partition of 30-50Mb; yet have the apache log
file there by default - and no rotation.. that of course is asking for
trouble big time...

> Doing this is likely to use more resources than just rotating the logs.
> Querying, deleting, writing a couple hundred thousand rows to disk takes
> forever.  I recently culled 200000 rows from a table with 5mil rows, and it
> took about 30 min just to do the delete.

Eek - you want to avoid this :-) do not forget - logging tables are keyed
on a temporal basis.

So should you use a database; the common method is to rotate on table's;
i.e.  you have, say 7, tables; one for each day. At any given moment the
table of the day is used for writing; the 6 other tables are made
available in one 'virtual' multietable view/join which makes it look like
one. There are some well established further tune techiques if you need to
have todays table to be part of the view too.

At rotate time you simply create a totally new table for the next day; and
drop the oldest table and then recreate the virtual multi table view for
the dayly analysis work. You typically do the table drop last; or in such
a way that you do not suffer if it blocks long.

And in larger installs you want to dicuss things like backup schedules
with the DBA.

But then again - the DBA and you will have a lot to coordinate anyway if
you are going to saddle him or her with a write/append/insert only
database :-) with several 100's transactions per second :-)

Any good book on oracle/postgress/db2/sybase optimization will show you
examples of this - as this is what needs to happen in company order
systems as well.

Dw
-- 
Dirk-Willem van Gulik


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org