You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Josh Wyatt <jo...@hcssystems.com> on 2005/01/05 16:53:56 UTC
High performance apache logging
Hi all,
I'm interested in hearing anyone's thoughts on better performing logging
mechanisms for apache. My environment consists of four servers, load-balanced
using round-robin DNS (which works very well). We are currently logging using
just the standard local file logging, and it is pushing system load pretty high
due to our load. I'm leaning toward something using a perl script, a big
logserver, and pipes...?
Thanks in advance,
Josh
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: High performance apache logging
Posted by Joshua Slive <js...@gmail.com>.
On Wed, 05 Jan 2005 10:53:56 -0500, Josh Wyatt
<jo...@hcssystems.com> wrote:
> Hi all,
>
> I'm interested in hearing anyone's thoughts on better performing logging
> mechanisms for apache. My environment consists of four servers, load-balanced
> using round-robin DNS (which works very well). We are currently logging using
> just the standard local file logging, and it is pushing system load pretty high
> due to our load. I'm leaning toward something using a perl script, a big
> logserver, and pipes...?
Ouch. Local file logging is pretty-much always going to be the
fastest unless you have serious disk-write-speed constraints. You can
consider turning on buffered logging (see the source code to
mod_log_config.c).
Joshua.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
[users@httpd] Re: High performance apache logging
Posted by Nick Kew <ni...@webthing.com>.
On Wed, 5 Jan 2005, Josh Wyatt wrote:
> I'm interested in hearing anyone's thoughts on better performing logging
> mechanisms for apache. My environment consists of four servers, load-balanced
> using round-robin DNS (which works very well). We are currently logging using
> just the standard local file logging, and it is pushing system load pretty high
> due to our load.
Hmmm. Logging to local disc sounds like an unlikely culprit, unless it's
really something silly like an NFS mount.
Anyway, google for mod_log_spread. From memory, I think it makes some
claims about being good for multiple load-balanced servers under high
load.
--
Nick Kew
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
[users@httpd] Re: High performance apache logging
Posted by Martin Sperl <ap...@martin.sperl.org>.
Hi Josh!
I do not think that file-logging should be able to push the system load
that high - that might be more of a general IO problem to push the
content from the disk to the net. In this case you probably have not got
enough memory for disk-caching...
We are using a specialized logging script on our 5 webservers, that can
filter lines and modify fields using regular expressions before logging
to a central mysql database.
My estimate is that we are having about 8M Requests per Webserver and
day and we are logging the "essential" 1.5M Lines per webserver and day
(not logging certain application requests and all the GIF/JPG/... Requests).
This does not provide to much system load on our machines and also due
to a design of the logging-script the actual writing to the mysql is
buffered so that the DB-Server may be locked from time to time for
"online" reporting purposes, like "Which IP Network is generating most
requests" (e.g: google-bot), "where are application-performance
bottlenecks", what the most often requested scripts,...
I have to warn you that if you do not take care in the design of the
script the logging easily can become a bottleneck for the webserver...
With our older version of the script there were times when almost all
apache processes where waiting to log a request (as can be seen in
/status...) so you have to take care, that such bottleneck situations
may NOT occure... So buffering to the local disk is required and an
asyncronous process that pushes the data to the central server - the
timestamp naturally is fixed by apache so delayed logging does not give
you problems, except your "realtime" data may just be a little off...
If you are logging to a database please also keep in mind the size of
the Tables! So some "compression" by using in the central tyble only
foreign keys to separate tables for URL, IP,QUERY,Referer,Agent,...
We can keep 13 Days of such logfiles in a mysql-database of a total of
currently 15GB, but have to leave space, as the "Query" table is
expanding rapidly even though there is some rewriting applied already...
If there is interrest, then I can see if we can publish the script
somewhere...
Cheers,
Martin
P.s: We also have some scripts that allow the near-realtime generation
of RRD Graphs from the database to see how some marketing campaigns work...
Another neat thing that we have implemented for a specific predominantly
WAP-website, is a script that can find abnormal traffic situations and
alert them to the operations team... (This way we have found several times
failing WAP gateways and when the customer called, that there was a
problem we could immediately say: "Yes, we know - your WAP Gateways
are currently having problems...") But these results have to be taken
with a grain of salt...
Josh Wyatt wrote:
> Hi all,
>
> I'm interested in hearing anyone's thoughts on better performing logging
> mechanisms for apache. My environment consists of four servers,
> load-balanced
> using round-robin DNS (which works very well). We are currently
> logging using
> just the standard local file logging, and it is pushing system load
> pretty high
> due to our load. I'm leaning toward something using a perl script, a big
> logserver, and pipes...?
>
> Thanks in advance,
> Josh
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org