log split odyssey

[Alejandro Fernandez <al...@e-group.org>]

Hi!

I'm trying to figure out the best way to split a logfile for a statistics system which needs to take very high numbers of hits per minute - it's for an entire ISP and hosting company to use everywhere, so it's expecting 2000 hits a second at most peak times. 

The split is based either on the referrer (logs are like logs for an ad server: the referrer is the site we host, so the stats need to go to that site's owner), or on a site id encoded in the query string of each request.

I thought of a few ways to do this, but I wonder if I could get some suggestions... 

1) Split with log-rotate every million logfile lines or so, then break it up with a script.
2) Split using the following apache commands:

        LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" common
        SetEnvIf Referer spome\.website\.net mydomain-log
        CustomLog /var/log/httpd/statsserver/mydomain_log common env=mydomain-log
        CustomLog /var/log/httpd/pixel-access_log common env=!mydomain-log

all of which are inside the virtual host for my statistics site.
3) Use apache's pipe-log-to-external process feature.
4) Each hit goes to a persistently open cgi in c which writes to it's own logfile - comparing sids or referrers to what it gets periodically from a file or a database.

Can anyone suggest what is the best/easiest method for achieving this easily, or even a difficult method, as long as it's proven to work under high loads...

Thanks!

Ale

-- 
Alejandro Fernandez
Electronic Group Interactive
--+34-65-232-8086--

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: log split odyssey

[Joshua Slive <jo...@slive.ca>]

On Fri, 26 Apr 2002, Alejandro Fernandez wrote:
>
> 1) Split with log-rotate every million logfile lines or so, then break
> it up with a script.

> 3) Use apache's pipe-log-to-external process feature.

I would use one of these, or even just use the standard Apache logging
features with the pipe and do a graceful restart every couple hours to
rotate out the logs.  The goal should be to have the webserver as little
involved as possible.

Joshua.


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org