You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Ray Van Dolson <rv...@esri.com> on 2010/04/08 22:52:51 UTC
[users@httpd] Centralized logging.
So how are people out there in mailing list land handling centralized
logging?
I'd like to mirror my web farm's logfiles on a central server, but want
to be sure to preserve the W3C-ness so it's easy to run awstats, etc
against them from off this central box.
The options I see so far are:
- Piping to syslog via external process (perl script, etc).
* Downside being that we end up with syslog timestamps
- Pipe to something like netcat
* Custom-ish / hacky
- Fire off some sort of synchronization process from central box to
pull in logs (rsync) periodically.
* Asynchronous -- there's a somewhat large window of time where
logs will be missing on the internal server.
- Log directly to shared, remote file server (via NFS)
* Log server network issues or technical problems could affect
Apache servers
How do you guys typically deal with this?
Ray
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Centralized logging.
Posted by Sean Conner <sp...@conman.org>.
It was thus said that the Great Ray Van Dolson once stated:
> On Thu, Apr 08, 2010 at 02:18:28PM -0700, Jason Nunnelley wrote:
> > I'm using syslog-ng. It does the job.
> >
> > A nice little rsync script is nice, but you're still storing log files
> > on the individual servers until you run some rsync and delete script.
> > This can be risky if resources are limited or machines are ephemeral.
> >
> > NFS has its own issues as well. You've got to ask yourself "What's my
> > plan B" and write that into your logging config/scripts.
>
> Thanks Jason. We're using syslog-ng as well.
>
> So how exactly are you implementing this? Apache procs piping to a
> perl process which shunts things out to your syslog server?
I just tested this, and if you are on a Unix-like system, see if you have
the logger program. This is working for me:
<VirtualHost [fc00::1:3]:80>
ServerName lua.roswell.area51
ServerAdmin sean@conman.org
DocumentRoot /home/spc/web/lua/htdocs
CustomLog "|/usr/bin/logger" combined
ErrorLog syslog:daemon
LogLevel debug
# ...
</VirtualHost>
What gets logged is (and this is using a custom syslogd I wrote, so if the
format looks a bit odd to you, that's why):
/dev/log | logger | user notice | Apr 08 20:55:18 | ::1 - - [08/Apr/2010:20:55:18 -0400] "GET / HTTP/1.1" 304 - "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"
/dev/log | logger | user notice | Apr 08 20:55:38 | ::1 - - [08/Apr/2010:20:55:38 -0400] "GET /quot.html HTTP/1.1" 404 207 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"
/dev/log | logger | user notice | Apr 08 20:55:43 | ::1 - - [08/Apr/2010:20:55:43 -0400] "GET /quote.html HTTP/1.1" 200 395 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"
/dev/log | logger | user notice | Apr 08 20:55:43 | ::1 - - [08/Apr/2010:20:55:43 -0400] "GET /quote.css HTTP/1.1" 200 177 "http://lua.roswell.area51/quote.html" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"
> Do you do any post/pre-processing for timestamp duplication? We want
> to try and have the logs on the syslog server look exactly like they
> did on the origin server (ie no syslog timestamps, no syslog
> facility/priority stuff and no "hostname").
I don't use syslog-ng, so I can't say for sure, but since the actual
Apache logline is logged, it looks like it would be easy to strip the syslog
specific stuff and use just the Apache generated data.
-spc
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Centralized logging.
Posted by Ray Van Dolson <rv...@esri.com>.
On Thu, Apr 08, 2010 at 02:18:28PM -0700, Jason Nunnelley wrote:
> I'm using syslog-ng. It does the job.
>
> A nice little rsync script is nice, but you're still storing log files
> on the individual servers until you run some rsync and delete script.
> This can be risky if resources are limited or machines are ephemeral.
>
> NFS has its own issues as well. You've got to ask yourself "What's my
> plan B" and write that into your logging config/scripts.
Thanks Jason. We're using syslog-ng as well.
So how exactly are you implementing this? Apache procs piping to a
perl process which shunts things out to your syslog server?
Do you do any post/pre-processing for timestamp duplication? We want
to try and have the logs on the syslog server look exactly like they
did on the origin server (ie no syslog timestamps, no syslog
facility/priority stuff and no "hostname").
Ray
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Centralized logging.
Posted by Jason Nunnelley <ja...@jasonn.com>.
I'm using syslog-ng. It does the job.
A nice little rsync script is nice, but you're still storing log files
on the individual servers until you run some rsync and delete script.
This can be risky if resources are limited or machines are ephemeral.
NFS has its own issues as well. You've got to ask yourself "What's my
plan B" and write that into your logging config/scripts.
On 4/8/10 3:52 PM, Ray Van Dolson wrote:
> So how are people out there in mailing list land handling centralized
> logging?
>
> I'd like to mirror my web farm's logfiles on a central server, but want
> to be sure to preserve the W3C-ness so it's easy to run awstats, etc
> against them from off this central box.
>
> The options I see so far are:
>
> - Piping to syslog via external process (perl script, etc).
> * Downside being that we end up with syslog timestamps
> - Pipe to something like netcat
> * Custom-ish / hacky
> - Fire off some sort of synchronization process from central box to
> pull in logs (rsync) periodically.
> * Asynchronous -- there's a somewhat large window of time where
> logs will be missing on the internal server.
> - Log directly to shared, remote file server (via NFS)
> * Log server network issues or technical problems could affect
> Apache servers
>
> How do you guys typically deal with this?
>
> Ray
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See<URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
--
Jason A. Nunnelley
+1 2562971652
http://www.google.com/profiles/imjasonn
[Member Tekany, LLC]
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Centralized logging.
Posted by Mark Watts <m....@eris.qinetiq.com>.
On Thu, 2010-04-08 at 13:52 -0700, Ray Van Dolson wrote:
> So how are people out there in mailing list land handling centralized
> logging?
>
> I'd like to mirror my web farm's logfiles on a central server, but want
> to be sure to preserve the W3C-ness so it's easy to run awstats, etc
> against them from off this central box.
>
> The options I see so far are:
>
> - Piping to syslog via external process (perl script, etc).
> * Downside being that we end up with syslog timestamps
> - Pipe to something like netcat
> * Custom-ish / hacky
> - Fire off some sort of synchronization process from central box to
> pull in logs (rsync) periodically.
> * Asynchronous -- there's a somewhat large window of time where
> logs will be missing on the internal server.
> - Log directly to shared, remote file server (via NFS)
> * Log server network issues or technical problems could affect
> Apache servers
>
> How do you guys typically deal with this?
I log to a fifo with Apache, then have syslog-ng picking it up and
firing it to a central syslog-ng server.
Local system:
mkfifo /var/log/httpd/weblogs_vhost.fifo
Local Apache Vhost:
CustomLog "/var/log/httpd/weblogs.fifo" combined
Local syslog-ng:
source s_apache_vhost { pipe("/var/log/httpd/weblogs_vhost.fifo"); };
destination d_weblogs_vhost { tcp("log-server" port("5140") template("$MESSAGE\n") template_escape(no)); };
log { source(s_apache_vhost); destination(d_weblogs_vhost); };
Remote syslog-ng:
source s_weblogs_vhost { tcp(ip(0.0.0.0) port(5140)); };
destination d_weblogs_vhost { file("/export/syslog/httpd/<vhost>/$YEAR/$MONTH/$DAY/access.log" template("$MESSAGE\n") template_escape(no)); };
log { source(s_weblogs_vhost); destination(d_weblogs_vhost); };
I usually dedicate a syslog-ng port to each vhost so I don't need to
setup any filters. YMMV.
The template and template_escape stuff ensures that the log entry on the
log servers disk is the exact same format as you'd get directly from
Apache using normal logs. You won't get syslog timestamps with this.
Mark.
--
Mark Watts BSc RHCE MBCS
Senior Systems Engineer, Managed Services Manpower
www.QinetiQ.com
QinetiQ - Delivering customer-focused solutions
GPG Key: http://www.linux-corner.info/mwatts.gpg
Re: [users@httpd] Centralized logging.
Posted by Mike Cardwell <ap...@lists.grepular.com>.
On 09/04/2010 03:38, Sander Temme wrote:
>> The options I see so far are:
>
> Anyone using mod_log_spread and a spread ring to distribute logs?
I used that at my previous job which I left a few years ago. I'm
assuming they still have it in place. IIRC there were eight shared
hosting web servers using it to log to a central location. It just worked.
You can embed perl in the spread log daemon to process/modify log
entries before writing them out. I used this functionality to change the
log format to W3C Extended so the logs were in the same format as those
created by IIS on our Windows based web servers.
--
Mike Cardwell - Perl/Java/Web developer, Linux admin, Email admin
Read my tech Blog - https://secure.grepular.com/
Follow me on Twitter - http://twitter.com/mickeyc
Hire me - http://cardwellit.com/ http://uk.linkedin.com/in/mikecardwell
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Centralized logging.
Posted by Sander Temme <sc...@apache.org>.
On Apr 8, 2010, at 1:52 PM, Ray Van Dolson wrote:
> The options I see so far are:
Anyone using mod_log_spread and a spread ring to distribute logs?
S.
--
Sander Temme
sctemme@apache.org
PGP FP: 51B4 8727 466A 0BC3 69F4 B7B8 B2BE BC40 1529 24AF
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Centralized logging.
Posted by Sandro Tosi <sa...@register.it>.
On 04/08/2010 10:52 PM, Ray Van Dolson wrote:
> So how are people out there in mailing list land handling centralized
> logging?
>
We are considering using Scribe[1] to send logs to a remote log server
and aggregate/elaborate them in real-time.
[1] http://en.wikipedia.org/wiki/Scribe_%28log_server%29
Regards,
Sandro
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Centralized logging.
Posted by Jason Nunnelley <ja...@jasonn.com>.
On 4/10/10 10:04 AM, Ray Van Dolson wrote:
> Why not use splunk.
--
> Way too expensive. I think this space could use some competition. :)
>
> (Splunk is pretty awesome however though it wouldn't help me in this
> particular case really...).
>
I keep hearing about folks and Splunk, and would love to try it out.
But, I'm pulling 5-50G a day in log files.
I'd love some conversation here about better log reading techniques and
reporting packages.
--
Jason A. Nunnelley
+1 2562971652
http://www.google.com/profiles/imjasonn
[Member Tekany, LLC]
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Centralized logging.
Posted by Ray Van Dolson <rv...@esri.com>.
On Fri, Apr 09, 2010 at 10:44:54PM -0700, Brent Clark wrote:
> On 08/04/2010 22:52, Ray Van Dolson wrote:
> > So how are people out there in mailing list land handling centralized
> > logging?
> >
> > I'd like to mirror my web farm's logfiles on a central server, but want
> > to be sure to preserve the W3C-ness so it's easy to run awstats, etc
> > against them from off this central box.
> >
> > The options I see so far are:
> >
> > - Piping to syslog via external process (perl script, etc).
> > * Downside being that we end up with syslog timestamps
> > - Pipe to something like netcat
> > * Custom-ish / hacky
> > - Fire off some sort of synchronization process from central box to
> > pull in logs (rsync) periodically.
> > * Asynchronous -- there's a somewhat large window of time where
> > logs will be missing on the internal server.
> > - Log directly to shared, remote file server (via NFS)
> > * Log server network issues or technical problems could affect
> > Apache servers
> >
> > How do you guys typically deal with this?
> >
> >
>
> Hiya
>
> Why not use splunk.
>
> HTH
>
> Brent Clark
Way too expensive. I think this space could use some competition. :)
(Splunk is pretty awesome however though it wouldn't help me in this
particular case really...).
Ray
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Centralized logging.
Posted by Brent Clark <br...@gmail.com>.
On 08/04/2010 22:52, Ray Van Dolson wrote:
> So how are people out there in mailing list land handling centralized
> logging?
>
> I'd like to mirror my web farm's logfiles on a central server, but want
> to be sure to preserve the W3C-ness so it's easy to run awstats, etc
> against them from off this central box.
>
> The options I see so far are:
>
> - Piping to syslog via external process (perl script, etc).
> * Downside being that we end up with syslog timestamps
> - Pipe to something like netcat
> * Custom-ish / hacky
> - Fire off some sort of synchronization process from central box to
> pull in logs (rsync) periodically.
> * Asynchronous -- there's a somewhat large window of time where
> logs will be missing on the internal server.
> - Log directly to shared, remote file server (via NFS)
> * Log server network issues or technical problems could affect
> Apache servers
>
> How do you guys typically deal with this?
>
>
Hiya
Why not use splunk.
HTH
Brent Clark
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org