You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Bill Moran <wm...@potentialtech.com> on 2004/09/11 20:34:55 UTC
[users@httpd] Logging many, many virtualhosts to a centralized logging server
I tried some searches on this topic, and simply wasn't able to get any
relevent hits. So suggestions on searching are welcome as well as
direct suggestions.
We have 4 web servers, each of which has 20-40 virtual hosts on it, and
we want to log them all to a central location so a dedicated server can
generate stats (plus, the log files are taking up quite a bit of space).
In order to get stats, we need each virtualhost to log to its own
location. Thus, syslog isn't handling things too well, as there are
too many different destinations to easily handle. The fellow who
managed this before me designed a custom logging solution, but it's
been giving us trouble and I'm looking to see if I can come up with
something better.
I can't believe I'm the only one with this scenerio. What are other
people doing to make it happen?
--
Bill Moran
Potential Technologies
http://www.potentialtech.com
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Logging many, many virtualhosts to a centralized
logging server
Posted by Bill Moran <wm...@potentialtech.com>.
Ryan Thompson <ht...@sasknow.com> wrote:
> Bill Moran wrote to users@httpd.apache.org:
>
> Hi Bill,
>
> > I tried some searches on this topic, and simply wasn't able to get any
> > relevent hits. So suggestions on searching are welcome as well as
> > direct suggestions.
> >
> > We have 4 web servers, each of which has 20-40 virtual hosts on it, and
> > we want to log them all to a central location so a dedicated server can
> > generate stats (plus, the log files are taking up quite a bit of space).
> >
> > In order to get stats, we need each virtualhost to log to its own
> > location.
>
> I would recommend against that. Just log to one file (per server), and
> use CustomLog and LogFormat to include the %v (virtual host) for every
> request, and then /periodically/ run a cron job to split the output into
> virtual hosts for webalizer, grouping on that %v directive.
>
> Each server logs to its own local access.log, which you can, at regular
> intervals (minutes to hours), split and copy or SCP to your master
> server with webalizer. That way, apache is always logging locally, and
> if your link to the master log server goes down, apache is still happy,
> and the only thing that happens is the updates time out and don't get
> published until the next interval.
>
> And, to save space, you can of course rotate the access.log once the
> information has been delivered to the master.
>
> You get reliability, scalability, and efficiency, and it's pretty darned
> trivial to set up.
Thanks for the input, Ryan. I'm seriously considering this approach.
--
Bill Moran
Potential Technologies
http://www.potentialtech.com
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Logging many, many virtualhosts to a centralized
logging server
Posted by Ryan Thompson <ht...@sasknow.com>.
Bill Moran wrote to users@httpd.apache.org:
Hi Bill,
> I tried some searches on this topic, and simply wasn't able to get any
> relevent hits. So suggestions on searching are welcome as well as
> direct suggestions.
>
> We have 4 web servers, each of which has 20-40 virtual hosts on it, and
> we want to log them all to a central location so a dedicated server can
> generate stats (plus, the log files are taking up quite a bit of space).
>
> In order to get stats, we need each virtualhost to log to its own
> location.
I would recommend against that. Just log to one file (per server), and
use CustomLog and LogFormat to include the %v (virtual host) for every
request, and then /periodically/ run a cron job to split the output into
virtual hosts for webalizer, grouping on that %v directive.
Each server logs to its own local access.log, which you can, at regular
intervals (minutes to hours), split and copy or SCP to your master
server with webalizer. That way, apache is always logging locally, and
if your link to the master log server goes down, apache is still happy,
and the only thing that happens is the updates time out and don't get
published until the next interval.
And, to save space, you can of course rotate the access.log once the
information has been delivered to the master.
You get reliability, scalability, and efficiency, and it's pretty darned
trivial to set up.
- Ryan
> Thus, syslog isn't handling things too well, as there are
> too many different destinations to easily handle. The fellow who
> managed this before me designed a custom logging solution, but it's
> been giving us trouble and I'm looking to see if I can come up with
> something better.
>
> I can't believe I'm the only one with this scenerio. What are other
> people doing to make it happen?
>
>
--
Ryan Thompson <ry...@sasknow.com>
SaskNow Technologies - http://www.sasknow.com
901-1st Avenue North - Saskatoon, SK - S7K 1Y4
Tel: 306-664-3600 Fax: 306-244-7037 Saskatoon
Toll-Free: 877-727-5669 (877-SASKNOW) North America
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Logging many, many virtualhosts to a centralized
logging server
Posted by Nick Kew <ni...@webthing.com>.
On Sat, 11 Sep 2004, Bill Moran wrote:
> We have 4 web servers, each of which has 20-40 virtual hosts on it, and
> we want to log them all to a central location so a dedicated server can
> generate stats (plus, the log files are taking up quite a bit of space).
>
> In order to get stats, we need each virtualhost to log to its own
> location.
Would an SQL-based logging module meet your needs? Just use a
frontend that queries on virtualhost.
Your exact options may depend on what platform and apache version/MPM
you're using.
--
Nick Kew
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Logging many, many virtualhosts to a centralized logging server
Posted by Hongwei Zhang <ra...@gmail.com>.
Hi Bill,
In my experience this problem happen only when the web contents are
reside on NFS share. I can start Apache and delete the log files, and
Apache is still running fine.
On Sat, 11 Sep 2004 16:20:02 -0400, Bill Moran <wm...@potentialtech.com> wrote:
> Hongwei Zhang <ra...@gmail.com> wrote:
> > I have all my logs write to an NFS share and run webalizer on that
> > server. Hope this will help.
>
> The biggest problem with this is what happens if the NFS server is
> unavailable for a period of time. In our experience, Apache freezes up
> until the NFS server comes back. The current system (which uses syslog)
> has this advantage ... even though log entries are lost, Apache keeps
> serving. We could workaround this (somewhat) by using soft, interruptable
> mounts, I suppose.
>
>
>
> > On Sat, 11 Sep 2004 14:34:55 -0400, Bill Moran <wm...@potentialtech.com> wrote:
> > >
> > > I tried some searches on this topic, and simply wasn't able to get any
> > > relevent hits. So suggestions on searching are welcome as well as
> > > direct suggestions.
> > >
> > > We have 4 web servers, each of which has 20-40 virtual hosts on it, and
> > > we want to log them all to a central location so a dedicated server can
> > > generate stats (plus, the log files are taking up quite a bit of space).
> > >
> > > In order to get stats, we need each virtualhost to log to its own
> > > location. Thus, syslog isn't handling things too well, as there are
> > > too many different destinations to easily handle. The fellow who
> > > managed this before me designed a custom logging solution, but it's
> > > been giving us trouble and I'm looking to see if I can come up with
> > > something better.
> > >
> > > I can't believe I'm the only one with this scenerio. What are other
> > > people doing to make it happen?
>
> --
> Bill Moran
> Potential Technologies
> http://www.potentialtech.com
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Re: Logging many, many virtualhosts to a
centralized logging server
Posted by Bill Moran <wm...@potentialtech.com>.
Michelle Konzack <li...@freenet.de> wrote:
> Am 2004-09-11 16:20:02, schrieb Bill Moran:
>
> > The biggest problem with this is what happens if the NFS server is
> > unavailable for a period of time. In our experience, Apache freezes up
> > until the NFS server comes back. The current system (which uses syslog)
>
> Why not using "syslog-ng" ?
> It can operate via Network and is fully configurable...
Unless I'm missing something ...
syslong-ng still only supports so many facilities ... thus it's difficult
to direct 50+ different sites to 50+ different destination files.
--
Bill Moran
Potential Technologies
http://www.potentialtech.com
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
[users@httpd] Re: Logging many, many virtualhosts to a centralized logging server
Posted by Michelle Konzack <li...@freenet.de>.
Am 2004-09-11 16:20:02, schrieb Bill Moran:
> The biggest problem with this is what happens if the NFS server is
> unavailable for a period of time. In our experience, Apache freezes up
> until the NFS server comes back. The current system (which uses syslog)
Why not using "syslog-ng" ?
It can operate via Network and is fully configurable...
> has this advantage ... even though log entries are lost, Apache keeps
> serving. We could workaround this (somewhat) by using soft, interruptable
> mounts, I suppose.
Greetings
Michelle
--
Linux-User #280138 with the Linux Counter, http://counter.li.org/
Michelle Konzack Apt. 917 ICQ #328449886
50, rue de Soultz MSM LinuxMichi
0033/3/88452356 67100 Strasbourg/France IRC #Debian (irc.icq.com)
Re: [users@httpd] Logging many, many virtualhosts to a centralized
logging server
Posted by Bill Moran <wm...@potentialtech.com>.
Hongwei Zhang <ra...@gmail.com> wrote:
> I have all my logs write to an NFS share and run webalizer on that
> server. Hope this will help.
The biggest problem with this is what happens if the NFS server is
unavailable for a period of time. In our experience, Apache freezes up
until the NFS server comes back. The current system (which uses syslog)
has this advantage ... even though log entries are lost, Apache keeps
serving. We could workaround this (somewhat) by using soft, interruptable
mounts, I suppose.
> On Sat, 11 Sep 2004 14:34:55 -0400, Bill Moran <wm...@potentialtech.com> wrote:
> >
> > I tried some searches on this topic, and simply wasn't able to get any
> > relevent hits. So suggestions on searching are welcome as well as
> > direct suggestions.
> >
> > We have 4 web servers, each of which has 20-40 virtual hosts on it, and
> > we want to log them all to a central location so a dedicated server can
> > generate stats (plus, the log files are taking up quite a bit of space).
> >
> > In order to get stats, we need each virtualhost to log to its own
> > location. Thus, syslog isn't handling things too well, as there are
> > too many different destinations to easily handle. The fellow who
> > managed this before me designed a custom logging solution, but it's
> > been giving us trouble and I'm looking to see if I can come up with
> > something better.
> >
> > I can't believe I'm the only one with this scenerio. What are other
> > people doing to make it happen?
--
Bill Moran
Potential Technologies
http://www.potentialtech.com
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Logging many, many virtualhosts to a centralized logging server
Posted by Hongwei Zhang <ra...@gmail.com>.
I have all my logs write to an NFS share and run webalizer on that
server. Hope this will help.
On Sat, 11 Sep 2004 14:34:55 -0400, Bill Moran <wm...@potentialtech.com> wrote:
>
> I tried some searches on this topic, and simply wasn't able to get any
> relevent hits. So suggestions on searching are welcome as well as
> direct suggestions.
>
> We have 4 web servers, each of which has 20-40 virtual hosts on it, and
> we want to log them all to a central location so a dedicated server can
> generate stats (plus, the log files are taking up quite a bit of space).
>
> In order to get stats, we need each virtualhost to log to its own
> location. Thus, syslog isn't handling things too well, as there are
> too many different destinations to easily handle. The fellow who
> managed this before me designed a custom logging solution, but it's
> been giving us trouble and I'm looking to see if I can come up with
> something better.
>
> I can't believe I'm the only one with this scenerio. What are other
> people doing to make it happen?
>
> --
> Bill Moran
> Potential Technologies
> http://www.potentialtech.com
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Logging many, many virtualhosts to a centralized
logging server
Posted by Bill Moran <wm...@potentialtech.com>.
Paul Querna <ch...@force-elite.com> wrote:
> On Sat, 2004-09-11 at 14:34 -0400, Bill Moran wrote:
> > I can't believe I'm the only one with this scenerio. What are other
> > people doing to make it happen?
> >
>
> mod_log_spread -> http://www.backhand.org/mod_log_spread/
>
> It uses multi-cast to send the log info out. So you can have multiple
> machines listening to write the logs to disk...
Thanks for the input. This is definatly an interesting option!
--
Bill Moran
Potential Technologies
http://www.potentialtech.com
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Logging many, many virtualhosts to a centralized
logging server
Posted by Paul Querna <ch...@force-elite.com>.
On Sat, 2004-09-11 at 14:34 -0400, Bill Moran wrote:
> I can't believe I'm the only one with this scenerio. What are other
> people doing to make it happen?
>
mod_log_spread -> http://www.backhand.org/mod_log_spread/
It uses multi-cast to send the log info out. So you can have multiple
machines listening to write the logs to disk...
-Paul Querna
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Logging many, many virtualhosts to a centralized
logging server
Posted by Carl Brewer <ca...@vivitec.com.au>.
Bill Moran wrote:
> We have 4 web servers, each of which has 20-40 virtual hosts on it, and
> we want to log them all to a central location so a dedicated server can
> generate stats (plus, the log files are taking up quite a bit of space).
>
> In order to get stats, we need each virtualhost to log to its own
> location. Thus, syslog isn't handling things too well, as there are
> too many different destinations to easily handle. The fellow who
> managed this before me designed a custom logging solution, but it's
> been giving us trouble and I'm looking to see if I can come up with
> something better.
We do this as a batch process, using rsync to copy log files
to our log processor, and then deleting the logs on the
webservers themselves once we're sure they're onto the
log server. Since web stats processing is a batch job,
we don't see the batchy'ness of this as an issue.
We logroll, rsync, delete every few hours on
the webservers, and only delete when we're
sure the rsync's completed successfully.
It's been working very well for us for years.
HTH
Carl
--
=======================
Vivitec Pty. Ltd.
Suite 6, 51-55 City Rd.
Southbank, 3006.
Ph. +61 3 8626 5626
Fax +61 3 9682 1000
=======================
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org