You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Grant Peel <gp...@thenetnow.com> on 2008/09/11 01:53:50 UTC
[users@httpd] Logs
Hi all,
I am investigating useing apache rotatelogs pipe. My servers have about 250
virtual domains each on them, so I am curious about a couple of things:
How are people in a similar setup handling remove logs (so they dont build
up forever), say after 2 months?
Does piping the data through the rotatelogs util slow down the server much?
-Grant
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Logs
Posted by Justin Pasher <ju...@newmediagateway.com>.
Grant Peel wrote:
> Justin,
>
> Kewl!
>
> There are a few users in the home directory as well, and those users
> do not have a logs directory. How will logrotate handle that? (I am
> hoping you will say it just ignores a non existent path/file).
>
> -Grant
>
You will be safe. The wildcards simply use the shell's globbing
functionality, which means that it will only return files that exist.
Justin Pasher
> ----- Original Message ----- From: "Justin Pasher"
> <ju...@newmediagateway.com>
> To: <us...@httpd.apache.org>
> Sent: Thursday, September 11, 2008 6:10 PM
> Subject: Re: [users@httpd] Logs
>
>
>> Grant Peel wrote:
>>> Hi Justin,
>>>
>>> Thanks for the reply. FYI I am using UNIX (freebsd).
>>>
>>> Up tp this point, I have been using an sh script to rotate logs.
>>>
>>> The logs in question are the access_log and error_log in each one of
>>> my (Apache) virtual hosts.
>>>
>>> logrotate looks like the cats meow!
>>>
>>> I have read the man page and it states to use wildcards with caution
>>> (as always). So I have one question:
>>>
>>> Can I use a wildcard as such,
>>>
>>> /home/*/logs/access_log
>>>
>>> /home/*/logs/error_log
>>>
>>> The '*' being the wildcard to denote the home dir for
>>>
>>> virt_domain1.com
>>> virt_domain2.ca
>>> virt_domain3.net
>>> ...
>>>
>>> -Grant
>>
>> Wildcards are fine. The man pages problem say to use them with
>> caution to avoid matching more files than you really intend. The
>> easiest way to be sure is just test the results of the wildcard with ls.
>>
>> ls /home/*/logs/access_log
>>
>> If the results are what you expect, then you are good to go. You can
>> also combine the two log filenames in the same logrotate rule to
>> avoid duplicating your settings.
>>
>> Just keep in mind that with this setup, if a user has a file such as
>> /home/username/logs/access_log, then it will be rotated too. However,
>> if your setup doesn't allow users to create arbitrary
>> directories/files like that, then there is no need to worry.
>>
>> --
>> Justin Pasher
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Logs
Posted by Grant Peel <gp...@thenetnow.com>.
Justin,
Kewl!
There are a few users in the home directory as well, and those users do not
have a logs directory. How will logrotate handle that? (I am hoping you will
say it just ignores a non existent path/file).
-Grant
----- Original Message -----
From: "Justin Pasher" <ju...@newmediagateway.com>
To: <us...@httpd.apache.org>
Sent: Thursday, September 11, 2008 6:10 PM
Subject: Re: [users@httpd] Logs
> Grant Peel wrote:
>> Hi Justin,
>>
>> Thanks for the reply. FYI I am using UNIX (freebsd).
>>
>> Up tp this point, I have been using an sh script to rotate logs.
>>
>> The logs in question are the access_log and error_log in each one of my
>> (Apache) virtual hosts.
>>
>> logrotate looks like the cats meow!
>>
>> I have read the man page and it states to use wildcards with caution (as
>> always). So I have one question:
>>
>> Can I use a wildcard as such,
>>
>> /home/*/logs/access_log
>>
>> /home/*/logs/error_log
>>
>> The '*' being the wildcard to denote the home dir for
>>
>> virt_domain1.com
>> virt_domain2.ca
>> virt_domain3.net
>> ...
>>
>> -Grant
>
> Wildcards are fine. The man pages problem say to use them with caution to
> avoid matching more files than you really intend. The easiest way to be
> sure is just test the results of the wildcard with ls.
>
> ls /home/*/logs/access_log
>
> If the results are what you expect, then you are good to go. You can also
> combine the two log filenames in the same logrotate rule to avoid
> duplicating your settings.
>
> Just keep in mind that with this setup, if a user has a file such as
> /home/username/logs/access_log, then it will be rotated too. However, if
> your setup doesn't allow users to create arbitrary directories/files like
> that, then there is no need to worry.
>
> --
> Justin Pasher
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Logs
Posted by Justin Pasher <ju...@newmediagateway.com>.
Grant Peel wrote:
> Hi Justin,
>
> Thanks for the reply. FYI I am using UNIX (freebsd).
>
> Up tp this point, I have been using an sh script to rotate logs.
>
> The logs in question are the access_log and error_log in each one of
> my (Apache) virtual hosts.
>
> logrotate looks like the cats meow!
>
> I have read the man page and it states to use wildcards with caution
> (as always). So I have one question:
>
> Can I use a wildcard as such,
>
> /home/*/logs/access_log
>
> /home/*/logs/error_log
>
> The '*' being the wildcard to denote the home dir for
>
> virt_domain1.com
> virt_domain2.ca
> virt_domain3.net
> ...
>
> -Grant
Wildcards are fine. The man pages problem say to use them with caution
to avoid matching more files than you really intend. The easiest way to
be sure is just test the results of the wildcard with ls.
ls /home/*/logs/access_log
If the results are what you expect, then you are good to go. You can
also combine the two log filenames in the same logrotate rule to avoid
duplicating your settings.
Just keep in mind that with this setup, if a user has a file such as
/home/username/logs/access_log, then it will be rotated too. However, if
your setup doesn't allow users to create arbitrary directories/files
like that, then there is no need to worry.
--
Justin Pasher
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Logs
Posted by Alexandru David Constantinescu <al...@gmail.com>.
Grant Peel wrote:
> Hi Justin,
>
> Thanks for the reply. FYI I am using UNIX (freebsd).
>
> Up tp this point, I have been using an sh script to rotate logs.
>
> The logs in question are the access_log and error_log in each one of my
> (Apache) virtual hosts.
>
> logrotate looks like the cats meow!
>
> I have read the man page and it states to use wildcards with caution (as
> always). So I have one question:
>
> Can I use a wildcard as such,
>
> /home/*/logs/access_log
>
> /home/*/logs/error_log
>
> The '*' being the wildcard to denote the home dir for
>
> virt_domain1.com
> virt_domain2.ca
> virt_domain3.net
> ...
>
> -Grant
>
> ----- Original Message ----- From: "Justin Pasher"
> <ju...@newmediagateway.com>
> To: <us...@httpd.apache.org>
> Sent: Wednesday, September 10, 2008 10:18 PM
> Subject: RE: [users@httpd] Logs
>
>
>>> -----Original Message-----
>>> From: Grant Peel [mailto:gpeel@thenetnow.com]
>>> Sent: Wednesday, September 10, 2008 6:54 PM
>>> To: users@httpd.apache.org
>>> Subject: [users@httpd] Logs
>>>
>>> Hi all,
>>>
>>> I am investigating useing apache rotatelogs pipe. My servers have about
>>> 250
>>> virtual domains each on them, so I am curious about a couple of things:
>>>
>>> How are people in a similar setup handling remove logs (so they dont
>>> build
>>> up forever), say after 2 months?
>>>
>>> Does piping the data through the rotatelogs util slow down the server
>>> much?
>>
>> Have you considered using logrotate? I had never actually heard of
>> rotatelogs until now (my apache experience is primarily with Apache 1). I
>> would imagine that it would have a little bit of overhead (albeit a
>> relatively small amount, I hope). logrotate has a lot more options
>> available, and there's also a chance that it's already in use on your
>> system
>> to rotate system logs in /var/log. Since it's run as a cron job, you only
>> experience the overheard (VERY small) when the script run each night
>>
>> Of course, all of this assumes you are running in a *nix environment as
>> opposed to Windows. I'm not sure about the availability on Windows.
>>
>> --
>> Justin Pasher
>>
>>
>> ---------------------------------------------------------------------
>> The official User-To-User support forum of the Apache HTTP Server
>> Project.
>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
I don't think that the idea to keep separate logs (real time) for each
virtualhost it's a good idea. Think to the IO involve in this
operations. Let's say you have 300 vhost, your system must write in 600
separate files.
Once I have the same problem. The solution was :
1 - to keep just two file access log and error log;
2 - logrotate rotate this files every 1 hour (on 00 I have 48 gzip files);
3 - I use split-logfile (a perl script from apache I think) to split
every log for each vhost and put the chunk in homedir of each vhost
4 - move (copy) that 48 gzip log files on separate server for archiving.
Alex
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Logs
Posted by Grant Peel <gp...@thenetnow.com>.
Hi Justin,
Thanks for the reply. FYI I am using UNIX (freebsd).
Up tp this point, I have been using an sh script to rotate logs.
The logs in question are the access_log and error_log in each one of my
(Apache) virtual hosts.
logrotate looks like the cats meow!
I have read the man page and it states to use wildcards with caution (as
always). So I have one question:
Can I use a wildcard as such,
/home/*/logs/access_log
/home/*/logs/error_log
The '*' being the wildcard to denote the home dir for
virt_domain1.com
virt_domain2.ca
virt_domain3.net
...
-Grant
----- Original Message -----
From: "Justin Pasher" <ju...@newmediagateway.com>
To: <us...@httpd.apache.org>
Sent: Wednesday, September 10, 2008 10:18 PM
Subject: RE: [users@httpd] Logs
>> -----Original Message-----
>> From: Grant Peel [mailto:gpeel@thenetnow.com]
>> Sent: Wednesday, September 10, 2008 6:54 PM
>> To: users@httpd.apache.org
>> Subject: [users@httpd] Logs
>>
>> Hi all,
>>
>> I am investigating useing apache rotatelogs pipe. My servers have about
>> 250
>> virtual domains each on them, so I am curious about a couple of things:
>>
>> How are people in a similar setup handling remove logs (so they dont
>> build
>> up forever), say after 2 months?
>>
>> Does piping the data through the rotatelogs util slow down the server
>> much?
>
> Have you considered using logrotate? I had never actually heard of
> rotatelogs until now (my apache experience is primarily with Apache 1). I
> would imagine that it would have a little bit of overhead (albeit a
> relatively small amount, I hope). logrotate has a lot more options
> available, and there's also a chance that it's already in use on your
> system
> to rotate system logs in /var/log. Since it's run as a cron job, you only
> experience the overheard (VERY small) when the script run each night
>
> Of course, all of this assumes you are running in a *nix environment as
> opposed to Windows. I'm not sure about the availability on Windows.
>
> --
> Justin Pasher
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Logs
Posted by Justin Pasher <ju...@newmediagateway.com>.
> -----Original Message-----
> From: Grant Peel [mailto:gpeel@thenetnow.com]
> Sent: Wednesday, September 10, 2008 6:54 PM
> To: users@httpd.apache.org
> Subject: [users@httpd] Logs
>
> Hi all,
>
> I am investigating useing apache rotatelogs pipe. My servers have about
> 250
> virtual domains each on them, so I am curious about a couple of things:
>
> How are people in a similar setup handling remove logs (so they dont build
> up forever), say after 2 months?
>
> Does piping the data through the rotatelogs util slow down the server
> much?
Have you considered using logrotate? I had never actually heard of
rotatelogs until now (my apache experience is primarily with Apache 1). I
would imagine that it would have a little bit of overhead (albeit a
relatively small amount, I hope). logrotate has a lot more options
available, and there's also a chance that it's already in use on your system
to rotate system logs in /var/log. Since it's run as a cron job, you only
experience the overheard (VERY small) when the script run each night
Of course, all of this assumes you are running in a *nix environment as
opposed to Windows. I'm not sure about the availability on Windows.
--
Justin Pasher
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org