You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Jaspreet Singh <js...@ensim.com> on 2005/05/13 13:36:30 UTC
[users@httpd] Filtering/Interception Error log strings
hi,
I want to use apache to support 2000+ virt-hosts and log errors
seperately for each virt-host.
But apache opens and maintains fd's for each error_log and access_log.
And hence the select syscall ( which has a fd_set of max 1024 fds)
fails .. and hence apache fails to start up.
Then i tried to pipe access log .. and filter it to seperate files
(depending upon which virthost it belongs to). But error log can't be
filtered because it doen't contain sufficient information.
Now , i want to write a module ( or if possible some other way) so that
i can intercept error log strings and depending upon the
request->server->server_hostname .. filter it to different files.
I read in src/httd-2.0.46/CHANGES +3274
*) Add a new request hook, error_log. This phase allows modules
to act on the error log string _after_ it has been written
to the error log. The goal for this hook is to allow monitoring
modules to send the error string to the monitoring agent.
[Ryan Bloom]
Could anyone tell me nore about this hook ?
Thanx,
Jaspreet
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Filtering/Interception Error log strings
Posted by Joe Orton <jo...@redhat.com>.
On Mon, May 16, 2005 at 12:45:48PM +0530, Jaspreet Singh wrote:
> hey,
>
> On Fri, 2005-05-13 at 16:25 +0100, Joe Orton wrote:
> > On Fri, May 13, 2005 at 05:06:30PM +0530, Jaspreet Singh wrote:
> > > hi,
> > >
> > > I want to use apache to support 2000+ virt-hosts and log errors
> > > seperately for each virt-host.
> > >
> > > But apache opens and maintains fd's for each error_log and access_log.
> > > And hence the select syscall ( which has a fd_set of max 1024 fds)
> > > fails .. and hence apache fails to start up.
> >
> > 2.0 will use poll() in preference to select() on platforms where poll()
> > is available, which fixes this issue. Third-party modules/libraries may
> > still be an issue, of course.
> >
> > joe
>
>
> No joe , you are wrong. read
> /src/httpd-2.0.46/server/mpm/experimental/perchild/perchild.c +563
>
> Which clearly says .. you have to rebuild (default) apache so that it
> supports larger FD_SETSIZE for select.
To be clear: the checks you reference were not necessary and were
removed in more recent 2.0 releases. I've tested use of >1024 log files
with 2.0 and it works fine out of the box (after you increase the
ulimit), so long as you avoid use of certain modules (PHP, anything
using OpenSSL).
joe
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Filtering/Interception Error log strings
Posted by Jaspreet Singh <js...@ensim.com>.
hey,
On Fri, 2005-05-13 at 16:25 +0100, Joe Orton wrote:
> On Fri, May 13, 2005 at 05:06:30PM +0530, Jaspreet Singh wrote:
> > hi,
> >
> > I want to use apache to support 2000+ virt-hosts and log errors
> > seperately for each virt-host.
> >
> > But apache opens and maintains fd's for each error_log and access_log.
> > And hence the select syscall ( which has a fd_set of max 1024 fds)
> > fails .. and hence apache fails to start up.
>
> 2.0 will use poll() in preference to select() on platforms where poll()
> is available, which fixes this issue. Third-party modules/libraries may
> still be an issue, of course.
>
> joe
No joe , you are wrong. read
/src/httpd-2.0.46/server/mpm/experimental/perchild/perchild.c +563
Which clearly says .. you have to rebuild (default) apache so that it
supports larger FD_SETSIZE for select.
Anyways, i have formulated a workarround the error_log problem (Thanx to
Ryan Bloom ). I use error_log hook defined in httpd_log.h and intercept
error_log strings modify them and re-report them.
Thanx,
jaspreet
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Filtering/Interception Error log strings
Posted by Joe Orton <jo...@redhat.com>.
On Fri, May 13, 2005 at 05:06:30PM +0530, Jaspreet Singh wrote:
> hi,
>
> I want to use apache to support 2000+ virt-hosts and log errors
> seperately for each virt-host.
>
> But apache opens and maintains fd's for each error_log and access_log.
> And hence the select syscall ( which has a fd_set of max 1024 fds)
> fails .. and hence apache fails to start up.
2.0 will use poll() in preference to select() on platforms where poll()
is available, which fixes this issue. Third-party modules/libraries may
still be an issue, of course.
joe
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Filtering/Interception Error log strings
Posted by Jaspreet Singh <js...@ensim.com>.
Hi,
thanx for the reply.
On Fri, 2005-05-13 at 14:19 +0100, Nick Kew wrote:
> Jaspreet Singh wrote:
>
> Not really - a google search would tell you more. There are modules
> written for the purpose (mod_vhost_alias or third-party alternatives),
> or there's a solution described on the mod_rewrite pages.
>
can't use them as i have custom configuration for each virthost
container.
> Good point. Adding the host to normal error logging would seem to be
> the simple solution. I'm not sure if there's a simpler solution.
>
But, How do i append virt_host/server name ????
> >
> >
> > If, i am not mistaken .. "__after__" means that the msg is already
> > logged to error_log and them passed to hook. Thats ok with me, as i can
> > get the server name from "request" structure and log the same msg again
> > to syslog appending server name.
>
> Yes, probably. But logging twice doesn't seem very efficient to me.
>
If somehow i can intercept the error logs .. only then can i log them
(again) with virt_host/server name appended. I couldn't find this hook
in apache headers/code.
any clues ???
thanx,
jaspreet singh
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Filtering/Interception Error log strings
Posted by Nick Kew <ni...@webthing.com>.
Jaspreet Singh wrote:
>>If you use conventional vhosts, yes. That's one reason there are
>>*different* options for mass virtual hosting. You should use one of
>>them for this and other reasons.
>>
>
> could, you please elaborate on different options.
Not really - a google search would tell you more. There are modules
written for the purpose (mod_vhost_alias or third-party alternatives),
or there's a solution described on the mod_rewrite pages.
>>To get separate logs, either postprocess a common log file, or use some
>>alternative such as logging to an SQL database.
>>
>
> Error log doesn't contain virt_host/server name so If i log errors to a
> common file/sql i can't split them later based on virt_hosts which
> caused them.
Good point. Adding the host to normal error logging would seem to be
the simple solution. I'm not sure if there's a simpler solution.
>>>I read in src/httd-2.0.46/CHANGES +3274
>>> *) Add a new request hook, error_log. This phase allows modules
>>> to act on the error log string _after_ it has been written
>>> to the error log.
>>
>>Note the word _after_. It's addressing a different requirement.
>
>
> If, i am not mistaken .. "__after__" means that the msg is already
> logged to error_log and them passed to hook. Thats ok with me, as i can
> get the server name from "request" structure and log the same msg again
> to syslog appending server name.
Yes, probably. But logging twice doesn't seem very efficient to me.
--
Nick Kew
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Filtering/Interception Error log strings
Posted by Jaspreet Singh <js...@ensim.com>.
Hi,
On Fri, 2005-05-13 at 13:36 +0100, Nick Kew wrote:
> Jaspreet Singh wrote:
>
> [ Please don't scatter this to so many list and individual addresses.
> If I see more than one copy of any followup you post, I will certainly
> not take the time to help you any further ]
>
sorry about that, i wan't sure witch list would be better suited for my
post. I individual addresses beacause i could find there posts regarding
the same. Sorry for the inconvinience.
> If you use conventional vhosts, yes. That's one reason there are
> *different* options for mass virtual hosting. You should use one of
> them for this and other reasons.
>
could, you please elaborate on different options.
> To get separate logs, either postprocess a common log file, or use some
> alternative such as logging to an SQL database.
>
Error log doesn't contain virt_host/server name so If i log errors to a
common file/sql i can't split them later based on virt_hosts which
caused them.
> > I read in src/httd-2.0.46/CHANGES +3274
> > *) Add a new request hook, error_log. This phase allows modules
> > to act on the error log string _after_ it has been written
> > to the error log.
>
> Note the word _after_. It's addressing a different requirement.
If, i am not mistaken .. "__after__" means that the msg is already
logged to error_log and them passed to hook. Thats ok with me, as i can
get the server name from "request" structure and log the same msg again
to syslog appending server name.
thanx for the help,
jaspreet singh.
>
--
Don't Walk as if you own the world,
Walk as if you don't damn who owns it.
Jaspreet Singh
Software Engineer,
Ensim India.
jsingh@ensim.com
+91 9890712226
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Filtering/Interception Error log strings
Posted by Nick Kew <ni...@webthing.com>.
Jaspreet Singh wrote:
[ Please don't scatter this to so many list and individual addresses.
If I see more than one copy of any followup you post, I will certainly
not take the time to help you any further ]
> I want to use apache to support 2000+ virt-hosts and log errors
> seperately for each virt-host.
>
> But apache opens and maintains fd's for each error_log and access_log.
If you use conventional vhosts, yes. That's one reason there are
*different* options for mass virtual hosting. You should use one of
them for this and other reasons.
To get separate logs, either postprocess a common log file, or use some
alternative such as logging to an SQL database.
> I read in src/httd-2.0.46/CHANGES +3274
> *) Add a new request hook, error_log. This phase allows modules
> to act on the error log string _after_ it has been written
> to the error log.
Note the word _after_. It's addressing a different requirement.
--
Nick Kew
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
[users@httpd] Re: [apache-modules] Filtering/Interception Error log strings
Posted by Jaspreet Singh <js...@ensim.com>.
hi,
On Fri, 2005-05-13 at 13:46 +0200, rm@fabula.de wrote:
> > I want to use apache to support 2000+ virt-hosts and log errors
> > seperately for each virt-host.
> >
> > But apache opens and maintains fd's for each error_log and access_log.
> > And hence the select syscall ( which has a fd_set of max 1024 fds)
> > fails .. and hence apache fails to start up.
>
> Are you _shure_ about this diagnosis? Why would Apache need to 'select' on
> the log file descriptors? How did you come to this conclusion?
>
> Ralf Mattes
thanx a lot for the reply.
You getting me wrong ... the number of open fd for the httpd process
becomes 1023+ . So when httpd opens sockets to listen they get fd_num >
1024.
Now when apache selects on these sockets .. its gets an error , because
select syscall's fd_set is defined in select.h as
__fd_mask fds_bits[__FD_SETSIZE / __NFDBITS];
where __FD_SETSIZE = 1024.
apache's recomiple can solve the problem .. but i dont want to do it.
Alternatively i want to itercept error logs (using custom module or
otherwise) and log them myself. Access logs can always be constructed
using request_rec , so no need to intercept them.
Regards,
Jaspreet Singh
--
Don't Walk as if you own the world,
Walk as if you don't damn who owns it.
Jaspreet Singh
Software Engineer,
Ensim India.
jsingh@ensim.com
+91 9890712226
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org