You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by YorHel <yo...@gmail.com> on 2010/11/05 14:36:36 UTC
[users@httpd] Buffering I/O for (Fast)CGI
Hello,
I am considering migrating my sites from Lighttpd to Apache. And while
migrating the configuration went fine on my test server (using
mod_fcgid), there's one problem I have been unable to solve.
Lighttpd buffers the entire HTTP request in memory before passing it
to the content generator (which runs as plain CGI or FastCGI), and
buffers the entire HTTP response in case the client connection is not
fast enough. My FastCGI applications are written to process requests
as fast as possible (usually within 100ms, pages that take longer than
500ms are very rare) and with my lighttpd setup one or two processes
are enough to handle all requests while keeping the site responsive.
As each process takes almost 20MB of RAM, you can imagine that I'd
like to keep the number of processes that run simultaneously to a
minimum. Buffering I/O within the httpd server really helps me with
that.
I have not been able to find a similar feature built-in into Apache,
and while looking for solutions I came across mod_buffer in Apache
2.3. Which, if I understood its working right, does not buffer input
*before* the request is passed to the handler, but while the handler
is already "handling" the request. Running an alpha release on a
production server does not seem like a good idea, either.
I also found mod_security, which is a sophisticated-looking external
module that seems to be able to buffer request data, but doesn't seem
to buffer output.
This makes me wonder: is my situation that rare, or am I simply
looking in the wrong direction? What is the usual Apache way to handle
situations like this?
Thanks,
Yoran.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Buffering I/O for (Fast)CGI
Posted by YorHel <yo...@gmail.com>.
A short follow-up on my problem:
On 5 November 2010 15:38, YorHel <yo...@gmail.com> wrote:
> On 5 November 2010 15:18, Jeff Trawick <tr...@gmail.com> wrote:
>> For input, mod_fcgid already does what you want (in fact it doesn't
>> implement any alternative).
>
> Ah, I wasn't aware of that. I did test mod_cgi and found out that the
> httpd wasn't buffering the input in that case. I had assumed mod_fcgid
> to work similar in that regard, but I guess I was wrong. I'll do a
> similar test with mod_fcgid tomorrow.
Yup, I can confirm that mod_fcgid does the trick fine.
>> For output, have you looked at the FcgidOutputBufferSize directive?
>
> I'm surprised haven't noticed that option before. Looks like I was too
> focused on plain CGI in my research. (for which it's not even all that
> crucial to buffer I/O *sigh*) The documentation suggests it will do
> what I want. I'll play around with that as well tomorrow and report my
> results.
Tested that option and confirmed that it indeed does exactly what I want.
I was expecting a more generalized solution to this problem within
Apache, one that would also work with other content generators that
could benefit from it (e.g. mod_cgi and mod_proxy). But this solution
works fine as well.
Thanks again!
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Buffering I/O for (Fast)CGI
Posted by YorHel <yo...@gmail.com>.
Thanks for the quick reply!
On 5 November 2010 15:18, Jeff Trawick <tr...@gmail.com> wrote:
> On Fri, Nov 5, 2010 at 9:36 AM, YorHel <yo...@gmail.com> wrote:
>> Hello,
>> ..
>>
>> I have not been able to find a similar feature built-in into Apache,
>> and while looking for solutions I came across mod_buffer in Apache
>> 2.3. Which, if I understood its working right, does not buffer input
>> *before* the request is passed to the handler, but while the handler
>> is already "handling" the request. Running an alpha release on a
>> production server does not seem like a good idea, either.
>
> For input, mod_fcgid already does what you want (in fact it doesn't
> implement any alternative).
Ah, I wasn't aware of that. I did test mod_cgi and found out that the
httpd wasn't buffering the input in that case. I had assumed mod_fcgid
to work similar in that regard, but I guess I was wrong. I'll do a
similar test with mod_fcgid tomorrow.
> For output, have you looked at the FcgidOutputBufferSize directive?
I'm surprised haven't noticed that option before. Looks like I was too
focused on plain CGI in my research. (for which it's not even all that
crucial to buffer I/O *sigh*) The documentation suggests it will do
what I want. I'll play around with that as well tomorrow and report my
results.
> I looked at mod_buffer and similar logic in mod_fcgid some time ago
> with the thought that mod_buffer could be used, but IIRC fcgid has
> flushing unexpectedly integrated with buffering. It would be good for
> someone to separate that into separately configurable processing and
> hopefully allow mod_buffer to be used to the extent possible.
>
>> I also found mod_security, which is a sophisticated-looking external
>> module that seems to be able to buffer request data, but doesn't seem
>> to buffer output.
>>
>> This makes me wonder: is my situation that rare, or am I simply
>> looking in the wrong direction? What is the usual Apache way to handle
>> situations like this?
>
> not rare; somebody sitting next to me at the moment complains of some
> humongous Drupal (PHP) processes, 8 of which will take over a 48GB
> system
Ouch. I will consider myself lucky, then.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Buffering I/O for (Fast)CGI
Posted by Jeff Trawick <tr...@gmail.com>.
On Fri, Nov 5, 2010 at 9:36 AM, YorHel <yo...@gmail.com> wrote:
> Hello,
>
> I am considering migrating my sites from Lighttpd to Apache. And while
> migrating the configuration went fine on my test server (using
> mod_fcgid), there's one problem I have been unable to solve.
>
> Lighttpd buffers the entire HTTP request in memory before passing it
> to the content generator (which runs as plain CGI or FastCGI), and
> buffers the entire HTTP response in case the client connection is not
> fast enough. My FastCGI applications are written to process requests
> as fast as possible (usually within 100ms, pages that take longer than
> 500ms are very rare) and with my lighttpd setup one or two processes
> are enough to handle all requests while keeping the site responsive.
> As each process takes almost 20MB of RAM, you can imagine that I'd
> like to keep the number of processes that run simultaneously to a
> minimum. Buffering I/O within the httpd server really helps me with
> that.
>
> I have not been able to find a similar feature built-in into Apache,
> and while looking for solutions I came across mod_buffer in Apache
> 2.3. Which, if I understood its working right, does not buffer input
> *before* the request is passed to the handler, but while the handler
> is already "handling" the request. Running an alpha release on a
> production server does not seem like a good idea, either.
For input, mod_fcgid already does what you want (in fact it doesn't
implement any alternative).
For output, have you looked at the FcgidOutputBufferSize directive?
I looked at mod_buffer and similar logic in mod_fcgid some time ago
with the thought that mod_buffer could be used, but IIRC fcgid has
flushing unexpectedly integrated with buffering. It would be good for
someone to separate that into separately configurable processing and
hopefully allow mod_buffer to be used to the extent possible.
> I also found mod_security, which is a sophisticated-looking external
> module that seems to be able to buffer request data, but doesn't seem
> to buffer output.
>
> This makes me wonder: is my situation that rare, or am I simply
> looking in the wrong direction? What is the usual Apache way to handle
> situations like this?
not rare; somebody sitting next to me at the moment complains of some
humongous Drupal (PHP) processes, 8 of which will take over a 48GB
system
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org