You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Nadav Har'El <ny...@math.technion.ac.il> on 2012/07/16 16:07:23 UTC
[users@httpd] mod-cgi reads entire output into memory...
Hi, It's been 10 years since my last message to this mailing list, and
I'm happy to join it again :-)
I've encountered a surprising phenomenon with Apache's mod-cgi, which
unnecessarily slows it down for huge outputs, and as a "bonus" also
has a bug: taking up huge amounts of memory:
I have a CGI program which very quickly writes 512 MB of output.
When I use it in Apache, Apache itself (NOT the CGI process!) grows
by 512 MB (!). I was really surprised by this, because ideally Apache
should hardly grow at all, as at most (if at all) it should be reading
modest-sized buffers from the CGI script and writing them back to the
socket.
I looked at the httpd code, discovered (if I understand correctly) that
1. As I already guessed, Apache doesn't let the CGI write directly to the
socket, but rather asks it to write to a pipe, which Apache then reads.
2. When Apache reads this data from the pipe, it doesn't write it directly
but rather just adds it to a "bucket brigade" which collects more and
more data.
It appears there is no flow-control in this process: If the CGI outputs
faster than we can send to the network, the bucket brigade becomes
longer and longer, and with 512 MB of output quickly generated, up to
512 MB of buffers are allocated, and only much of it is only proccessed
and freed at the end. The peak memory usage, then, is 512 MB, and this
is also the process's memory usage when everything ends (because Apache
doesn't return this memory to the system).
I confirmed that this is indeed a flow-control problem by changing the
CGI to sleep for 1 second after outputting each 64 MB (i.e., 8 batches
of 64 MB output); Now, the memory usage was around 64 MB, not 512 MB,
because Apache had the time to output each batch and free its memory
before the next batch came.
By the way, the growth of the Apache process by 512 MB is only the start of
the problem, because not only every *process* grows by 512 MB, actually
even in the worker MPM every *thread* grows by 512 MB because apparently (?)
Apache's memory pools are separate for different threads, so the 512 MB
freed by one thread is not reused by a different threads. In my default
setup of 25 threads, all of the machine's memory and swap space was
consumed :(
So now I guess my questions are:
1. Has anyone ever thought of doing a "direct CGI" module, where the CGI
script writes directly to the socket, not to Apache's pipe, forgoing
any copying, buffering or filtering in Apache?
Does something like this already exist? Is "NPH" relevant here?
2. Even if we do want Apache's output filtering capabilities, are there
really no flow control capabilities? Can we tell Apache not to read
more input (i.e., CGI's output) if the bucket brigade is larger than
some predefined size (e.g., 1 MB)?
3. Some of you may think that CGI is antiquated, and I shouldn't be
using it - but I do have good reasons to use it ;-) But I wonder (I
didn't test) - is this problem specific to CGI? What happens when we
serve a huge disk *file*, and we can read it faster than we can send
it - does the bucket brigade also grow indefinitely?
Thanks,
Nadav.
--
Nadav Har'El | Monday, Jul 16 2012, 26 Tammuz 5772
nyh@math.technion.ac.il |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |Unix is user friendly - it's just picky
http://nadav.harel.org.il |about its friends.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod-cgi reads entire output into memory...
Posted by Nick Kew <ni...@webthing.com>.
On Tue, 17 Jul 2012 23:25:09 +0300
Nadav Har'El <ny...@math.technion.ac.il> wrote:
> The resulting code is ugly, hairy and Unix-only, but it works and
> performance is significantly better than normal mod_cgi for my usecase
> of very large program-generated files. For example, a simple CGI
> generating 512 MB of data took 0.9 seconds to fetch locally when using
> mod_cgi, but only 0.4 seconds with my "zero-copy mod_cgi".
>
> Additionally, after this local fetch with my "zero-copy mod_cgi", the
> httpd process remains tiny (2 MB resident), while with the original
> "pass through the do-nothing bucket brigade" mod_cgi, httpd grew to
> 86 MB resident.
Did you test your program with mod_cgid, mod_fcgid, mod_proxy_scgi?
I'm thinking, if they also grow out-of-control with your program
then it would probably be worth fixing.
--
Nick Kew
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod-cgi reads entire output into memory...
Posted by Nadav Har'El <ny...@math.technion.ac.il>.
On Tue, Jul 17, 2012, Nadav Har'El wrote about "Re: [users@httpd] mod-cgi reads entire output into memory...":
> I "almost" have such code, but ran into a mystery - where in the
> request_req can I find the client socket, so I can write to it directly?
> There are so many layers of output filters, APR, etc., that I can't seem
> to find this simple thing...
I'm answering myself ;-)
It turns out that ap_get_conn_socket(r->connection) extracts the APR
socket object from the connection, and then apr_os_sock_get() gets the
file-descriptor out of the this APR object, so I can have the child
process inherit it. One thing that took me a while to notice is that
apache sets this socket to be nonblocking - so I need to make it
blocking again (with fcntl(fd, F_SETFL,0)) to make it usable as the
CGI program's standard output.
The resulting code is ugly, hairy and Unix-only, but it works and
performance is significantly better than normal mod_cgi for my usecase
of very large program-generated files. For example, a simple CGI
generating 512 MB of data took 0.9 seconds to fetch locally when using
mod_cgi, but only 0.4 seconds with my "zero-copy mod_cgi".
Additionally, after this local fetch with my "zero-copy mod_cgi", the
httpd process remains tiny (2 MB resident), while with the original
"pass through the do-nothing bucket brigade" mod_cgi, httpd grew to
86 MB resident.
I'm still wondering what's the justification of mod_cgi always reading
(from a pipe) and writing again (to the socket) the output of the
program, even when there are no output filters and even when we have an
NPH cgi (a CGI which generates in its own headers). Is it just that
nobody ever cared about this usecase (and in general, mod_cgi went out
of fashion), or is there a genuine reason why it's important for Apache
to read the output and write it back to the socket?
Nadav.
--
Nadav Har'El | Tuesday, Jul 17 2012, 28 Tammuz 5772
nyh@math.technion.ac.il |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |Some people bring happiness wherever they
http://nadav.harel.org.il |go, others whenever they go.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod-cgi reads entire output into memory...
Posted by Nadav Har'El <ny...@math.technion.ac.il>.
On Mon, Jul 16, 2012, Nick Kew wrote about "Re: [users@httpd] mod-cgi reads entire output into memory...":
> On Mon, 16 Jul 2012 17:07:23 +0300
> Nadav Har'El <ny...@math.technion.ac.il> wrote:
> > I looked at the httpd code, discovered (if I understand correctly) that
> > 1. As I already guessed, Apache doesn't let the CGI write directly to the
> > socket, but rather asks it to write to a pipe, which Apache then reads.
>
> Yep. That's what CGI is all about.
:-)
I've set out to write a simple mod_cgi replacement which lets the child
process write its output directly to the client socket - it behaves like
NPH (no possibility for Apache to fix the CGI's headers or to filter its
output) but I think it will be useful in a lot of cases (who filters
CGI output anyway?) - and certainly more efficient in mine. In fact
I wonder why it shouldn't always work like that with NPH.
I "almost" have such code, but ran into a mystery - where in the
request_req can I find the client socket, so I can write to it directly?
There are so many layers of output filters, APR, etc., that I can't seem
to find this simple thing...
> > 2. When Apache reads this data from the pipe, it doesn't write it directly
> > but rather just adds it to a "bucket brigade" which collects more and
> > more data.
>
> No, it doesn't collect more and more data, unless some filter needs to
> buffer the entire output. Normally it passes data down the chain.
> Each filter's job is to process a chunk of data then pass it to the next.
In my tests definitely all the data was being collected, and I was not
using any output filter (at least not that I know of) - not using
deflate or anything of that sort.
I'm no longer sure about my original statement that the buffering happens
when the client reads the output slowly. In fact, it now looks to me
that extreme memory use actually happens when the client reads very
very quickly (i.e., the client is through localhost). I haven't got a
clue why this is happening - I don't suppose Apache has any time-based
bucket-brigade flow control or memory pool reuse algorithms?
> > I confirmed that this is indeed a flow-control problem by changing the
> > CGI to sleep for 1 second after outputting each 64 MB (i.e., 8 batches
> > of 64 MB output); Now, the memory usage was around 64 MB, not 512 MB,
> > because Apache had the time to output each batch and free its memory
> > before the next batch came.
>
> Sounds like the entire contents of the pipe got read into memory in
> a single read! Not good, but not as bad as you think.
Is this actually possible? Doesn't Apache allocate a relatively small
buffer and read into that? How can it read 512 MB in a single read?
> Sleeping is a drastic workaround. What happens if you just flush your
> CGI output every 8Mb (or, preferably, in smaller chunks than that)?
The CGI is a trivial one written in C, using stdio and puts()'ing 8192
strings of 65536 bytes each. I don't think that stdio buffers 512 MB
or anything close to that. Also like I said, the CGI program itself does
NOT grow in memory use - just Apache.
> You might want to look at the mod_proxy framework as an alternative harness
> to run your program.
Interesting idea. I'll take a look at that.
Thanks,
Nadav.
--
Nadav Har'El | Tuesday, Jul 17 2012, 27 Tammuz 5772
nyh@math.technion.ac.il |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |Despite the cost of living, have you
http://nadav.harel.org.il |noticed how it remains so popular?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod-cgi reads entire output into memory...
Posted by Nick Kew <ni...@webthing.com>.
On Mon, 16 Jul 2012 17:07:23 +0300
Nadav Har'El <ny...@math.technion.ac.il> wrote:
> When I use it in Apache, Apache itself (NOT the CGI process!) grows
> by 512 MB (!). I was really surprised by this, because ideally Apache
> should hardly grow at all, as at most (if at all) it should be reading
> modest-sized buffers from the CGI script and writing them back to the
> socket.
Indeed, that would be normal behaviour. I haven't encountered this problem
when generating large (albeit not quite that large) CGI output.
> I looked at the httpd code, discovered (if I understand correctly) that
> 1. As I already guessed, Apache doesn't let the CGI write directly to the
> socket, but rather asks it to write to a pipe, which Apache then reads.
Yep. That's what CGI is all about.
> 2. When Apache reads this data from the pipe, it doesn't write it directly
> but rather just adds it to a "bucket brigade" which collects more and
> more data.
No, it doesn't collect more and more data, unless some filter needs to
buffer the entire output. Normally it passes data down the chain.
Each filter's job is to process a chunk of data then pass it to the next.
> I confirmed that this is indeed a flow-control problem by changing the
> CGI to sleep for 1 second after outputting each 64 MB (i.e., 8 batches
> of 64 MB output); Now, the memory usage was around 64 MB, not 512 MB,
> because Apache had the time to output each batch and free its memory
> before the next batch came.
Sounds like the entire contents of the pipe got read into memory in
a single read! Not good, but not as bad as you think.
Sleeping is a drastic workaround. What happens if you just flush your
CGI output every 8Mb (or, preferably, in smaller chunks than that)?
> So now I guess my questions are:
>
> 1. Has anyone ever thought of doing a "direct CGI" module, where the CGI
> script writes directly to the socket, not to Apache's pipe, forgoing
> any copying, buffering or filtering in Apache?
> Does something like this already exist? Is "NPH" relevant here?
That would not be CGI. But there are indeed several CGI modules available:
others may behave differently.
You might want to look at the mod_proxy framework as an alternative harness
to run your program.
> 2. Even if we do want Apache's output filtering capabilities, are there
> really no flow control capabilities? Can we tell Apache not to read
> more input (i.e., CGI's output) if the bucket brigade is larger than
> some predefined size (e.g., 1 MB)?
A bug report might or might not attract a fix.
> 3. Some of you may think that CGI is antiquated, and I shouldn't be
> using it - but I do have good reasons to use it ;-) But I wonder (I
> didn't test) - is this problem specific to CGI? What happens when we
> serve a huge disk *file*, and we can read it faster than we can send
> it - does the bucket brigade also grow indefinitely?
Nothing to do with how fast you read it vs send it.
But you might want to trawl the dev list for when mod_substitute
was being developed, and the problem you describe motivated one
aspect of its implementation. Of course that fix wouldn't apply
unless you had mod_substitute filtering your CGI output.
--
Nick Kew
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org