Memory usage for libareq uploads

[Jeremy Howard <jh...@fastmail.fm>]

Way back in the annals of history (well, err, 24 Feb 2000, anyway) there
was some discussion about uploads chewing up memory in libareq...

Jim Winstead said:
<quote>
Is it just me, or is it pretty dodgy code? It allocates memory like
its going out of style (it doesn't appear to reuse any of the
buffers it allocates from the pool), resulting in server processes
that balloon up to five times the uploaded file's size. Its also
appears to be flat-out slow because of all the extra work it is
doing.

Just an observation. I (or a coworker if I'm lucky :) will be
rolling up my shirt-sleeves and diving in tomorrow. This is just
my found-the-leak-now-I-get-to-sleep-on-it-and-hope-it-was-fixed-
already-or-I'm-being-dumb mail.
</quote>

to which Doug replied:
<quote>
it's pretty dodgy code :)  the multipart buffer code in libapreq is
actually just a translation from CGI.pm's Perl code to C.  notice the
comment: /*XXX: we can optimize this loop*/ 
meaning, I'm aware it's not reusing the buffer, but it's using a subpool
which is cleared at the end of multipart_buffer_fill, each call to that
should be no more than 4k or so.  I have a recent mail (not to the list)
from somebody who experienced the balloon, looked into it, and thinks
that
ap_clear_pool() is not working.  anyhow, my plan is to get 1.22 out, then
revisit libapreq, then onto modperl-2.0
</quote>

Well, I'm facing this problem now. The parms() method (or parse() or
anything else that calls parms()) chews up enormous amounts of memory
when uploading large files. I like to give my webmail users freedom to
upload pretty much whatever size attachments they like, but that means my
Apache processes grow very large, and never shrink again.

What can I do to avoid this? I know I can use one of the various
directives that kills off large processes, but that seems a bit
aggressive... Has anyone got any sample code that spins off file upload
activity to an external CGI and then returns to a mod_perl handler when
the dirty work is done? ...Or is there some new code out there (or coming
Real Soon Now) that avoids this whole issue?

TIA,
  Jeremy

-- 
  Jeremy Howard
  jeremy@fastmail.fm

Re: Memory usage for libareq uploads

[Jie Gao <J....@isu.usyd.edu.au>]

On Tue, 6 Jun 2000, Jeremy Howard wrote:
>...
> 
> Well, I'm facing this problem now. The parms() method (or parse() or
> anything else that calls parms()) chews up enormous amounts of memory
> when uploading large files. I like to give my webmail users freedom to
> upload pretty much whatever size attachments they like, but that means my
> Apache processes grow very large, and never shrink again.
> 
> What can I do to avoid this? I know I can use one of the various
> directives that kills off large processes, but that seems a bit
> aggressive... Has anyone got any sample code that spins off file upload
> activity to an external CGI and then returns to a mod_perl handler when
> the dirty work is done? ...Or is there some new code out there (or coming
> Real Soon Now) that avoids this whole issue?

What I do is kill the child process after uploading is finished:
    
    $r->child_terminate();

Not an elegant way, but it works for the time being.


Jie