You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by "John S. Evans" <jo...@corp.notifyme.com> on 2000/04/11 02:32:52 UTC
Apache::Request
I'm looking for some help/advice with Apache::Request. I'm currently using
Apache::Request to parse the POST that is used to upload a bunch of files to
our server.
The problem I'm running into is that I seem to be running out of file
descriptors over a long period of time, if a large number of files (between
50 and 200) are posted at once.
I've noticed that Apache::Request seems to take advantage of the Apache
memory pools to do a lot of it's cleanup - all memory is allocated out of
the pools, and the temp files it creates are attached to the pools (via
ap_note_cleanups_for_file()) so they are closed when it terminates.
I have, on occasion, gotten the error message "[libapreq] could not create
tmpfile()" in my apache log file, and returned to my code by parse().
As time goes on, then number of POSTed files required to get the error
message seems to decrease (currently it's around 10).
Any clues? Here's what I can imagine might be happening (pick one, or
suggest another):
- the error condition (presumably caused by someone attempting to POST too
many files at once) causes a "leak" of the files in that POST.
- as the number of apache child processes increase, the max number of files
each one can handle decreases.
- none of the above.
Anyway, I'd appreciate any pointers anyone can give. I've never used the C
Apache Module API before, so I'm making a lot of assumptions here...
Thanks!
-jse
Apache::Request
Posted by "John S. Evans" <su...@jsevans.com>.
I'm looking for some help/advice with Apache::Request. I'm currently using
Apache::Request to parse the POST that is used to upload a bunch of files to
our server.
The problem I'm running into is that I seem to be running out of file
descriptors over a long period of time, if a large number of files (between
50 and 200) are posted at once.
I've noticed that Apache::Request seems to take advantage of the Apache
memory pools to do a lot of it's cleanup - all memory is allocated out of
the pools, and the temp files it creates are attached to the pools (via
ap_note_cleanups_for_file()) so they are closed when it terminates.
I have, on occasion, gotten the error message "[libapreq] could not create
tmpfile()" in my apache log file, and returned to my code by parse().
As time goes on, then number of POSTed files required to get the error
message seems to decrease (currently it's around 10).
Any clues? Here's what I can imagine might be happening (pick one, or
suggest another):
- the error condition (presumably caused by someone attempting to POST too
many files at once) causes a "leak" of the files in that POST.
- as the number of apache child processes increase, the max number of files
each one can handle decreases.
- none of the above.
Anyway, I'd appreciate any pointers anyone can give. I've never used the C
Apache Module API before, so I'm making a lot of assumptions here...
Thanks!
-jse
Re: Apache::Request
Posted by Michael hall <mh...@riverside.org>.
On Tue, Apr 11, 2000 at 11:52:44PM -0700, John S. Evans wrote:
> I saw (in the code) that there's one open file per uploaded file. That
> should be fine. I just need to find out if they're getting closed
> correctly.
>
> What is "lsof"?
'LiSt Open Files', its really a handy tool for diagnosing. I just
happened to be setting up a new machine today and checked for the
latest release to grab and install.
Its available at 'ftp://vic.cc.purdue.edu/pub/tools/unix/lsof'.
--
The results of your IQ test are back, they're negative.
Mike Hall,
Unix Admin - Rock Island Communications <mi...@rockisland.com>
System Admin - riverside.org <mh...@riverside.org>
Re: detecting fd leaks (was Re: Apache::Request)
Posted by Stas Bekman <sb...@stason.org>.
On Thu, 20 Apr 2000, Doug MacEachern wrote:
> On Thu, 13 Apr 2000, Stas Bekman wrote:
>
> > > I have no real conclusion to reach, except that it seems to be leaking
> > > files.
> >
> > Well, I wanted to write Apache::FileLeak or an extension to
> > Apache::VMonitor to show the opened file descriptors, the files and the
> > processes that have opened them, but this requires a root access unless
> > you want to see the information about the current process only.
>
> what i've found to be a nice solution for such a problem, is to use PlRPC,
> where the server is setuid root. access is limited to 127.0.0.1, which
> might not be enough, but is perfect for something like Audio-RaveMP
> (on cpan), it might also be a reasonable solution for your
> Apache::FileLeak.
Wow, looks like a nice toy^H^H^Htool :) I'll try it.
Thanks, Doug!
______________________________________________________________________
Stas Bekman | JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ | mod_perl Guide http://perl.apache.org/guide
mailto:stas@stason.org | http://perl.org http://stason.org/TULARC/
http://singlesheaven.com| http://perlmonth.com http://sourcegarden.org
----------------------------------------------------------------------
Re: detecting fd leaks (was Re: Apache::Request)
Posted by Doug MacEachern <do...@covalent.net>.
On Thu, 13 Apr 2000, Stas Bekman wrote:
> > I have no real conclusion to reach, except that it seems to be leaking
> > files.
>
> Well, I wanted to write Apache::FileLeak or an extension to
> Apache::VMonitor to show the opened file descriptors, the files and the
> processes that have opened them, but this requires a root access unless
> you want to see the information about the current process only.
what i've found to be a nice solution for such a problem, is to use PlRPC,
where the server is setuid root. access is limited to 127.0.0.1, which
might not be enough, but is perfect for something like Audio-RaveMP
(on cpan), it might also be a reasonable solution for your
Apache::FileLeak.
detecting fd leaks (was Re: Apache::Request)
Posted by Stas Bekman <sb...@stason.org>.
> I have no real conclusion to reach, except that it seems to be leaking
> files.
Well, I wanted to write Apache::FileLeak or an extension to
Apache::VMonitor to show the opened file descriptors, the files and the
processes that have opened them, but this requires a root access unless
you want to see the information about the current process only.
The other problem is that I couldn't find a generic cross platform
library, that provides this kind of information.
I've suggested to have a hook in the parent process that will be able to
deliver this only root accessible information to a non-root child. Just
like the information about apache processes can be retrieved by any child
thru mod_status or Apache::VMonitor (Apache::Scoreboard).
On linux I'd retrieve the information about opened fds by reading
/proc/PID/fd directory, probably it's the same on every OS that supports
proc fs.
So the two problems to be solved are:
1) finding a generic solution to retrieve the fd info as root
2) making this information available to the child processes
Ideas?
______________________________________________________________________
Stas Bekman | JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ | mod_perl Guide http://perl.apache.org/guide/
mailto:stas@stason.org | http://perl.org http://stason.org/TULARC/
http://singlesheaven.com| http://perlmonth.com http://sourcegarden.org
----------------------------------------------------------------------
Re: Apache::Request
Posted by "John S. Evans" <jo...@corp.notifyme.com>.
Ok. So I've installed lsof, although I'm a little unclear on how to use it.
Right now I'm pretty brute force. Here's what I execute:
lsof | grep httpd | grep myuser | wc -l
This (hopefully) gives me the number of files that my httpd children (owner
"myuser") have open.
When I first file up apache with my code loaded, it is 171. Then I start
submitting requests, posting a single file to Apache::Request with each
POST. Here's how the count changes with each request:
171 (starting point)
215
225
235
245
255
265
266
267
268
269
270
I have no real conclusion to reach, except that it seems to be leaking
files.
I then ran the same test on a server that had received a couple hundred
files over it's lifespan. It was at 1472. It was giving me the following
error on every new POST:
[Thu Apr 13 01:06:12 2000] [error] [Thu Apr 13 01:06:12 2000] null: fdopen
failed! at /usr/local/apache/notifi/perl/Handler/SubmitHandler.pm
line 285, <_GEN_243> chunk 15.
That line of code (line 285): "$blockFile = $nextBlock->fh()". It asks the
upload block for it's filehandle.
Anyway, short of using some of the other MIME parsing modules to do the
content parse myself, does anyone else have some suggestions? Unless Apache
isn't respecting the Apache::Request request to free the filehandles when
the pool is destroyed (or unless the pool is never destroyed!) I don't see
how the files aren't going away.
Thanks for the advice so far! If you have better instructions for using
lsof, I'd also appreciate that!
-jse
> From: Doug MacEachern <do...@covalent.net>
> Date: Wed, 12 Apr 2000 22:48:06 -0700 (PDT)
> To: "John S. Evans" <jo...@corp.notifyme.com>
> Cc: modperl <mo...@apache.org>
> Subject: Re: Apache::Request
>
> On Tue, 11 Apr 2000, John S. Evans wrote:
>
>> I'm using Solaris (SunOS 5.7, according to uname).
>>
>> The number of files varies, and I can control this if I know what the limits
>> are. Is the 256 limit per process or for the entire machine? For instance,
>> if I have 10 apache children going full bore, is the practical limit 25 per
>> child, or 256 per child?
>
> i think it's per-child.
>
>> I saw (in the code) that there's one open file per uploaded file. That
>> should be fine. I just need to find out if they're getting closed
>> correctly.
>>
>> What is "lsof"?
>
> look it up on freshmeat.net, it'll show you what files the process has
> open. should be useful to see if any files are not being closed.
>
Re: Apache::Request
Posted by Doug MacEachern <do...@covalent.net>.
On Tue, 11 Apr 2000, John S. Evans wrote:
> I'm using Solaris (SunOS 5.7, according to uname).
>
> The number of files varies, and I can control this if I know what the limits
> are. Is the 256 limit per process or for the entire machine? For instance,
> if I have 10 apache children going full bore, is the practical limit 25 per
> child, or 256 per child?
i think it's per-child.
> I saw (in the code) that there's one open file per uploaded file. That
> should be fine. I just need to find out if they're getting closed
> correctly.
>
> What is "lsof"?
look it up on freshmeat.net, it'll show you what files the process has
open. should be useful to see if any files are not being closed.
Re: Apache::Request
Posted by "John S. Evans" <jo...@corp.notifyme.com>.
I'm using Solaris (SunOS 5.7, according to uname).
The number of files varies, and I can control this if I know what the limits
are. Is the 256 limit per process or for the entire machine? For instance,
if I have 10 apache children going full bore, is the practical limit 25 per
child, or 256 per child?
I saw (in the code) that there's one open file per uploaded file. That
should be fine. I just need to find out if they're getting closed
correctly.
What is "lsof"?
Thanks!
-jse
> From: Doug MacEachern <do...@covalent.net>
> Date: Tue, 11 Apr 2000 21:26:19 -0700 (PDT)
> To: "John S. Evans" <jo...@corp.notifyme.com>
> Cc: modperl <mo...@apache.org>
> Subject: Re: Apache::Request
>
> On Mon, 10 Apr 2000, John S. Evans wrote:
>
>> I'm looking for some help/advice with Apache::Request. I'm currently using
>> Apache::Request to parse the POST that is used to upload a bunch of files to
>> our server.
>
> how many files? what os are you using? solaris has a 256 limit.
>
>> The problem I'm running into is that I seem to be running out of file
>> descriptors over a long period of time, if a large number of files (between
>> 50 and 200) are posted at once.
>
> there will be an open FILE* for each file uploaded in a given request,
> none of which are closed until the request is over. try using lsof to see
> if any of them are not being closed after the request is done.
>
Re: Apache::Request
Posted by Doug MacEachern <do...@covalent.net>.
On Mon, 10 Apr 2000, John S. Evans wrote:
> I'm looking for some help/advice with Apache::Request. I'm currently using
> Apache::Request to parse the POST that is used to upload a bunch of files to
> our server.
how many files? what os are you using? solaris has a 256 limit.
> The problem I'm running into is that I seem to be running out of file
> descriptors over a long period of time, if a large number of files (between
> 50 and 200) are posted at once.
there will be an open FILE* for each file uploaded in a given request,
none of which are closed until the request is over. try using lsof to see
if any of them are not being closed after the request is done.