You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Kamil Srot <ka...@nlogy.com> on 2005/08/02 11:56:39 UTC
[users@httpd] Memory consumption
Dear ALL,
Annotation:
my problem is, apache allocates so much memory, system wents out of
it and is unreachable for few minutes. I can help reporduce the problem,
please can someone help?
I found several similar issues in the bugreports but not exactly this
and they are quite old and not stalled for some reason... on top I'm not
sure if they are really the same - I don't know apache internals so good...
System:
apache 2.0.54 (selfcompiled)
PHP 4.4.0 (selfcompiled as module)
Debian Woody with latest updates
(more in-depth info available on request)
Server:
P4 3GHz with hyperthreading active
2GB RAM
2GB swap
(more in-depth info available on request)
Situation:
I have simple PHP script serving large files for autorized users...
after some time, all memory of the server is consumed by httpd processes
- their memory allocation does corelate with size of the files served...
I tried to use several methods of output prom PHP, flushing of data,
etc... it doesn't help... it seems apache does cache output of PHP into
RAM with NO LIMIT of memory allocation... clear and relyable way how to
reproduce the problem is ro resume broken download - Range headers seems
to make apache allocate whole output into memory immediatelly
Please can somebody help?
Thank you,
--
Kamil Srot
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Configuration access restriction to public_html.
Posted by Joshua Slive <js...@gmail.com>.
On 8/4/05, Chris Johnson <jo...@nmr.mgh.harvard.edu> wrote:
> > (But be careful; There are other ways to activate php including AddHandler.)
>
> Right. This particular server is still 1.3 at themoment but may
> change soon.
>
> So AddHandler and maybe SetHandler and possibly some others. Can
> these be turned off as well? There's still PERL that could get fired
> up here.
As I said, check the Override lines, which also exist in the 1.3 docs.
You'll probably find that your current setting is enough to prevent
most dynamic content from being activated in .htaccess files, but you
can also use the <Location> trick if you want to be extra sure.
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
[users@httpd] users-unsubscribe@httpd.apache.org
Posted by waRhawK <wa...@ufo-base.de>.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Configuration access restriction to public_html.
Posted by Chris Johnson <jo...@nmr.mgh.harvard.edu>.
On Wed, 3 Aug 2005, Joshua Slive wrote:
> On 8/3/05, Chris Johnson <jo...@nmr.mgh.harvard.edu> wrote:
> > Now, when I did this once before (it was a while ago) the
> > FileInfo override was what stopped .htaccess from doing AddType, yes?
> > That not being there is what does it. Right?
>
> Yes. See the "Override" line at
> http://httpd.apache.org/docs/2.0/mod/mod_mime.html#addtype
>
> (But be careful; There are other ways to activate php including AddHandler.)
>
> Joshua.
>
>
Right. This particular server is still 1.3 at themoment but may
change soon.
So AddHandler and maybe SetHandler and possibly some others. Can
these be turned off as well? There's still PERL that could get fired
up here.
-------------------------------------------------------------------------------
Chris Johnson |Internet: johnson@nmr.mgh.harvard.edu
Systems Administrator |Web: http://www.nmr.mgh.harvard.edu/~johnson
NMR Center |Voice: 617.726.0949
Mass. General Hospital |FAX: 617.726.7422
149 (2301) 13th Street |"Life is unfair. Anyone who tells you otherwise is
Charlestown, MA., 02129 USA |selling something." Wesley, "The Princess Bride"
-------------------------------------------------------------------------------
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Configuration access restriction to public_html.
Posted by Joshua Slive <js...@gmail.com>.
On 8/3/05, Chris Johnson <jo...@nmr.mgh.harvard.edu> wrote:
> Now, when I did this once before (it was a while ago) the
> FileInfo override was what stopped .htaccess from doing AddType, yes?
> That not being there is what does it. Right?
Yes. See the "Override" line at
http://httpd.apache.org/docs/2.0/mod/mod_mime.html#addtype
(But be careful; There are other ways to activate php including AddHandler.)
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Configuration access restriction to public_html.
Posted by Chris Johnson <jo...@nmr.mgh.harvard.edu>.
On Wed, 3 Aug 2005, Joshua Slive wrote:
> On 8/3/05, Chris Johnson <jo...@nmr.mgh.harvard.edu> wrote:
> > On Wed, 3 Aug 2005, Joshua Slive wrote:
> > > <Location /~>
> > > php_flag engine off
> > > </Location>
>
> >
> > For Apache PHP module, which we have. Not used to using Location
> > much. I only want to prevent its use in usr's public_html remember.
> > The above would turn it off everywhere. Are you then saying it should
> > be reenabled for other URLs? Can you please give an example?
>
> No -- notice the "~" above, which restricts applications to URLs
> beginning in "~". The advantage of using <Location> is that it is
> evaluated after .htaccess files and overrides them, so it is a good
> way to assure that .htaccess files are overridden. See
> http://httpd.apache.org/docs/2.0/howto/htaccess.html#how
>
> In your particular case, as long as you are turning on php only for
> specific directories that are not parent directories of your user
> directories, then you don't really need the <Location> section, since
> php will be off by default and your AllowOverride prevents it from
> being turned on in .htaccess files. The <Location> section is just an
> extra guarantee.
>
> Joshua.
>
I HAVE to get new eyes. Thank you.
Now, when I did this once before (it was a while ago) the
FileInfo override was what stopped .htaccess from doing AddType, yes?
That not being there is what does it. Right?
-------------------------------------------------------------------------------
Chris Johnson |Internet: johnson@nmr.mgh.harvard.edu
Systems Administrator |Web: http://www.nmr.mgh.harvard.edu/~johnson
NMR Center |Voice: 617.726.0949
Mass. General Hospital |FAX: 617.726.7422
149 (2301) 13th Street |Doctors don't save lives. The best they can hope
Charlestown, MA., 02129 USA |to do is save life. Not the same thing. Me
-------------------------------------------------------------------------------
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Configuration access restriction to public_html.
Posted by Joshua Slive <js...@gmail.com>.
On 8/3/05, Chris Johnson <jo...@nmr.mgh.harvard.edu> wrote:
> On Wed, 3 Aug 2005, Joshua Slive wrote:
> > <Location /~>
> > php_flag engine off
> > </Location>
>
> For Apache PHP module, which we have. Not used to using Location
> much. I only want to prevent its use in usr's public_html remember.
> The above would turn it off everywhere. Are you then saying it should
> be reenabled for other URLs? Can you please give an example?
No -- notice the "~" above, which restricts applications to URLs
beginning in "~". The advantage of using <Location> is that it is
evaluated after .htaccess files and overrides them, so it is a good
way to assure that .htaccess files are overridden. See
http://httpd.apache.org/docs/2.0/howto/htaccess.html#how
In your particular case, as long as you are turning on php only for
specific directories that are not parent directories of your user
directories, then you don't really need the <Location> section, since
php will be off by default and your AllowOverride prevents it from
being turned on in .htaccess files. The <Location> section is just an
extra guarantee.
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Configuration access restriction to public_html.
Posted by Chris Johnson <jo...@nmr.mgh.harvard.edu>.
On Wed, 3 Aug 2005, Joshua Slive wrote:
>
So now we know why I was confused.
> If you have php turned off by default, then the AllowOverride
> directive should prevent .htaccess files from being used to turn it on
> again. But you may want to add something like
>
> <Location /~>
> php_flag engine off
> </Location>
>
> to be sure.
>
> Joshua.
>
For Apache PHP module, which we have. Not used to using Location
much. I only want to prevent its use in usr's public_html remember.
The above would turn it off everywhere. Are you then saying it should
be reenabled for other URLs? Can you please give an example?
-------------------------------------------------------------------------------
Chris Johnson |Internet: johnson@nmr.mgh.harvard.edu
Systems Administrator |Web: http://www.nmr.mgh.harvard.edu/~johnson
NMR Center |Voice: 617.726.0949
Mass. General Hospital |FAX: 617.726.7422
149 (2301) 13th Street |"The two most abundant things in the Universe
Charlestown, MA., 02129 USA | are hydrogen and stupidity." Harlan Ellison
-------------------------------------------------------------------------------
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Configuration access restriction to public_html.
Posted by Joshua Slive <js...@gmail.com>.
On 8/3/05, Chris Johnson <jo...@nmr.mgh.harvard.edu> wrote:
> We need to prevent users with URLs of the ~<user> variety from
> doing PHP or other things which execute arbirtary code. But (there's
> always a but), at the same time we need to allow it for URLs not of
> that type, i.e. all other references. There's the following snippet
> from httpd.conf which we modified a tad
>
> So first question. Is that first Limit section necessary?
Usually, no. It depends on what you have in other sections of
httpd.conf that may deny access.
>
> Second question. What is it aboruit that second section, i.e. the
> LimitExcept, that dissallows PHP?
Nothing. That section does not disallow php; it disallows certain HTTP methods.
If you have php turned off by default, then the AllowOverride
directive should prevent .htaccess files from being used to turn it on
again. But you may want to add something like
<Location /~>
php_flag engine off
</Location>
to be sure.
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
[users@httpd] Configuration access restriction to public_html.
Posted by Chris Johnson <jo...@nmr.mgh.harvard.edu>.
Hey all,
We need to prevent users with URLs of the ~<user> variety from
doing PHP or other things which execute arbirtary code. But (there's
always a but), at the same time we need to allow it for URLs not of
that type, i.e. all other references. There's the following snippet
from httpd.conf which we modified a tad
#
# Control access to UserDir directories. The following is an example
# for a site where these directories are restricted to read-only.
#
<Directory "/homes/*/*/public_html">
# AllowOverride FileInfo AuthConfig Limit
AllowOverride AuthConfig Limit
Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec
<Limit GET POST OPTIONS PROPFIND>
Order allow,deny
Allow from all
</Limit>
<LimitExcept GET POST OPTIONS PROPFIND>
Order deny,allow
Deny from all
</LimitExcept>
</Directory>
The object is to prevent an AddType in a user's .htaccess from
allowing said user from doing PHP or whatever.
The goods news it seems to work. Now, I've lost track of the number
of times I've read the on-line run time configuation directives
documentation, including Limit and LimitExcept. I've used them for
years. I'm a little annoyed at find the above a bit convoluted. As I
understood it, the GET, POST, OPTIONS and PROPFIND methods were
already, by default, allowed.
So first question. Is that first Limit section necessary?
Second question. What is it aboruit that second section, i.e. the
LimitExcept, that dissallows PHP?
And last. Is there another/better way to do this?
Thank you.
-------------------------------------------------------------------------------
Chris Johnson |Internet: johnson@nmr.mgh.harvard.edu
Systems Administrator |Web: http://www.nmr.mgh.harvard.edu/~johnson
NMR Center |Voice: 617.726.0949
Mass. General Hospital |FAX: 617.726.7422
149 (2301) 13th Street |I'm continually amazed by mankind's seemingly
Charlestown, MA., 02129 USA |infinite capacity for stupidity. Me
-------------------------------------------------------------------------------
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Memory consumption
Posted by Joe Orton <jo...@redhat.com>.
On Tue, Aug 02, 2005 at 10:49:59PM +0200, Kamil Srot wrote:
> Joshua Slive wrote:
> >Indeed you have found the problem, I believe: Range headers. In order
> >to handle out-of-order range requests (which are allowed by the spec),
> >httpd must buffer the entire response. Of course, that is very bad
> >behavior. Instead, httpd should just ignore out-of-order range
> >requests (which the spec also allows) and serve the entire thing.
> >
> Do you know about any will to patch this and incorporate it into stable
> release?
> ...maybe current behavior is good in some cases (I don't understand) but
> I can imagine many people having the same problems as I do... maybe some
> configuration directive can take place in case both behaviors are correct...
This patch fixes the byterange filter memory consumption issue for
2.0.x: http://people.apache.org/~jorton/ap_byterange.diff
Regards,
joe
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Memory consumption
Posted by Kamil Srot <ka...@nlogy.com>.
Dear Joshua,
thank you for your reply!
Joshua Slive wrote:
>On 8/2/05, Kamil Srot <ka...@nlogy.com> wrote:
>
>
>
>>Situation:
>> I have simple PHP script serving large files for autorized users...
>>after some time, all memory of the server is consumed by httpd processes <CUT>
>>
>>
>Indeed you have found the problem, I believe: Range headers. In order
>to handle out-of-order range requests (which are allowed by the spec),
>httpd must buffer the entire response. Of course, that is very bad
>behavior. Instead, httpd should just ignore out-of-order range
>requests (which the spec also allows) and serve the entire thing.
>
Do you know about any will to patch this and incorporate it into stable
release?
...maybe current behavior is good in some cases (I don't understand) but
I can imagine many people having the same problems as I do... maybe some
configuration directive can take place in case both behaviors are correct...
Currently I was forced to switch to downloads directly from filesystem -
version 1 of apache didn't seen to have this problem, but I had other
issues which are all solved now... just this one remains... (as I hope :)
>There is probably some way to get httpd to
>ignore the range header, either using the RequestHeader directive or
>using one of the special environment variables. But I've never done
>it myself.
>
>
I got the same idea but failed to find any note about this in
documentation...
Any hints about what to look for?
I do appreciate your answer - thank you, Joshua.
Best Regards,
--
Kamil Srot
Re: [users@httpd] Memory consumption
Posted by Joshua Slive <js...@gmail.com>.
On 8/2/05, Kamil Srot <ka...@nlogy.com> wrote:
> Situation:
> I have simple PHP script serving large files for autorized users...
> after some time, all memory of the server is consumed by httpd processes
> - their memory allocation does corelate with size of the files served...
> I tried to use several methods of output prom PHP, flushing of data,
> etc... it doesn't help... it seems apache does cache output of PHP into
> RAM with NO LIMIT of memory allocation... clear and relyable way how to
> reproduce the problem is ro resume broken download - Range headers seems
> to make apache allocate whole output into memory immediatelly
Indeed you have found the problem, I believe: Range headers. In order
to handle out-of-order range requests (which are allowed by the spec),
httpd must buffer the entire response. Of course, that is very bad
behavior. Instead, httpd should just ignore out-of-order range
requests (which the spec also allows) and serve the entire thing. And
it shouldn't be necessary to do the buffering when the ranges are in
order. I know this has been discussed on the dev list, but I can't
remember the conclusion. There is probably some way to get httpd to
ignore the range header, either using the RequestHeader directive or
using one of the special environment variables. But I've never done
it myself.
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org