You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Graham Frank <gf...@neoservers.com> on 2008/01/11 02:36:19 UTC
[users@httpd] RSS of Apache Processes
Hello All,
I have three web servers which periodically crash due to Apache using up all
of the RAM on the server. No matter how much RAM we put in, it always
happens. Currently, there's 16GB of RAM on each server running Linux
x86_64.
Watching top, and the RSS of the Apache processes constantly grow. When
Apache first starts, the average RSS is 20MB per process. Within an hour or
so, it's 128MB per process. Do the math, and at 200 processes that's 25GB.
We've imposed memory limits in PHP, we use RLimitMEM in Apache (both are set
to 128M). I even wrote a Perl script which runs continuously checking all
Apache processes that kills individual children if they go over 80MB for
their RSS value. What's interesting is that when the average RSS hits 80MB
and my script kills everything, the RSS value resets back to 20MB and the
process starts over again.
Apache has PHP 5.2.5 compiled in as a DSO module using Apache 2.2.3. The
server is an Intel Xeon 5300 sequence server using the Linux kernel
2.6.18-53.1.4.el5smp.
Can anyone offer up an explanation for this? Thanks.
----
Graham Frank
Neoservers LLC - Founder and Owner
Ph: (608) 359-1593
Member of the Better Business Bureau
Re: [users@httpd] RSS of Apache Processes
Posted by Krist van Besien <kr...@gmail.com>.
On Jan 11, 2008 3:23 AM, Graham Frank <gf...@neoservers.com> wrote:
> Also,
>
> If it were a memory leak, as it appears to be, why do the new processes
> immediately go to the high RSS values?
Let me guess: Some of the memory include in the RSS value is actually
memory that is shared by all processes.
Krist
--
krist.vanbesien@gmail.com
krist@vanbesien.org
Bremgarten b. Bern, Switzerland
--
A: It reverses the normal flow of conversation.
Q: What's wrong with top-posting?
A: Top-posting.
Q: What's the biggest scourge on plain text email discussions?
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] RSS of Apache Processes
Posted by Graham Frank <gf...@neoservers.com>.
Yeah, I read that part. But, I could always hope that the documentation was
wrong. Haha.
This is just mind-numbingly annoying.
----
Graham Frank
Neoservers LLC Founder and Owner
Ph: (608) 359-1593
Member of the Better Business Bureau
-----Original Message-----
From: Torsten Foertsch [mailto:torsten.foertsch@gmx.net]
Sent: Saturday, January 12, 2008 4:59 AM
To: users@httpd.apache.org
Cc: Graham Frank
Subject: Re: [users@httpd] RSS of Apache Processes
On Sat 12 Jan 2008, Graham Frank wrote:
> Correct me if I'm wrong, but should RLimitMem help prevent the RSS value
>
> >from going insane? I disabled my Perl script on one of the web servers
>
> today, and after 11 hours each process got up to a RSS of 550MB each.
http://httpd.apache.org/docs/2.2/mod/core.html#rlimitmem says:
"This applies to processes forked off from Apache children servicing
requests,
not the Apache children themselves. This includes CGI scripts and SSI exec
commands, but not any processes forked off from the Apache parent such as
piped logs."
Does that help?
Torsten
--
A: It reverses the normal flow of conversation.
Q: What's wrong with top-posting?
A: Top-posting.
Q: What's the biggest scourge on plain text email discussions?
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] RSS of Apache Processes
Posted by Torsten Foertsch <to...@gmx.net>.
On Sat 12 Jan 2008, Graham Frank wrote:
> Correct me if I'm wrong, but should RLimitMem help prevent the RSS value
>
> >from going insane? I disabled my Perl script on one of the web servers
>
> today, and after 11 hours each process got up to a RSS of 550MB each.
http://httpd.apache.org/docs/2.2/mod/core.html#rlimitmem says:
"This applies to processes forked off from Apache children servicing requests,
not the Apache children themselves. This includes CGI scripts and SSI exec
commands, but not any processes forked off from the Apache parent such as
piped logs."
Does that help?
Torsten
--
A: It reverses the normal flow of conversation.
Q: What's wrong with top-posting?
A: Top-posting.
Q: What's the biggest scourge on plain text email discussions?
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] RSS of Apache Processes
Posted by Graham Frank <gf...@neoservers.com>.
Correct me if I'm wrong, but should RLimitMem help prevent the RSS value
from going insane? I disabled my Perl script on one of the web servers
today, and after 11 hours each process got up to a RSS of 550MB each.
----
Graham Frank
Neoservers LLC - Founder and Owner
Ph: (608) 359-1593
Member of the Better Business Bureau
-----Original Message-----
From: Graham Frank [mailto:gfrank@neoservers.com]
Sent: Friday, January 11, 2008 7:18 AM
To: users@httpd.apache.org
Subject: RE: [users@httpd] RSS of Apache Processes
Hi,
It looks like that's the case, yes.
----
Graham Frank
Neoservers LLC - Founder and Owner
Ph: (608) 359-1593
Member of the Better Business Bureau
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] RSS of Apache Processes
Posted by Graham Frank <gf...@neoservers.com>.
Hi,
It looks like that's the case, yes.
----
Graham Frank
Neoservers LLC - Founder and Owner
Ph: (608) 359-1593
Member of the Better Business Bureau
-----Original Message-----
From: Eric Covener [mailto:covener@gmail.com]
Sent: Friday, January 11, 2008 7:15 AM
To: users@httpd.apache.org
Subject: Re: [users@httpd] RSS of Apache Processes
On Jan 10, 2008 9:23 PM, Graham Frank <gf...@neoservers.com> wrote:
> Also,
>
> If it were a memory leak, as it appears to be, why do the new processes
> immediately go to the high RSS values?
Does the rss of the parent process increase as well? That would
explain the high rss in new children.
--
Eric Covener
covener@gmail.com
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] RSS of Apache Processes
Posted by Eric Covener <co...@gmail.com>.
On Jan 10, 2008 9:23 PM, Graham Frank <gf...@neoservers.com> wrote:
> Also,
>
> If it were a memory leak, as it appears to be, why do the new processes
> immediately go to the high RSS values?
Does the rss of the parent process increase as well? That would
explain the high rss in new children.
--
Eric Covener
covener@gmail.com
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] RSS of Apache Processes
Posted by Graham Frank <gf...@neoservers.com>.
Also,
If it were a memory leak, as it appears to be, why do the new processes
immediately go to the high RSS values? My Perl script kills all of the
processes over the limit and voila the leak is reset. Being a prefork
environment, should the leak happen independently in each process and not be
impacted by the other procs?
----
Graham Frank
Neoservers LLC - Founder and Owner
Ph: (608) 359-1593
Member of the Better Business Bureau
-----Original Message-----
From: Graham Frank [mailto:gfrank@neoservers.com]
Sent: Thursday, January 10, 2008 8:19 PM
To: users@httpd.apache.org; colm@allcosts.net
Subject: RE: [users@httpd] RSS of Apache Processes
Hey,
We actually have the MaxRequestsPerChild set to 1000 and KeepAlive off.
Know of any way that we could find the leak considering the vastness of all
the possibilities?
Thanks.
----
Graham Frank
Neoservers LLC - Founder and Owner
Ph: (608) 359-1593
Member of the Better Business Bureau
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] RSS of Apache Processes
Posted by Colm MacCarthaigh <co...@allcosts.net>.
On Thu, Jan 10, 2008 at 08:18:54PM -0600, Graham Frank wrote:
> We actually have the MaxRequestsPerChild set to 1000 and KeepAlive off.
That means the leak is more than 100KB per request on average, although
it could be one very bad leak :/
> Know of any way that we could find the leak considering the vastness of all
> the possibilities?
Unfortunately this isn't so simple, it requires actually debugging the
code. You can use a tracer or debugger attached to one of the children
to try and determine which requests trigger the leak. Another less
involved way might be to replay the access log against another instance
of apache, line by line (or perform a binary search, splitting the
requests in half progressively) and checking the RSS each time.
--
Colm MacCárthaigh Public Key: colm+pgp@stdlib.net
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] RSS of Apache Processes
Posted by Graham Frank <gf...@neoservers.com>.
Hey,
We actually have the MaxRequestsPerChild set to 1000 and KeepAlive off.
Know of any way that we could find the leak considering the vastness of all
the possibilities?
Thanks.
----
Graham Frank
Neoservers LLC Founder and Owner
Ph: (608) 359-1593
Member of the Better Business Bureau
-----Original Message-----
From: Colm MacCarthaigh [mailto:colm@allcosts.net]
Sent: Thursday, January 10, 2008 8:10 PM
To: users@httpd.apache.org
Subject: Re: [users@httpd] RSS of Apache Processes
On Thu, Jan 10, 2008 at 07:36:19PM -0600, Graham Frank wrote:
> Can anyone offer up an explanation for this? Thanks.
Everything you describe is incredibly normal behaviour associated with a
memory leak :-) You can also set a maximum number of requests per child
to alleviate the problem.
Of course the leak could be in Apache or php itself somewhere, but
vastly more likely is that some php code is responsible for it. Probably
the best place to look first, if you want to fix it.
--
Colm MacCárthaigh Public Key: colm+pgp@stdlib.net
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] RSS of Apache Processes
Posted by Colm MacCarthaigh <co...@allcosts.net>.
On Thu, Jan 10, 2008 at 07:36:19PM -0600, Graham Frank wrote:
> Can anyone offer up an explanation for this? Thanks.
Everything you describe is incredibly normal behaviour associated with a
memory leak :-) You can also set a maximum number of requests per child
to alleviate the problem.
Of course the leak could be in Apache or php itself somewhere, but
vastly more likely is that some php code is responsible for it. Probably
the best place to look first, if you want to fix it.
--
Colm MacCárthaigh Public Key: colm+pgp@stdlib.net
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org