You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Preston Podaril <pp...@ispi.net> on 2005/01/14 16:26:35 UTC
[users@httpd] Weird Apache Memory Leak or something
We have been having some weird issues with Apache for several months
now. I have been trying to dig out the reason for it with truss and
lsof, but I have been unable to catch anything abnormal.
We are running Solaris 8. We upgraded to Apache (mod_ssl) 1.3.33 and
PHP 4.3.10 a week and a half ago. Before that we were running Apache
(mod_ssl) 1.3.29 and PHP 4.3.4. In both scenarios this problem is
occurring.
Our problem is that the libhttpd.ep processes appear to be leaking
memory for some reason. They also will go to sleep indefinitely (as
seen in a truss). The processes that are problematic will grow up to
80M in RES size, checking out a truss on them, doesn't seem to show
anything abnormal, nothing that would denote its size getting so large.
When we get a lot of these processes between 50M and 80M the machines
RAM and swap will get filled up, and then it will become fairly
unusable. I have left a truss go overnight on one of these processes
that is "dead" and this is all it would show, over and over again in
thet output of the truss.
sigsuspend(0xFFBDDE68) (sleeping...)
signotifywait() (sleeping...)
lwp_sema_wait(0xFDF0DE30) (sleeping...)
signotifywait() = 16
lwp_sigredirect(0, SIGUSR1, 0x00000000) = 0
sigsuspend(0xFFBDDE68) (sleeping...)
signotifywait() (sleeping...)
lwp_sema_wait(0xFDF0DE30) (sleeping...)
signotifywait() = 16
lwp_sigredirect(0, SIGUSR1, 0x00000000) = 0
It has proven really hard to determine what is causing the problem here.
MaxClients are set at 70 and 90 for the two different sized
machines in the cluster. I have messed with the MaxRequestsPerChild and
MaxKeepAliveRequests with no avail, both are set at 1000 currently.
Does anyone have any ideas on what might be causing this? or any other
testing utilities I can use to help track down the problem? I believe
it to be something with php or something php is using, although I have
nothing to show me that yet.
Thanks for any ideas!
--
Preston Podaril
ppod@ispi.net
Network Administrator
ispi of Lincoln
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Weird Apache Memory Leak or something
Posted by Preston Podaril <pp...@ispi.net>.
lsof also tells me that the client using that httpd process that is
stuck sleeping, still has its TCP connection open or atleast idly
libhttpd. 21766 nobody 4u IPv4 0x3000696d1a8 0t9834 TCP
cl1-9:*->62-3-252-225.dyn.gotadsl.co.uk:* (IDLE)
I know who had this connection, they were working on things in there,
their stuff hosed up, and links to tools were no longer working, so they
closed their browser. Shouldn't this connection close the TCP
connection when it is idle and move on to the next? Especially when our
httpd.conf has this declared.
KeepAliveTimeout 5
Thanks for your insight on this.
Preston Podaril
ppod@ispi.net
Network Administrator
ispi of Lincoln
Preston Podaril wrote:
> I have noticed in an lsof of a libhttpd.ep process on our machines that
> it has open /usr/local/ssl/lib/libssl.so.0.9.7 and
> /usr/local/ssl/lib/libcrypto.so.0.9.7, but in httpd.conf we have setup
> the following:
> <IfDefine SSL>
> LoadModule ssl_module libexec/libssl.so
> </IfDefine>
>
> and we are not starting with startssl, and according to our script this
> should be fine. Can anyone shed any light on why an apache process
> would be opening these files despite libssl not being loaded?
>
> Preston Podaril
> ppod@ispi.net
> Network Administrator
> ispi of Lincoln
>
>>>> Preston Podaril wrote:
>>>>
>>>>> We have been having some weird issues with Apache for several
>>>>> months now. I have been trying to dig out the reason for it with
>>>>> truss and lsof, but I have been unable to catch anything abnormal.
>>>>>
>>>>> We are running Solaris 8. We upgraded to Apache (mod_ssl) 1.3.33
>>>>> and PHP 4.3.10 a week and a half ago. Before that we were running
>>>>> Apache (mod_ssl) 1.3.29 and PHP 4.3.4. In both scenarios this
>>>>> problem is occurring.
>>>>>
>>>>> Our problem is that the libhttpd.ep processes appear to be leaking
>>>>> memory for some reason. They also will go to sleep indefinitely
>>>>> (as seen in a truss). The processes that are problematic will grow
>>>>> up to 80M in RES size, checking out a truss on them, doesn't seem
>>>>> to show anything abnormal, nothing that would denote its size
>>>>> getting so large. When we get a lot of these processes between 50M
>>>>> and 80M the machines RAM and swap will get filled up, and then it
>>>>> will become fairly unusable. I have left a truss go overnight on
>>>>> one of these processes that is "dead" and this is all it would
>>>>> show, over and over again in thet output of the truss.
>>>>>
>>>>> sigsuspend(0xFFBDDE68) (sleeping...)
>>>>> signotifywait() (sleeping...)
>>>>> lwp_sema_wait(0xFDF0DE30) (sleeping...)
>>>>> signotifywait() = 16
>>>>> lwp_sigredirect(0, SIGUSR1, 0x00000000) = 0
>>>>> sigsuspend(0xFFBDDE68) (sleeping...)
>>>>> signotifywait() (sleeping...)
>>>>> lwp_sema_wait(0xFDF0DE30) (sleeping...)
>>>>> signotifywait() = 16
>>>>> lwp_sigredirect(0, SIGUSR1, 0x00000000) = 0
>>>>>
>>>>> It has proven really hard to determine what is causing the problem
>>>>> here. MaxClients are set at 70 and 90 for the two different
>>>>> sized machines in the cluster. I have messed with the
>>>>> MaxRequestsPerChild and MaxKeepAliveRequests with no avail, both
>>>>> are set at 1000 currently.
>>>>>
>>>>> Does anyone have any ideas on what might be causing this? or any
>>>>> other testing utilities I can use to help track down the problem?
>>>>> I believe it to be something with php or something php is using,
>>>>> although I have nothing to show me that yet.
>>>>>
>>>>> Thanks for any ideas!
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> The official User-To-User support forum of the Apache HTTP Server
>>>> Project.
>>>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>>>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>>>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>>>> For additional commands, e-mail: users-help@httpd.apache.org
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> The official User-To-User support forum of the Apache HTTP Server
>>> Project.
>>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>>> For additional commands, e-mail: users-help@httpd.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> The official User-To-User support forum of the Apache HTTP Server
>> Project.
>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Weird Apache Memory Leak or something
Posted by Preston Podaril <pp...@ispi.net>.
I have noticed in an lsof of a libhttpd.ep process on our machines that
it has open /usr/local/ssl/lib/libssl.so.0.9.7 and
/usr/local/ssl/lib/libcrypto.so.0.9.7, but in httpd.conf we have setup
the following:
<IfDefine SSL>
LoadModule ssl_module libexec/libssl.so
</IfDefine>
and we are not starting with startssl, and according to our script this
should be fine. Can anyone shed any light on why an apache process
would be opening these files despite libssl not being loaded?
Preston Podaril
ppod@ispi.net
Network Administrator
ispi of Lincoln
>>> Preston Podaril wrote:
>>>
>>>> We have been having some weird issues with Apache for several months
>>>> now. I have been trying to dig out the reason for it with truss and
>>>> lsof, but I have been unable to catch anything abnormal.
>>>>
>>>> We are running Solaris 8. We upgraded to Apache (mod_ssl) 1.3.33
>>>> and PHP 4.3.10 a week and a half ago. Before that we were running
>>>> Apache (mod_ssl) 1.3.29 and PHP 4.3.4. In both scenarios this
>>>> problem is occurring.
>>>>
>>>> Our problem is that the libhttpd.ep processes appear to be leaking
>>>> memory for some reason. They also will go to sleep indefinitely (as
>>>> seen in a truss). The processes that are problematic will grow up
>>>> to 80M in RES size, checking out a truss on them, doesn't seem to
>>>> show anything abnormal, nothing that would denote its size getting
>>>> so large. When we get a lot of these processes between 50M and 80M
>>>> the machines RAM and swap will get filled up, and then it will
>>>> become fairly unusable. I have left a truss go overnight on one of
>>>> these processes that is "dead" and this is all it would show, over
>>>> and over again in thet output of the truss.
>>>>
>>>> sigsuspend(0xFFBDDE68) (sleeping...)
>>>> signotifywait() (sleeping...)
>>>> lwp_sema_wait(0xFDF0DE30) (sleeping...)
>>>> signotifywait() = 16
>>>> lwp_sigredirect(0, SIGUSR1, 0x00000000) = 0
>>>> sigsuspend(0xFFBDDE68) (sleeping...)
>>>> signotifywait() (sleeping...)
>>>> lwp_sema_wait(0xFDF0DE30) (sleeping...)
>>>> signotifywait() = 16
>>>> lwp_sigredirect(0, SIGUSR1, 0x00000000) = 0
>>>>
>>>> It has proven really hard to determine what is causing the problem
>>>> here. MaxClients are set at 70 and 90 for the two different
>>>> sized machines in the cluster. I have messed with the
>>>> MaxRequestsPerChild and MaxKeepAliveRequests with no avail, both are
>>>> set at 1000 currently.
>>>>
>>>> Does anyone have any ideas on what might be causing this? or any
>>>> other testing utilities I can use to help track down the problem? I
>>>> believe it to be something with php or something php is using,
>>>> although I have nothing to show me that yet.
>>>>
>>>> Thanks for any ideas!
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> The official User-To-User support forum of the Apache HTTP Server
>>> Project.
>>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>>> For additional commands, e-mail: users-help@httpd.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> The official User-To-User support forum of the Apache HTTP Server
>> Project.
>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Weird Apache Memory Leak or something
Posted by Preston Podaril <pp...@ispi.net>.
SunOS 5.8 Generic_108528-16 sun4u sparc SUNW,Ultra-Enterprise
to be more precise, that is our main file server, with all of the
webpage data stored on a raid device. Http requests are balanced out by
a coyote point systems load balancer, to currently 17 other sparc boxes
that mount their binaries and webpage data from the Enterprise server
through NFS. We have several clusters the load balancer deals with.
One that handles SSL requests (contains 1 server), one for the
media/static content (contains 3 servers), and one for the php and more
dynamically loaded content (contains 13 servers). The servers that do
not load the php modules (media/static ones) do not show any of the
problems the dynamic cluster shows, which is what makes me believe it
has something to do possibly with phps modules, or some libraries apache
is using while loading the php modules.
My efforts to determine what haven't seemed to be constant goose chases.
Preston Podaril
ppod@ispi.net
Network Administrator
ispi of Lincoln
Ivan Barrera A. wrote:
> oh.... didnt know that.
> Which architecture are you running solaris ?
>
> Preston Podaril wrote:
>
>> "truss" is "trace" on Solaris
>>
>> Preston Podaril
>> ppod@ispi.net
>> Network Administrator
>> ispi of Lincoln
>>
>>
>> Ivan Barrera A. wrote:
>>
>>> Sorry, but i just need to ask..
>>> where do i get "truss" ?
>>>
>>> Preston Podaril wrote:
>>>
>>>> We have been having some weird issues with Apache for several months
>>>> now. I have been trying to dig out the reason for it with truss and
>>>> lsof, but I have been unable to catch anything abnormal.
>>>>
>>>> We are running Solaris 8. We upgraded to Apache (mod_ssl) 1.3.33
>>>> and PHP 4.3.10 a week and a half ago. Before that we were running
>>>> Apache (mod_ssl) 1.3.29 and PHP 4.3.4. In both scenarios this
>>>> problem is occurring.
>>>>
>>>> Our problem is that the libhttpd.ep processes appear to be leaking
>>>> memory for some reason. They also will go to sleep indefinitely (as
>>>> seen in a truss). The processes that are problematic will grow up
>>>> to 80M in RES size, checking out a truss on them, doesn't seem to
>>>> show anything abnormal, nothing that would denote its size getting
>>>> so large. When we get a lot of these processes between 50M and 80M
>>>> the machines RAM and swap will get filled up, and then it will
>>>> become fairly unusable. I have left a truss go overnight on one of
>>>> these processes that is "dead" and this is all it would show, over
>>>> and over again in thet output of the truss.
>>>>
>>>> sigsuspend(0xFFBDDE68) (sleeping...)
>>>> signotifywait() (sleeping...)
>>>> lwp_sema_wait(0xFDF0DE30) (sleeping...)
>>>> signotifywait() = 16
>>>> lwp_sigredirect(0, SIGUSR1, 0x00000000) = 0
>>>> sigsuspend(0xFFBDDE68) (sleeping...)
>>>> signotifywait() (sleeping...)
>>>> lwp_sema_wait(0xFDF0DE30) (sleeping...)
>>>> signotifywait() = 16
>>>> lwp_sigredirect(0, SIGUSR1, 0x00000000) = 0
>>>>
>>>> It has proven really hard to determine what is causing the problem
>>>> here. MaxClients are set at 70 and 90 for the two different
>>>> sized machines in the cluster. I have messed with the
>>>> MaxRequestsPerChild and MaxKeepAliveRequests with no avail, both are
>>>> set at 1000 currently.
>>>>
>>>> Does anyone have any ideas on what might be causing this? or any
>>>> other testing utilities I can use to help track down the problem? I
>>>> believe it to be something with php or something php is using,
>>>> although I have nothing to show me that yet.
>>>>
>>>> Thanks for any ideas!
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> The official User-To-User support forum of the Apache HTTP Server
>>> Project.
>>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>>> For additional commands, e-mail: users-help@httpd.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> The official User-To-User support forum of the Apache HTTP Server
>> Project.
>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Weird Apache Memory Leak or something
Posted by "Ivan Barrera A." <Br...@Ivn.cl>.
oh.... didnt know that.
Which architecture are you running solaris ?
Preston Podaril wrote:
> "truss" is "trace" on Solaris
>
> Preston Podaril
> ppod@ispi.net
> Network Administrator
> ispi of Lincoln
>
>
> Ivan Barrera A. wrote:
>
>> Sorry, but i just need to ask..
>> where do i get "truss" ?
>>
>> Preston Podaril wrote:
>>
>>> We have been having some weird issues with Apache for several months
>>> now. I have been trying to dig out the reason for it with truss and
>>> lsof, but I have been unable to catch anything abnormal.
>>>
>>> We are running Solaris 8. We upgraded to Apache (mod_ssl) 1.3.33 and
>>> PHP 4.3.10 a week and a half ago. Before that we were running Apache
>>> (mod_ssl) 1.3.29 and PHP 4.3.4. In both scenarios this problem is
>>> occurring.
>>>
>>> Our problem is that the libhttpd.ep processes appear to be leaking
>>> memory for some reason. They also will go to sleep indefinitely (as
>>> seen in a truss). The processes that are problematic will grow up to
>>> 80M in RES size, checking out a truss on them, doesn't seem to show
>>> anything abnormal, nothing that would denote its size getting so
>>> large. When we get a lot of these processes between 50M and 80M the
>>> machines RAM and swap will get filled up, and then it will become
>>> fairly unusable. I have left a truss go overnight on one of these
>>> processes that is "dead" and this is all it would show, over and over
>>> again in thet output of the truss.
>>>
>>> sigsuspend(0xFFBDDE68) (sleeping...)
>>> signotifywait() (sleeping...)
>>> lwp_sema_wait(0xFDF0DE30) (sleeping...)
>>> signotifywait() = 16
>>> lwp_sigredirect(0, SIGUSR1, 0x00000000) = 0
>>> sigsuspend(0xFFBDDE68) (sleeping...)
>>> signotifywait() (sleeping...)
>>> lwp_sema_wait(0xFDF0DE30) (sleeping...)
>>> signotifywait() = 16
>>> lwp_sigredirect(0, SIGUSR1, 0x00000000) = 0
>>>
>>> It has proven really hard to determine what is causing the problem
>>> here. MaxClients are set at 70 and 90 for the two different
>>> sized machines in the cluster. I have messed with the
>>> MaxRequestsPerChild and MaxKeepAliveRequests with no avail, both are
>>> set at 1000 currently.
>>>
>>> Does anyone have any ideas on what might be causing this? or any
>>> other testing utilities I can use to help track down the problem? I
>>> believe it to be something with php or something php is using,
>>> although I have nothing to show me that yet.
>>>
>>> Thanks for any ideas!
>>
>>
>>
>> ---------------------------------------------------------------------
>> The official User-To-User support forum of the Apache HTTP Server
>> Project.
>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Weird Apache Memory Leak or something
Posted by Preston Podaril <pp...@ispi.net>.
in /etc/system:
set rlim_fd_cur=2048
on some of the cluster machines handling the requests.
set rlim_fd_cur=1024
on others.
Preston Podaril
ppod@ispi.net
Network Administrator
ispi of Lincoln
Robert F Hall wrote:
> Preston,
>
> What is your max file handle limit per process / user set to?
> We've seen some memory issues in httpd that were traceable to these limits.
>
> -Robert
>
> Preston Podaril wrote:
>
>> "truss" is "trace" on Solaris
>>
>> Preston Podaril
>> ppod@ispi.net
>> Network Administrator
>> ispi of Lincoln
>>
>>
>> Ivan Barrera A. wrote:
>>
>>> Sorry, but i just need to ask..
>>> where do i get "truss" ?
>>>
>>> Preston Podaril wrote:
>>>
>>>> We have been having some weird issues with Apache for several months
>>>> now. I have been trying to dig out the reason for it with truss and
>>>> lsof, but I have been unable to catch anything abnormal.
>>>>
>>>> We are running Solaris 8. We upgraded to Apache (mod_ssl) 1.3.33
>>>> and PHP 4.3.10 a week and a half ago. Before that we were running
>>>> Apache (mod_ssl) 1.3.29 and PHP 4.3.4. In both scenarios this
>>>> problem is occurring.
>>>>
>>>> Our problem is that the libhttpd.ep processes appear to be leaking
>>>> memory for some reason. They also will go to sleep indefinitely (as
>>>> seen in a truss). The processes that are problematic will grow up
>>>> to 80M in RES size, checking out a truss on them, doesn't seem to
>>>> show anything abnormal, nothing that would denote its size getting
>>>> so large. When we get a lot of these processes between 50M and 80M
>>>> the machines RAM and swap will get filled up, and then it will
>>>> become fairly unusable. I have left a truss go overnight on one of
>>>> these processes that is "dead" and this is all it would show, over
>>>> and over again in thet output of the truss.
>>>>
>>>> sigsuspend(0xFFBDDE68) (sleeping...)
>>>> signotifywait() (sleeping...)
>>>> lwp_sema_wait(0xFDF0DE30) (sleeping...)
>>>> signotifywait() = 16
>>>> lwp_sigredirect(0, SIGUSR1, 0x00000000) = 0
>>>> sigsuspend(0xFFBDDE68) (sleeping...)
>>>> signotifywait() (sleeping...)
>>>> lwp_sema_wait(0xFDF0DE30) (sleeping...)
>>>> signotifywait() = 16
>>>> lwp_sigredirect(0, SIGUSR1, 0x00000000) = 0
>>>>
>>>> It has proven really hard to determine what is causing the problem
>>>> here. MaxClients are set at 70 and 90 for the two different
>>>> sized machines in the cluster. I have messed with the
>>>> MaxRequestsPerChild and MaxKeepAliveRequests with no avail, both are
>>>> set at 1000 currently.
>>>>
>>>> Does anyone have any ideas on what might be causing this? or any
>>>> other testing utilities I can use to help track down the problem? I
>>>> believe it to be something with php or something php is using,
>>>> although I have nothing to show me that yet.
>>>>
>>>> Thanks for any ideas!
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> The official User-To-User support forum of the Apache HTTP Server
>>> Project.
>>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>>> For additional commands, e-mail: users-help@httpd.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> The official User-To-User support forum of the Apache HTTP Server
>> Project.
>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Weird Apache Memory Leak or something
Posted by Preston Podaril <pp...@ispi.net>.
Yeah I would wonder how this could cause it also. We haven't messed
with any kernel settings either, the servers are setup like they were
when I started working here, and we haven't always been having this
problem. I wish there had been something we had done when the problem
was noticed, some specific upgrade or something, but can't remember any,
and have found any clues on where to go.
Thanks again for your continued help with trying to dig this out.
Preston Podaril
ppod@ispi.net
Network Administrator
ispi of Lincoln
Qingshan Xie wrote:
>>What is your max file handle limit per process /
>>user set to? We've seen some memory issues in httpd
>
> that were traceable to these limits.
>
> Robert, could you explain why Max File Handle Limit
> would possibly cause the Memory Leak issues?
>
> Thanks,
> Q.Xie
>
>>-Robert
>>
>>Preston Podaril wrote:
>>
>>
>>>"truss" is "trace" on Solaris
>>>
>>>Preston Podaril
>>>ppod@ispi.net
>>>Network Administrator
>>>ispi of Lincoln
>>>
>>>
>>>Ivan Barrera A. wrote:
>>>
>>>
>>>>Sorry, but i just need to ask..
>>>>where do i get "truss" ?
>>>>
>>>>Preston Podaril wrote:
>>>>
>>>>
>>>>>We have been having some weird issues with
>>
>>Apache for several months
>>
>>>>>now. I have been trying to dig out the reason
>>
>>for it with truss and
>>
>>>>>lsof, but I have been unable to catch anything
>>
>>abnormal.
>>
>>>>>We are running Solaris 8. We upgraded to Apache
>>
>>(mod_ssl) 1.3.33
>>
>>>>>and PHP 4.3.10 a week and a half ago. Before
>>
>>that we were running
>>
>>>>>Apache (mod_ssl) 1.3.29 and PHP 4.3.4. In both
>>
>>scenarios this
>>
>>>>>problem is occurring.
>>>>>
>>>>>Our problem is that the libhttpd.ep processes
>>
>>appear to be leaking
>>
>>>>>memory for some reason. They also will go to
>>
>>sleep indefinitely (as
>>
>>>>>seen in a truss). The processes that are
>>
>>problematic will grow up
>>
>>>>>to 80M in RES size, checking out a truss on
>>
>>them, doesn't seem to
>>
>>>>>show anything abnormal, nothing that would
>>
>>denote its size getting
>>
>>>>>so large. When we get a lot of these processes
>>
>>between 50M and 80M
>>
>>>>>the machines RAM and swap will get filled up,
>>
>>and then it will
>>
>>>>>become fairly unusable. I have left a truss go
>>
>>overnight on one of
>>
>>>>>these processes that is "dead" and this is all
>>
>>it would show, over
>>
>>>>>and over again in thet output of the truss.
>>>>>
>>>>>sigsuspend(0xFFBDDE68) (sleeping...)
>>>>>signotifywait() (sleeping...)
>>>>>lwp_sema_wait(0xFDF0DE30) (sleeping...)
>>>>>signotifywait() = 16
>>>>>lwp_sigredirect(0, SIGUSR1, 0x00000000) = 0
>>>>>sigsuspend(0xFFBDDE68) (sleeping...)
>>>>>signotifywait() (sleeping...)
>>>>>lwp_sema_wait(0xFDF0DE30) (sleeping...)
>>>>>signotifywait() = 16
>>>>>lwp_sigredirect(0, SIGUSR1, 0x00000000) = 0
>>>>>
>>>>>It has proven really hard to determine what is
>>
>>causing the problem
>>
>>>>>here. MaxClients are set at 70 and 90 for
>>
>>the two different
>>
>>>>>sized machines in the cluster. I have messed
>>
>>with the
>>
>>>>>MaxRequestsPerChild and MaxKeepAliveRequests
>>
>>with no avail, both are
>>
>>>>>set at 1000 currently.
>>>>>
>>>>>Does anyone have any ideas on what might be
>>
>>causing this? or any
>>
>>>>>other testing utilities I can use to help track
>>
>>down the problem? I
>>
>>>>>believe it to be something with php or something
>>
>>php is using,
>>
>>>>>although I have nothing to show me that yet.
>>>>>
>>>>>Thanks for any ideas!
>>>>
>>>>
>>>>
>>>>
> ---------------------------------------------------------------------
>
>>>>The official User-To-User support forum of the
>>
>>Apache HTTP Server
>>
>>>>Project.
>>>>See <URL:http://httpd.apache.org/userslist.html>
>>
>>for more info.
>>
>>>>To unsubscribe, e-mail:
>>
>>users-unsubscribe@httpd.apache.org
>>
>>>> " from the digest:
>>
>>users-digest-unsubscribe@httpd.apache.org
>>
>>>>For additional commands, e-mail:
>>
>>users-help@httpd.apache.org
>>
>>>>
>>>
> ---------------------------------------------------------------------
>
>>>The official User-To-User support forum of the
>>
>>Apache HTTP Server
>>
>>>Project.
>>>See <URL:http://httpd.apache.org/userslist.html>
>>
>>for more info.
>>
>>>To unsubscribe, e-mail:
>>
>>users-unsubscribe@httpd.apache.org
>>
>>> " from the digest:
>>
>>users-digest-unsubscribe@httpd.apache.org
>>
>>>For additional commands, e-mail:
>>
>>users-help@httpd.apache.org
>>
>>>
>>
>>
> ---------------------------------------------------------------------
>
>>The official User-To-User support forum of the
>>Apache HTTP Server Project.
>>See <URL:http://httpd.apache.org/userslist.html> for
>>more info.
>>To unsubscribe, e-mail:
>>users-unsubscribe@httpd.apache.org
>> " from the digest:
>>users-digest-unsubscribe@httpd.apache.org
>>For additional commands, e-mail:
>>users-help@httpd.apache.org
>>
>>
>
>
>
>
>
> __________________________________
> Do you Yahoo!?
> Yahoo! Mail - 250MB free storage. Do more. Manage less.
> http://info.mail.yahoo.com/mail_250
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Weird Apache Memory Leak or something
Posted by Qingshan Xie <xi...@yahoo.com>.
> What is your max file handle limit per process /
> user set to? We've seen some memory issues in httpd
that were traceable to these limits.
Robert, could you explain why Max File Handle Limit
would possibly cause the Memory Leak issues?
Thanks,
Q.Xie
>
> -Robert
>
> Preston Podaril wrote:
>
> > "truss" is "trace" on Solaris
> >
> > Preston Podaril
> > ppod@ispi.net
> > Network Administrator
> > ispi of Lincoln
> >
> >
> > Ivan Barrera A. wrote:
> >
> >> Sorry, but i just need to ask..
> >> where do i get "truss" ?
> >>
> >> Preston Podaril wrote:
> >>
> >>> We have been having some weird issues with
> Apache for several months
> >>> now. I have been trying to dig out the reason
> for it with truss and
> >>> lsof, but I have been unable to catch anything
> abnormal.
> >>>
> >>> We are running Solaris 8. We upgraded to Apache
> (mod_ssl) 1.3.33
> >>> and PHP 4.3.10 a week and a half ago. Before
> that we were running
> >>> Apache (mod_ssl) 1.3.29 and PHP 4.3.4. In both
> scenarios this
> >>> problem is occurring.
> >>>
> >>> Our problem is that the libhttpd.ep processes
> appear to be leaking
> >>> memory for some reason. They also will go to
> sleep indefinitely (as
> >>> seen in a truss). The processes that are
> problematic will grow up
> >>> to 80M in RES size, checking out a truss on
> them, doesn't seem to
> >>> show anything abnormal, nothing that would
> denote its size getting
> >>> so large. When we get a lot of these processes
> between 50M and 80M
> >>> the machines RAM and swap will get filled up,
> and then it will
> >>> become fairly unusable. I have left a truss go
> overnight on one of
> >>> these processes that is "dead" and this is all
> it would show, over
> >>> and over again in thet output of the truss.
> >>>
> >>> sigsuspend(0xFFBDDE68) (sleeping...)
> >>> signotifywait() (sleeping...)
> >>> lwp_sema_wait(0xFDF0DE30) (sleeping...)
> >>> signotifywait() = 16
> >>> lwp_sigredirect(0, SIGUSR1, 0x00000000) = 0
> >>> sigsuspend(0xFFBDDE68) (sleeping...)
> >>> signotifywait() (sleeping...)
> >>> lwp_sema_wait(0xFDF0DE30) (sleeping...)
> >>> signotifywait() = 16
> >>> lwp_sigredirect(0, SIGUSR1, 0x00000000) = 0
> >>>
> >>> It has proven really hard to determine what is
> causing the problem
> >>> here. MaxClients are set at 70 and 90 for
> the two different
> >>> sized machines in the cluster. I have messed
> with the
> >>> MaxRequestsPerChild and MaxKeepAliveRequests
> with no avail, both are
> >>> set at 1000 currently.
> >>>
> >>> Does anyone have any ideas on what might be
> causing this? or any
> >>> other testing utilities I can use to help track
> down the problem? I
> >>> believe it to be something with php or something
> php is using,
> >>> although I have nothing to show me that yet.
> >>>
> >>> Thanks for any ideas!
> >>
> >>
> >>
> >>
>
---------------------------------------------------------------------
> >> The official User-To-User support forum of the
> Apache HTTP Server
> >> Project.
> >> See <URL:http://httpd.apache.org/userslist.html>
> for more info.
> >> To unsubscribe, e-mail:
> users-unsubscribe@httpd.apache.org
> >> " from the digest:
> users-digest-unsubscribe@httpd.apache.org
> >> For additional commands, e-mail:
> users-help@httpd.apache.org
> >>
> >>
> >
> >
>
---------------------------------------------------------------------
> > The official User-To-User support forum of the
> Apache HTTP Server
> > Project.
> > See <URL:http://httpd.apache.org/userslist.html>
> for more info.
> > To unsubscribe, e-mail:
> users-unsubscribe@httpd.apache.org
> > " from the digest:
> users-digest-unsubscribe@httpd.apache.org
> > For additional commands, e-mail:
> users-help@httpd.apache.org
> >
> >
>
>
>
---------------------------------------------------------------------
> The official User-To-User support forum of the
> Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for
> more info.
> To unsubscribe, e-mail:
> users-unsubscribe@httpd.apache.org
> " from the digest:
> users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail:
> users-help@httpd.apache.org
>
>
__________________________________
Do you Yahoo!?
Yahoo! Mail - 250MB free storage. Do more. Manage less.
http://info.mail.yahoo.com/mail_250
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Weird Apache Memory Leak or something
Posted by Robert F Hall <rf...@berkeley.edu>.
Preston,
What is your max file handle limit per process / user set to?
We've seen some memory issues in httpd that were traceable to these limits.
-Robert
Preston Podaril wrote:
> "truss" is "trace" on Solaris
>
> Preston Podaril
> ppod@ispi.net
> Network Administrator
> ispi of Lincoln
>
>
> Ivan Barrera A. wrote:
>
>> Sorry, but i just need to ask..
>> where do i get "truss" ?
>>
>> Preston Podaril wrote:
>>
>>> We have been having some weird issues with Apache for several months
>>> now. I have been trying to dig out the reason for it with truss and
>>> lsof, but I have been unable to catch anything abnormal.
>>>
>>> We are running Solaris 8. We upgraded to Apache (mod_ssl) 1.3.33
>>> and PHP 4.3.10 a week and a half ago. Before that we were running
>>> Apache (mod_ssl) 1.3.29 and PHP 4.3.4. In both scenarios this
>>> problem is occurring.
>>>
>>> Our problem is that the libhttpd.ep processes appear to be leaking
>>> memory for some reason. They also will go to sleep indefinitely (as
>>> seen in a truss). The processes that are problematic will grow up
>>> to 80M in RES size, checking out a truss on them, doesn't seem to
>>> show anything abnormal, nothing that would denote its size getting
>>> so large. When we get a lot of these processes between 50M and 80M
>>> the machines RAM and swap will get filled up, and then it will
>>> become fairly unusable. I have left a truss go overnight on one of
>>> these processes that is "dead" and this is all it would show, over
>>> and over again in thet output of the truss.
>>>
>>> sigsuspend(0xFFBDDE68) (sleeping...)
>>> signotifywait() (sleeping...)
>>> lwp_sema_wait(0xFDF0DE30) (sleeping...)
>>> signotifywait() = 16
>>> lwp_sigredirect(0, SIGUSR1, 0x00000000) = 0
>>> sigsuspend(0xFFBDDE68) (sleeping...)
>>> signotifywait() (sleeping...)
>>> lwp_sema_wait(0xFDF0DE30) (sleeping...)
>>> signotifywait() = 16
>>> lwp_sigredirect(0, SIGUSR1, 0x00000000) = 0
>>>
>>> It has proven really hard to determine what is causing the problem
>>> here. MaxClients are set at 70 and 90 for the two different
>>> sized machines in the cluster. I have messed with the
>>> MaxRequestsPerChild and MaxKeepAliveRequests with no avail, both are
>>> set at 1000 currently.
>>>
>>> Does anyone have any ideas on what might be causing this? or any
>>> other testing utilities I can use to help track down the problem? I
>>> believe it to be something with php or something php is using,
>>> although I have nothing to show me that yet.
>>>
>>> Thanks for any ideas!
>>
>>
>>
>> ---------------------------------------------------------------------
>> The official User-To-User support forum of the Apache HTTP Server
>> Project.
>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server
> Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Weird Apache Memory Leak or something
Posted by Preston Podaril <pp...@ispi.net>.
"truss" is "trace" on Solaris
Preston Podaril
ppod@ispi.net
Network Administrator
ispi of Lincoln
Ivan Barrera A. wrote:
> Sorry, but i just need to ask..
> where do i get "truss" ?
>
> Preston Podaril wrote:
>
>> We have been having some weird issues with Apache for several months
>> now. I have been trying to dig out the reason for it with truss and
>> lsof, but I have been unable to catch anything abnormal.
>>
>> We are running Solaris 8. We upgraded to Apache (mod_ssl) 1.3.33 and
>> PHP 4.3.10 a week and a half ago. Before that we were running Apache
>> (mod_ssl) 1.3.29 and PHP 4.3.4. In both scenarios this problem is
>> occurring.
>>
>> Our problem is that the libhttpd.ep processes appear to be leaking
>> memory for some reason. They also will go to sleep indefinitely (as
>> seen in a truss). The processes that are problematic will grow up to
>> 80M in RES size, checking out a truss on them, doesn't seem to show
>> anything abnormal, nothing that would denote its size getting so
>> large. When we get a lot of these processes between 50M and 80M the
>> machines RAM and swap will get filled up, and then it will become
>> fairly unusable. I have left a truss go overnight on one of these
>> processes that is "dead" and this is all it would show, over and over
>> again in thet output of the truss.
>>
>> sigsuspend(0xFFBDDE68) (sleeping...)
>> signotifywait() (sleeping...)
>> lwp_sema_wait(0xFDF0DE30) (sleeping...)
>> signotifywait() = 16
>> lwp_sigredirect(0, SIGUSR1, 0x00000000) = 0
>> sigsuspend(0xFFBDDE68) (sleeping...)
>> signotifywait() (sleeping...)
>> lwp_sema_wait(0xFDF0DE30) (sleeping...)
>> signotifywait() = 16
>> lwp_sigredirect(0, SIGUSR1, 0x00000000) = 0
>>
>> It has proven really hard to determine what is causing the problem
>> here. MaxClients are set at 70 and 90 for the two different
>> sized machines in the cluster. I have messed with the
>> MaxRequestsPerChild and MaxKeepAliveRequests with no avail, both are
>> set at 1000 currently.
>>
>> Does anyone have any ideas on what might be causing this? or any
>> other testing utilities I can use to help track down the problem? I
>> believe it to be something with php or something php is using,
>> although I have nothing to show me that yet.
>>
>> Thanks for any ideas!
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Weird Apache Memory Leak or something
Posted by "Ivan Barrera A." <Br...@Ivn.cl>.
Sorry, but i just need to ask..
where do i get "truss" ?
Preston Podaril wrote:
> We have been having some weird issues with Apache for several months
> now. I have been trying to dig out the reason for it with truss and
> lsof, but I have been unable to catch anything abnormal.
>
> We are running Solaris 8. We upgraded to Apache (mod_ssl) 1.3.33 and
> PHP 4.3.10 a week and a half ago. Before that we were running Apache
> (mod_ssl) 1.3.29 and PHP 4.3.4. In both scenarios this problem is
> occurring.
>
> Our problem is that the libhttpd.ep processes appear to be leaking
> memory for some reason. They also will go to sleep indefinitely (as
> seen in a truss). The processes that are problematic will grow up to
> 80M in RES size, checking out a truss on them, doesn't seem to show
> anything abnormal, nothing that would denote its size getting so large.
> When we get a lot of these processes between 50M and 80M the machines
> RAM and swap will get filled up, and then it will become fairly
> unusable. I have left a truss go overnight on one of these processes
> that is "dead" and this is all it would show, over and over again in
> thet output of the truss.
>
> sigsuspend(0xFFBDDE68) (sleeping...)
> signotifywait() (sleeping...)
> lwp_sema_wait(0xFDF0DE30) (sleeping...)
> signotifywait() = 16
> lwp_sigredirect(0, SIGUSR1, 0x00000000) = 0
> sigsuspend(0xFFBDDE68) (sleeping...)
> signotifywait() (sleeping...)
> lwp_sema_wait(0xFDF0DE30) (sleeping...)
> signotifywait() = 16
> lwp_sigredirect(0, SIGUSR1, 0x00000000) = 0
>
> It has proven really hard to determine what is causing the problem here.
> MaxClients are set at 70 and 90 for the two different sized
> machines in the cluster. I have messed with the MaxRequestsPerChild and
> MaxKeepAliveRequests with no avail, both are set at 1000 currently.
>
> Does anyone have any ideas on what might be causing this? or any other
> testing utilities I can use to help track down the problem? I believe
> it to be something with php or something php is using, although I have
> nothing to show me that yet.
>
> Thanks for any ideas!
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org