You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by "Nigel B. Peck" <ni...@gmail.com> on 2019/08/21 06:21:14 UTC
Re: [users@httpd] Apache process crashes, utilizing high memory
On Mon, 19 Aug 2019 at 11:24 pm, Arnav Garg <ar...@india.nec.com>
wrote:
> Hi All,
>
>
>
> I have an apache web server(v2.4) + ckan (v2.7.2) running in a docker
> container.
> In our scenario, CKAN is being hit with large number of requests at
> certain intervals (like 10 requests/sec). Then after a certain period of
> time, the memory consumption tends to increase towards it's threshold value.
>
> After a certain interval/time, if we access the home page of ckan at
> https://url-address/ckan with large number of request(Using Jmeter), the
> memory usage of the Apache process keeps on increasing and ultimately the
> process crashes and does not work properly.
>
> As a quick remedy for this, restarting the docker container reduces the
> memory usage.
>
> My concern is that, are there any set of conditions/parameters in Apache
> Webserver that needs to be taken care of to prevent this behavior. for eg:
> MaxConnectionPerChild or any other Parameter? Or is there any way to limit
> the memory consumption of Apache web server?
>
> Some of the values set in apache2.conf are:
>
> 1. Timeout 300
>
> 2. KeepAlive On
>
> 3. MaxKeepAliveRequests 100
>
> 4. KeepAliveTimeout 5
>
> Also /apache2/mods-enabled/mpm_event.conf consists of the following
> parameters and values:
>
> 1. StartServers 2
>
> 2. MinSpareThreads 25
>
> 3. MaxSpareThreads 75
>
> 4. ThreadLimit 64
>
> 5. ThreadsPerChild 25
>
> 6. MaxRequestWorkers 150
>
> 7. MaxConnectionsPerChild 0
>
> Thanks & Regards
> Arnav Garg
>
Setting MaxConnectionsPerChild to something not too high (100 as a
temporary fix?) should avoid you needing to restart your Docker container,
sounds like you probably have a memory leak in your application that needs
fixing.
Nigel B. Peck
Web Technologies and Linux Server Admin Mentor
https://codementor.io/nigelbpeck
>
Re: [users@httpd] Apache process crashes, utilizing high memory
Posted by "Nigel B. Peck" <ni...@gmail.com>.
On Wed, 28 Aug 2019 at 12:28 am, Arnav Garg <ar...@india.nec.com>
wrote:
> Are you having any idea, whether the below memory leak issue is resolved
> in any of the latest releases of apache?
>
I’m not suggesting it’s a memory leak in Apache, rather in the
application(s) that you’re running on Apache.
>
RE: [users@httpd] Apache process crashes, utilizing high memory
Posted by Arnav Garg <ar...@india.nec.com>.
Hi Nigel,
Thank you for prompt reply.
The apache version we are using is : Apache/2.4.10 (Debian)
Are you having any idea, whether the below memory leak issue is resolved in any of the latest releases of apache?
Thanks
Arnav
From: Nigel B. Peck [mailto:nigelbpeck@gmail.com]
Sent: Thursday, August 22, 2019 11:10 AM
To: users@httpd.apache.org
Subject: Re: [users@httpd] Apache process crashes, utilizing high memory
On Thu, 22 Aug 2019 at 12:02 am, Arnav Garg <ar...@india.nec.com>> wrote:
Hi Nigel,
Thanks for your prompt reply.
Can you please help me in clarifying some of my doubts?
1. What is the significance of setting MaxConnectionsPerChild to a value between 1-100 ?
2. What are the possible reasons for not setting MaxConnectionsPerChild to a high value, greater than 100 ?
It specifies the number of connections that a child process will serve before being closed and a new one spawned. Having it as 0 means there’s no limit so they just keep running.
I’m suggesting 100 because it seems you have a memory leak, so looking to make sure they are restarted before that takes up too much memory. The cost is the overhead of spawning a new process every X number of connections.
If you set it higher then your memory leak will have more effect.
Best thing is to set it and then monitor and load test the server to get data to guide decisions on what works for your situation. And identify and fix the memory leak :)
Hope that helps.
Nigel
Web Technologies and Linux Admin Mentor
https://codementor.io/nigelbpeck
________________________________
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NECTI or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NECTI or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Re: [users@httpd] Apache process crashes, utilizing high memory
Posted by "Nigel B. Peck" <ni...@gmail.com>.
P.S. to clarify, your application is running within that process, which is
why a memory like makes it grow
On Thu, 29 Aug 2019 at 8:50 am, Nigel B. Peck <ni...@gmail.com> wrote:
>
> It stops when you change to 1 because it kills the child process after
> every single request! Try upgrading Apache to a recent version if you’re
> concerned about that, but this is not likely at all to because of Apache.
> Perhaps others with more experience have thoughts on this situation?
>
> On Thu, 29 Aug 2019 at 3:52 am, Arnav Garg <ar...@india.nec.com>
> wrote:
>
>> Hi Nigel Sir,
>>
>>
>>
>> Thanks for your continuous support and prompt reply. I had one more query
>> as follows:
>>
>>
>>
>> >> I’m not suggesting it’s a memory leak in Apache, rather in the
>> application(s) that you’re running on Apache.
>>
>>
>>
>> After my investigation and your suggestion, I had tuned my apache-server
>> with MaxConnectionPerChild to a value “1” and things started to change. The
>> memory consumption did not rise in comparison with other values. Also, I
>> tried with MaxConnectionPerChild to “100”, but the results were same as
>> MaxConnectionPerChild set to “0” i.e. high memory consumption within few
>> seconds. So I couldn’t understand why memory leak is related to our
>> application(CKAN) & not apache, as memory consumption changes with this
>> parameter(i.e. MaxConnectionPerChild).
>>
>>
>>
>> So, if you can please suggest/clarify as to, why do you think my
>> “application”- CKAN(an open data portal) is having a problem? I think
>> memory leak is from Apache side.
>>
>>
>>
>> Thanks & regards,
>>
>> Arnav
>>
>>
>>
>>
>>
>>
>>
>> *From:* Arnav Garg [mailto:arnav.garg@india.nec.com]
>> *Sent:* Wednesday, August 28, 2019 12:00 PM
>> *To:* users@httpd.apache.org
>> *Subject:* RE: [users@httpd] Apache process crashes, utilizing high
>> memory
>>
>>
>>
>> Hi Nigel,
>>
>>
>>
>> Thanks for your prompt reply, I had one more doubt:
>>
>>
>>
>> While keeping the value of MaxConnectionPerChild to 1, my memory
>> consumption do not increase as rapidly as it was earlier when it was set to
>> 0.
>>
>>
>>
>> Here are the statistics:
>>
>>
>>
>> *My initial memory was : *
>>
>> *USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND*
>>
>> ckan-ap+ 12793 0.2 0.0 304068 6096 ? Sl 06:35 0:00
>> ckan_default -k start
>>
>> ckan-ap+ 12794 0.2 0.0 304068 6104 ? Sl 06:35 0:00
>> ckan_default -k start
>>
>> ckan-ap+ 12795 0.0 0.0 304076 6104 ? Sl 06:35 0:00
>> ckan_default -k start
>>
>> ckan-ap+ 12796 0.0 0.0 304068 6104 ? Sl 06:35 0:00
>> ckan_default -k start
>>
>>
>>
>> After firing 105 requests/per sec for 5 times with a Thread delay of 360
>> milliseconds using Jmeter.
>>
>>
>>
>> *The memory after firing the requests*
>>
>>
>>
>> *Case 1: **When I used to have MaxConnectionPerChild “0”:*
>>
>>
>>
>> The output observed using *“top”* command:
>>
>>
>>
>> *USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND*
>>
>> ckan-ap+ 12793 14.4 1.5 1444920 124880 ? Sl 06:35 0:13
>> ckan_default -k start
>>
>> ckan-ap+ 12794 14.0 1.4 1441080 119300 ? Sl 06:35 0:13
>> ckan_default -k start
>>
>> ckan-ap+ 12795 14.8 1.5 1442120 120724 ? Sl 06:35 0:14
>> ckan_default -k start
>>
>> ckan-ap+ 12796 14.5 1.5 1442616 120904 ? Sl 06:35 0:13
>> ckan_default -k start
>>
>>
>>
>> *Case 2: **When MaxConnection PerChild was set to “1”.*
>>
>>
>>
>> *USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND*
>>
>> ckan-ap+ 13448 13.6 1.3 *845368* 108904 ? Sl 06:39 0:11
>> ckan_default -k start
>>
>> ckan-ap+ 13449 11.6 1.3 *845376* 104560 ? Sl 06:39 0:09
>> ckan_default -k start
>>
>> ckan-ap+ 13450 11.7 1.2 *844608* 102696 ? Sl 06:39 0:09
>> ckan_default -k start
>>
>> ckan-ap+ 13451 11.8 1.2 *844344* 102392 ? Sl 06:39 0:09
>> ckan_default -k start
>>
>>
>>
>> The value highlighted in yellow remains almost the same or nearby when
>> the performance testing was done for 2 hours.
>>
>>
>>
>> *Case 3: **When MaxConnectionPerChild was set to “100”.*
>>
>> Output: same as Case 1.
>>
>>
>>
>> *Doubt:*
>>
>> if you can please let us know on the performance or any side effects if I
>> keep the value of MaxConnectionPerChild as 0 , 1 and 100?
>>
>>
>>
>> Thanks & Regards
>>
>> Arnav
>>
>>
>>
>> >> I’m suggesting 100 because it seems you have a memory leak, so looking
>> to make sure they are restarted before that takes up too much memory. The
>> cost is the overhead of spawning a new process every X number of
>> connections.
>>
>>
>>
>> >> If you set it higher then your memory leak will have more effect.
>>
>>
>>
>> Will there be any performance impact
>>
>>
>>
>>
>>
>> *From:* Nigel B. Peck [mailto:nigelbpeck@gmail.com <ni...@gmail.com>]
>>
>> *Sent:* Thursday, August 22, 2019 11:10 AM
>> *To:* users@httpd.apache.org
>> *Subject:* Re: [users@httpd] Apache process crashes, utilizing high
>> memory
>>
>>
>>
>> On Thu, 22 Aug 2019 at 12:02 am, Arnav Garg <ar...@india.nec.com>
>> wrote:
>>
>> Hi Nigel,
>>
>>
>>
>> Thanks for your prompt reply.
>>
>> Can you please help me in clarifying some of my doubts?
>>
>> 1. What is the significance of setting MaxConnectionsPerChild to a
>> value between 1-100 ?
>>
>> 2. What are the possible reasons for not setting
>> MaxConnectionsPerChild to a high value, greater than 100 ?
>>
>> It specifies the number of connections that a child process will serve
>> before being closed and a new one spawned. Having it as 0 means there’s no
>> limit so they just keep running.
>>
>>
>>
>> I’m suggesting 100 because it seems you have a memory leak, so looking to
>> make sure they are restarted before that takes up too much memory. The cost
>> is the overhead of spawning a new process every X number of connections.
>>
>>
>>
>> If you set it higher then your memory leak will have more effect.
>>
>>
>>
>> Best thing is to set it and then monitor and load test the server to get
>> data to guide decisions on what works for your situation. And identify and
>> fix the memory leak :)
>>
>>
>>
>> Hope that helps.
>>
>>
>>
>> Nigel
>>
>>
>>
>> Web Technologies and Linux Admin Mentor
>>
>> https://codementor.io/nigelbpeck
>>
>>
>>
>>
>>
>>
>> ------------------------------
>>
>> The contents of this e-mail and any attachment(s) are confidential and
>> intended for the named recipient(s) only. It shall not attach any liability
>> on the originator or NECTI or its affiliates. Any views or opinions
>> presented in this email are solely those of the author and may not
>> necessarily reflect the opinions of NECTI or its affiliates. Any form of
>> reproduction, dissemination, copying, disclosure, modification,
>> distribution and / or publication of this message without the prior written
>> consent of the author of this e-mail is strictly prohibited. If you have
>> received this email in error please delete it and notify the sender
>> immediately.
>> ------------------------------
>> The contents of this e-mail and any attachment(s) are confidential and
>> intended for the named recipient(s) only. It shall not attach any liability
>> on the originator or NECTI or its affiliates. Any views or opinions
>> presented in this email are solely those of the author and may not
>> necessarily reflect the opinions of NECTI or its affiliates. Any form of
>> reproduction, dissemination, copying, disclosure, modification,
>> distribution and / or publication of this message without the prior written
>> consent of the author of this e-mail is strictly prohibited. If you have
>> received this email in error please delete it and notify the sender
>> immediately.
>>
>
Re: [users@httpd] Apache process crashes, utilizing high memory
Posted by "Nigel B. Peck" <ni...@gmail.com>.
It stops when you change to 1 because it kills the child process after
every single request! Try upgrading Apache to a recent version if you’re
concerned about that, but this is not likely at all to because of Apache.
Perhaps others with more experience have thoughts on this situation?
On Thu, 29 Aug 2019 at 3:52 am, Arnav Garg <ar...@india.nec.com> wrote:
> Hi Nigel Sir,
>
>
>
> Thanks for your continuous support and prompt reply. I had one more query
> as follows:
>
>
>
> >> I’m not suggesting it’s a memory leak in Apache, rather in the
> application(s) that you’re running on Apache.
>
>
>
> After my investigation and your suggestion, I had tuned my apache-server
> with MaxConnectionPerChild to a value “1” and things started to change. The
> memory consumption did not rise in comparison with other values. Also, I
> tried with MaxConnectionPerChild to “100”, but the results were same as
> MaxConnectionPerChild set to “0” i.e. high memory consumption within few
> seconds. So I couldn’t understand why memory leak is related to our
> application(CKAN) & not apache, as memory consumption changes with this
> parameter(i.e. MaxConnectionPerChild).
>
>
>
> So, if you can please suggest/clarify as to, why do you think my
> “application”- CKAN(an open data portal) is having a problem? I think
> memory leak is from Apache side.
>
>
>
> Thanks & regards,
>
> Arnav
>
>
>
>
>
>
>
> *From:* Arnav Garg [mailto:arnav.garg@india.nec.com]
> *Sent:* Wednesday, August 28, 2019 12:00 PM
> *To:* users@httpd.apache.org
> *Subject:* RE: [users@httpd] Apache process crashes, utilizing high memory
>
>
>
> Hi Nigel,
>
>
>
> Thanks for your prompt reply, I had one more doubt:
>
>
>
> While keeping the value of MaxConnectionPerChild to 1, my memory
> consumption do not increase as rapidly as it was earlier when it was set to
> 0.
>
>
>
> Here are the statistics:
>
>
>
> *My initial memory was : *
>
> *USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND*
>
> ckan-ap+ 12793 0.2 0.0 304068 6096 ? Sl 06:35 0:00
> ckan_default -k start
>
> ckan-ap+ 12794 0.2 0.0 304068 6104 ? Sl 06:35 0:00
> ckan_default -k start
>
> ckan-ap+ 12795 0.0 0.0 304076 6104 ? Sl 06:35 0:00
> ckan_default -k start
>
> ckan-ap+ 12796 0.0 0.0 304068 6104 ? Sl 06:35 0:00
> ckan_default -k start
>
>
>
> After firing 105 requests/per sec for 5 times with a Thread delay of 360
> milliseconds using Jmeter.
>
>
>
> *The memory after firing the requests*
>
>
>
> *Case 1: **When I used to have MaxConnectionPerChild “0”:*
>
>
>
> The output observed using *“top”* command:
>
>
>
> *USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND*
>
> ckan-ap+ 12793 14.4 1.5 1444920 124880 ? Sl 06:35 0:13
> ckan_default -k start
>
> ckan-ap+ 12794 14.0 1.4 1441080 119300 ? Sl 06:35 0:13
> ckan_default -k start
>
> ckan-ap+ 12795 14.8 1.5 1442120 120724 ? Sl 06:35 0:14
> ckan_default -k start
>
> ckan-ap+ 12796 14.5 1.5 1442616 120904 ? Sl 06:35 0:13
> ckan_default -k start
>
>
>
> *Case 2: **When MaxConnection PerChild was set to “1”.*
>
>
>
> *USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND*
>
> ckan-ap+ 13448 13.6 1.3 *845368* 108904 ? Sl 06:39 0:11
> ckan_default -k start
>
> ckan-ap+ 13449 11.6 1.3 *845376* 104560 ? Sl 06:39 0:09
> ckan_default -k start
>
> ckan-ap+ 13450 11.7 1.2 *844608* 102696 ? Sl 06:39 0:09
> ckan_default -k start
>
> ckan-ap+ 13451 11.8 1.2 *844344* 102392 ? Sl 06:39 0:09
> ckan_default -k start
>
>
>
> The value highlighted in yellow remains almost the same or nearby when the
> performance testing was done for 2 hours.
>
>
>
> *Case 3: **When MaxConnectionPerChild was set to “100”.*
>
> Output: same as Case 1.
>
>
>
> *Doubt:*
>
> if you can please let us know on the performance or any side effects if I
> keep the value of MaxConnectionPerChild as 0 , 1 and 100?
>
>
>
> Thanks & Regards
>
> Arnav
>
>
>
> >> I’m suggesting 100 because it seems you have a memory leak, so looking
> to make sure they are restarted before that takes up too much memory. The
> cost is the overhead of spawning a new process every X number of
> connections.
>
>
>
> >> If you set it higher then your memory leak will have more effect.
>
>
>
> Will there be any performance impact
>
>
>
>
>
> *From:* Nigel B. Peck [mailto:nigelbpeck@gmail.com <ni...@gmail.com>]
>
> *Sent:* Thursday, August 22, 2019 11:10 AM
> *To:* users@httpd.apache.org
> *Subject:* Re: [users@httpd] Apache process crashes, utilizing high memory
>
>
>
> On Thu, 22 Aug 2019 at 12:02 am, Arnav Garg <ar...@india.nec.com>
> wrote:
>
> Hi Nigel,
>
>
>
> Thanks for your prompt reply.
>
> Can you please help me in clarifying some of my doubts?
>
> 1. What is the significance of setting MaxConnectionsPerChild to a
> value between 1-100 ?
>
> 2. What are the possible reasons for not setting
> MaxConnectionsPerChild to a high value, greater than 100 ?
>
> It specifies the number of connections that a child process will serve
> before being closed and a new one spawned. Having it as 0 means there’s no
> limit so they just keep running.
>
>
>
> I’m suggesting 100 because it seems you have a memory leak, so looking to
> make sure they are restarted before that takes up too much memory. The cost
> is the overhead of spawning a new process every X number of connections.
>
>
>
> If you set it higher then your memory leak will have more effect.
>
>
>
> Best thing is to set it and then monitor and load test the server to get
> data to guide decisions on what works for your situation. And identify and
> fix the memory leak :)
>
>
>
> Hope that helps.
>
>
>
> Nigel
>
>
>
> Web Technologies and Linux Admin Mentor
>
> https://codementor.io/nigelbpeck
>
>
>
>
>
>
> ------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only. It shall not attach any liability
> on the originator or NECTI or its affiliates. Any views or opinions
> presented in this email are solely those of the author and may not
> necessarily reflect the opinions of NECTI or its affiliates. Any form of
> reproduction, dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior written
> consent of the author of this e-mail is strictly prohibited. If you have
> received this email in error please delete it and notify the sender
> immediately.
> ------------------------------
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only. It shall not attach any liability
> on the originator or NECTI or its affiliates. Any views or opinions
> presented in this email are solely those of the author and may not
> necessarily reflect the opinions of NECTI or its affiliates. Any form of
> reproduction, dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior written
> consent of the author of this e-mail is strictly prohibited. If you have
> received this email in error please delete it and notify the sender
> immediately.
>
RE: [users@httpd] Apache process crashes, utilizing high memory
Posted by Arnav Garg <ar...@india.nec.com>.
Hi Nigel Sir,
Thanks for your continuous support and prompt reply. I had one more query as follows:
>> I'm not suggesting it's a memory leak in Apache, rather in the application(s) that you're running on Apache.
After my investigation and your suggestion, I had tuned my apache-server with MaxConnectionPerChild to a value "1" and things started to change. The memory consumption did not rise in comparison with other values. Also, I tried with MaxConnectionPerChild to "100", but the results were same as MaxConnectionPerChild set to "0" i.e. high memory consumption within few seconds. So I couldn't understand why memory leak is related to our application(CKAN) & not apache, as memory consumption changes with this parameter(i.e. MaxConnectionPerChild).
So, if you can please suggest/clarify as to, why do you think my "application"- CKAN(an open data portal) is having a problem? I think memory leak is from Apache side.
Thanks & regards,
Arnav
From: Arnav Garg [mailto:arnav.garg@india.nec.com]
Sent: Wednesday, August 28, 2019 12:00 PM
To: users@httpd.apache.org
Subject: RE: [users@httpd] Apache process crashes, utilizing high memory
Hi Nigel,
Thanks for your prompt reply, I had one more doubt:
While keeping the value of MaxConnectionPerChild to 1, my memory consumption do not increase as rapidly as it was earlier when it was set to 0.
Here are the statistics:
My initial memory was :
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
ckan-ap+ 12793 0.2 0.0 304068 6096 ? Sl 06:35 0:00 ckan_default -k start
ckan-ap+ 12794 0.2 0.0 304068 6104 ? Sl 06:35 0:00 ckan_default -k start
ckan-ap+ 12795 0.0 0.0 304076 6104 ? Sl 06:35 0:00 ckan_default -k start
ckan-ap+ 12796 0.0 0.0 304068 6104 ? Sl 06:35 0:00 ckan_default -k start
After firing 105 requests/per sec for 5 times with a Thread delay of 360 milliseconds using Jmeter.
The memory after firing the requests
Case 1: When I used to have MaxConnectionPerChild "0":
The output observed using "top" command:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
ckan-ap+ 12793 14.4 1.5 1444920 124880 ? Sl 06:35 0:13 ckan_default -k start
ckan-ap+ 12794 14.0 1.4 1441080 119300 ? Sl 06:35 0:13 ckan_default -k start
ckan-ap+ 12795 14.8 1.5 1442120 120724 ? Sl 06:35 0:14 ckan_default -k start
ckan-ap+ 12796 14.5 1.5 1442616 120904 ? Sl 06:35 0:13 ckan_default -k start
Case 2: When MaxConnection PerChild was set to "1".
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
ckan-ap+ 13448 13.6 1.3 845368 108904 ? Sl 06:39 0:11 ckan_default -k start
ckan-ap+ 13449 11.6 1.3 845376 104560 ? Sl 06:39 0:09 ckan_default -k start
ckan-ap+ 13450 11.7 1.2 844608 102696 ? Sl 06:39 0:09 ckan_default -k start
ckan-ap+ 13451 11.8 1.2 844344 102392 ? Sl 06:39 0:09 ckan_default -k start
The value highlighted in yellow remains almost the same or nearby when the performance testing was done for 2 hours.
Case 3: When MaxConnectionPerChild was set to "100".
Output: same as Case 1.
Doubt:
if you can please let us know on the performance or any side effects if I keep the value of MaxConnectionPerChild as 0 , 1 and 100?
Thanks & Regards
Arnav
>> I'm suggesting 100 because it seems you have a memory leak, so looking to make sure they are restarted before that takes up too much memory. The cost is the overhead of spawning a new process every X number of connections.
>> If you set it higher then your memory leak will have more effect.
Will there be any performance impact
From: Nigel B. Peck [mailto:nigelbpeck@gmail.com]
Sent: Thursday, August 22, 2019 11:10 AM
To: users@httpd.apache.org<ma...@httpd.apache.org>
Subject: Re: [users@httpd] Apache process crashes, utilizing high memory
On Thu, 22 Aug 2019 at 12:02 am, Arnav Garg <ar...@india.nec.com>> wrote:
Hi Nigel,
Thanks for your prompt reply.
Can you please help me in clarifying some of my doubts?
1. What is the significance of setting MaxConnectionsPerChild to a value between 1-100 ?
2. What are the possible reasons for not setting MaxConnectionsPerChild to a high value, greater than 100 ?
It specifies the number of connections that a child process will serve before being closed and a new one spawned. Having it as 0 means there's no limit so they just keep running.
I'm suggesting 100 because it seems you have a memory leak, so looking to make sure they are restarted before that takes up too much memory. The cost is the overhead of spawning a new process every X number of connections.
If you set it higher then your memory leak will have more effect.
Best thing is to set it and then monitor and load test the server to get data to guide decisions on what works for your situation. And identify and fix the memory leak :)
Hope that helps.
Nigel
Web Technologies and Linux Admin Mentor
https://codementor.io/nigelbpeck
________________________________
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NECTI or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NECTI or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
________________________________
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NECTI or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NECTI or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
RE: [users@httpd] Apache process crashes, utilizing high memory
Posted by Arnav Garg <ar...@india.nec.com>.
Hi Nigel,
Thanks for your prompt reply, I had one more doubt:
While keeping the value of MaxConnectionPerChild to 1, my memory consumption do not increase as rapidly as it was earlier when it was set to 0.
Here are the statistics:
My initial memory was :
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
ckan-ap+ 12793 0.2 0.0 304068 6096 ? Sl 06:35 0:00 ckan_default -k start
ckan-ap+ 12794 0.2 0.0 304068 6104 ? Sl 06:35 0:00 ckan_default -k start
ckan-ap+ 12795 0.0 0.0 304076 6104 ? Sl 06:35 0:00 ckan_default -k start
ckan-ap+ 12796 0.0 0.0 304068 6104 ? Sl 06:35 0:00 ckan_default -k start
After firing 105 requests/per sec for 5 times with a Thread delay of 360 milliseconds using Jmeter.
The memory after firing the requests
Case 1: When I used to have MaxConnectionPerChild “0”:
The output observed using “top” command:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
ckan-ap+ 12793 14.4 1.5 1444920 124880 ? Sl 06:35 0:13 ckan_default -k start
ckan-ap+ 12794 14.0 1.4 1441080 119300 ? Sl 06:35 0:13 ckan_default -k start
ckan-ap+ 12795 14.8 1.5 1442120 120724 ? Sl 06:35 0:14 ckan_default -k start
ckan-ap+ 12796 14.5 1.5 1442616 120904 ? Sl 06:35 0:13 ckan_default -k start
Case 2: When MaxConnection PerChild was set to “1”.
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
ckan-ap+ 13448 13.6 1.3 845368 108904 ? Sl 06:39 0:11 ckan_default -k start
ckan-ap+ 13449 11.6 1.3 845376 104560 ? Sl 06:39 0:09 ckan_default -k start
ckan-ap+ 13450 11.7 1.2 844608 102696 ? Sl 06:39 0:09 ckan_default -k start
ckan-ap+ 13451 11.8 1.2 844344 102392 ? Sl 06:39 0:09 ckan_default -k start
The value highlighted in yellow remains almost the same or nearby when the performance testing was done for 2 hours.
Case 3: When MaxConnectionPerChild was set to “100”.
Output: same as Case 1.
Doubt:
if you can please let us know on the performance or any side effects if I keep the value of MaxConnectionPerChild as 0 , 1 and 100?
Thanks & Regards
Arnav
>> I’m suggesting 100 because it seems you have a memory leak, so looking to make sure they are restarted before that takes up too much memory. The cost is the overhead of spawning a new process every X number of connections.
>> If you set it higher then your memory leak will have more effect.
Will there be any performance impact
From: Nigel B. Peck [mailto:nigelbpeck@gmail.com]
Sent: Thursday, August 22, 2019 11:10 AM
To: users@httpd.apache.org
Subject: Re: [users@httpd] Apache process crashes, utilizing high memory
On Thu, 22 Aug 2019 at 12:02 am, Arnav Garg <ar...@india.nec.com>> wrote:
Hi Nigel,
Thanks for your prompt reply.
Can you please help me in clarifying some of my doubts?
1. What is the significance of setting MaxConnectionsPerChild to a value between 1-100 ?
2. What are the possible reasons for not setting MaxConnectionsPerChild to a high value, greater than 100 ?
It specifies the number of connections that a child process will serve before being closed and a new one spawned. Having it as 0 means there’s no limit so they just keep running.
I’m suggesting 100 because it seems you have a memory leak, so looking to make sure they are restarted before that takes up too much memory. The cost is the overhead of spawning a new process every X number of connections.
If you set it higher then your memory leak will have more effect.
Best thing is to set it and then monitor and load test the server to get data to guide decisions on what works for your situation. And identify and fix the memory leak :)
Hope that helps.
Nigel
Web Technologies and Linux Admin Mentor
https://codementor.io/nigelbpeck
________________________________
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NECTI or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NECTI or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Re: [users@httpd] Apache process crashes, utilizing high memory
Posted by "Nigel B. Peck" <ni...@gmail.com>.
On Thu, 22 Aug 2019 at 12:02 am, Arnav Garg <ar...@india.nec.com>
wrote:
> Hi Nigel,
>
>
>
> Thanks for your prompt reply.
>
> Can you please help me in clarifying some of my doubts?
>
> 1. What is the significance of setting MaxConnectionsPerChild to a
> value between 1-100 ?
>
> 2. What are the possible reasons for not setting
> MaxConnectionsPerChild to a high value, greater than 100 ?
>
It specifies the number of connections that a child process will serve
before being closed and a new one spawned. Having it as 0 means there’s no
limit so they just keep running.
I’m suggesting 100 because it seems you have a memory leak, so looking to
make sure they are restarted before that takes up too much memory. The cost
is the overhead of spawning a new process every X number of connections.
If you set it higher then your memory leak will have more effect.
Best thing is to set it and then monitor and load test the server to get
data to guide decisions on what works for your situation. And identify and
fix the memory leak :)
Hope that helps.
Nigel
Web Technologies and Linux Admin Mentor
https://codementor.io/nigelbpeck
RE: [users@httpd] Apache process crashes, utilizing high memory
Posted by Arnav Garg <ar...@india.nec.com>.
Hi Nigel,
Thanks for your prompt reply.
Can you please help me in clarifying some of my doubts?
1. What is the significance of setting MaxConnectionsPerChild to a value between 1-100 ?
2. What are the possible reasons for not setting MaxConnectionsPerChild to a high value, greater than 100 ?
Thanks & Regards
Arnav Garg
From: Nigel B. Peck [mailto:nigelbpeck@gmail.com]
Sent: Wednesday, August 21, 2019 11:51 AM
To: users@httpd.apache.org
Subject: Re: [users@httpd] Apache process crashes, utilizing high memory
On Mon, 19 Aug 2019 at 11:24 pm, Arnav Garg <ar...@india.nec.com>> wrote:
Hi All,
I have an apache web server(v2.4) + ckan (v2.7.2) running in a docker container.
In our scenario, CKAN is being hit with large number of requests at certain intervals (like 10 requests/sec). Then after a certain period of time, the memory consumption tends to increase towards it's threshold value.
After a certain interval/time, if we access the home page of ckan at https://url-address/ckan with large number of request(Using Jmeter), the memory usage of the Apache process keeps on increasing and ultimately the process crashes and does not work properly.
As a quick remedy for this, restarting the docker container reduces the memory usage.
My concern is that, are there any set of conditions/parameters in Apache Webserver that needs to be taken care of to prevent this behavior. for eg: MaxConnectionPerChild or any other Parameter? Or is there any way to limit the memory consumption of Apache web server?
Some of the values set in apache2.conf are:
1. Timeout 300
2. KeepAlive On
3. MaxKeepAliveRequests 100
4. KeepAliveTimeout 5
Also /apache2/mods-enabled/mpm_event.conf consists of the following parameters and values:
1. StartServers 2
2. MinSpareThreads 25
3. MaxSpareThreads 75
4. ThreadLimit 64
5. ThreadsPerChild 25
6. MaxRequestWorkers 150
7. MaxConnectionsPerChild 0
Thanks & Regards
Arnav Garg
Setting MaxConnectionsPerChild to something not too high (100 as a temporary fix?) should avoid you needing to restart your Docker container, sounds like you probably have a memory leak in your application that needs fixing.
Nigel B. Peck
Web Technologies and Linux Server Admin Mentor
https://codementor.io/nigelbpeck
________________________________
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. It shall not attach any liability on the originator or NECTI or its affiliates. Any views or opinions presented in this email are solely those of the author and may not necessarily reflect the opinions of NECTI or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.