You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2020/08/31 16:04:13 UTC
[Bug 64700] New: Apache crash in 2.4.43
https://bz.apache.org/bugzilla/show_bug.cgi?id=64700
Bug ID: 64700
Summary: Apache crash in 2.4.43
Product: Apache httpd-2
Version: 2.4.43
Hardware: PC
Status: NEW
Severity: normal
Priority: P2
Component: All
Assignee: bugs@httpd.apache.org
Reporter: janani.rajan@wellsfargo.com
Target Milestone: ---
Apache crash was first noticed in 2.4.41 .Error stack looked similar to the
bug which was fixed in 2.4.43 but still the issue was seen in 2.4.43.We made
system level memory setting changes even after that we could see that the issue
was happening .
# vi rc.local
# (Add the lines below to the end. Replace eth0, eth1 with the actual names)
# sysctl -w net.core.rmem_max=16777216
# sysctl -w net.core.wmem_max=16777216
# sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"
# sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216"
# sysctl -w net.ipv4.tcp_fin_timeout=10
# ifconfig eth0 txqueuelen 10000
# ifconfig eth2 txqueuelen 10000
# reboot
14) Configure new ulimit settings
# pbrun su-root
# cd /etc/security
# vi limits.conf
# (Add these lines to the end)
# root hard nofile 16384
# root soft nofile 16384
# apache hard nofile 16384
# apache soft nofile 16384
# :wq
# reboot
We suspected that the load on the server is more which may be due increase in
the number of application users . We increased the apache mpm values and since
we are using worker settings which deals with increase load on the server with
respect to the increased connection .
./apachectl -V |grep MPM
Server MPM: worker
Server limit set was 300 .We are increasing the value to 600.
Start servers from 200 to 500 and Max client to 15000
Minspare and Maxspare thread values are changed as per the
recommendation(75 and 250) from the apache documentation.
ServerLimit 600
StartServers 500
MaxClients 15000
MinSpareThreads 75 (recommended value)
MaxSpareThreads 250 (recommended value)
ThreadsPerChild 25
MaxRequestsPerChild 0
But the issue was still seen with below exception post implementing the change
I was researching on this issue again and fortunately got the source code for
apache .
Please find the source code link for apache.
people.apache.org/~igalic/checks/httpd/2012-09-14-1/report-69Y1He.html
After the below exception Server gets hung and cannot serve any request.
[Fri Aug 07 16:12:12.684173 2020] [mpm_event:debug] [pid 19580:tid
139740527363840] event.c(1810): Too many open connections (25), not accepting
new conns in this process
This exception is seen from the time we enabled debugs .
[Fri Aug 07 16:12:12.889710 2020] [mpm_event:debug] [pid 19609:tid
139740820498176] event.c(2314): AH02471: start_threads: Using epoll (wakeable)
[Fri Aug 07 16:12:12.889934 2020] [mpm_event:debug] [pid 19610:tid
139740820498176] event.c(2314): AH02471: start_threads: Using epoll (wakeable)
Lines in the source code referring to the exception
1805 else if (connections_above_limit()) {
1806 disable_listensocks();
1807 ap_log_error(APLOG_MARK, APLOG_DEBUG, 0,
ap_server_conf,
1808 "Too many open connections (%u), "
1809 "not accepting new conns in this
process",
1810 apr_atomic_read32(&connection_count));
1811 ap_log_error(APLOG_MARK, APLOG_TRACE1, 0,
ap_server_conf,
1812 "Idle workers: %u",
This %u refers to the threads per child value in mpm which we is the default
recommended value but looks like that request on server is high where there is
lack of this parameter . Increased the thread per child to 50 and made other
parameter changes to match the value .but still the issue was seen .
From
ServerLimit 600
StartServers 500
MaxClients 15000
MinSpareThreads 75
MaxSpareThreads 250
ThreadsPerChild 25
New value :
ServerLimit 300
StartServers 200
MaxClients 15000
MinSpareThreads 75
MaxSpareThreads 250
ThreadsPerChild 50
we were getting other exception post the changes as below
AH00486: server seems busy, (you may need to increase StartServers,
ThreadsPerChild or Min/MaxSpareThreads), spawning 8 children, there are around
54 idle threads, 6 active children, and 6 children that are shutting down.
Made the threads per child to 40 and min and max spare s 25 and 75 respectively
with the semaphore changes at the system level but still the issue is seen
Changes made on the server level
# 17) Increase semaphore limits
# pbrun su-root
# vi /etc/sysctl.conf
# Add this line:
# # Add additional semaphores for Channel Secure and mod_rewrite
# kernel.sem = 4096 512000 1600 9000
# :wq
# sysctl -p /etc/sysctl.conf
What I notice is webagent fails to initialise all the time and the server
loses the connection ( no connections hit from F5 or the server does not take u
the connection which like very low count of user when netstat performed say 25
and then the apacheURL goes down for the individual apache server though apache
process is up .
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64700] Apache crash in 2.4.43
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64700
--- Comment #3 from janani <ja...@wellsfargo.com> ---
can you assist us on this issue
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64700] Apache crash in 2.4.43
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64700
--- Comment #2 from janani <ja...@wellsfargo.com> ---
we are usin event MPM with AsyncRequestWorkerFactor not been set so it behaves
like worker mpm .
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64700] Apache crash in 2.4.43
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64700
janani <ja...@wellsfargo.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
OS| |All
Severity|normal |critical
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 64700] Apache crash in 2.4.43
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64700
--- Comment #1 from Christophe JAILLET <ch...@wanadoo.fr> ---
Hi,
just to make things clear, you state that you are using the worker mpm, but you
provide logs with 'mpm_event'.
Could you clarify?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org