You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2015/10/21 10:34:53 UTC
[Bug 58511] New: apache-tomcat - APR does not understand this error
code
https://bz.apache.org/bugzilla/show_bug.cgi?id=58511
Bug ID: 58511
Summary: apache-tomcat - APR does not understand this error
code
Product: Apache httpd-2
Version: 2.2.27
Hardware: PC
Status: NEW
Severity: normal
Priority: P2
Component: mod_proxy_ajp
Assignee: bugs@httpd.apache.org
Reporter: nihil.somani@gmail.com
We are on production with Apache/2.2.27 and Tomcat Version 7.0.55 on
linux(2.6.32-431.37.1.el6.x86_64) with 6 core. what we're seeing is below error
on webserver which has no pattern as to when it occurs.
(104)Connection reset by peer: ajp_ilink_receive() can't receive header
ajp_read_header: ajp_ilink_receive failed
(120006)APR does not understand this error code: proxy: read response failed
from <server ip>:<port> (<server ip>)
it happens quite frequently when we do load test and on normal traffic it
happens but rarely. As load ramps up we see cpu utilization going up(not
hinting that this is reason) but we start seeing this error usually happens
once we cross about 300 user load.
we also have firewall in play between webserver and appserver.
After we see above error webserver jumps from one app server to another app
server. probably thinks original server is marked down. This has become very
comman with load going up hence I took thread dump to see what's happening and
I did not find any blocked threads. I can post whole thread dump if required
here is unique thread stack trace where most of threads are under load.
1) "ajp-bio-8009-exec-3396" daemon prio=10 tid=0x00007fa165b99800 nid=0x538b
runnable [0x00007fa069ad9000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:312)
at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:367)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:118)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
- locked <0x00000007d2998118> (a org.apache.tomcat.util.net.SocketWrapper)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
2) "pool-8-thread-1281" prio=10 tid=0x00007fa0de140800 nid=0x5368 waiting on
condition [0x00007fa0721d9000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000050e5f11f8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
3) "PooledThread[50]" prio=10 tid=0x00007fa0c4125000 nid=0x734 in Object.wait()
[0x00007fa0f064e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006915ccf60> (a
de.hybris.platform.util.threadpool.PoolableThread)
at java.lang.Object.wait(Object.java:503)
at
de.hybris.platform.util.threadpool.PoolableThread.resetAndReturnToPool(PoolableThread.java:246)
- locked <0x00000006915ccf60> (a
de.hybris.platform.util.threadpool.PoolableThread)
at
de.hybris.platform.util.threadpool.PoolableThread.run(PoolableThread.java:219)
4) "PooledThread[49]" prio=10 tid=0x00007fa0b801f800 nid=0x733 in Object.wait()
[0x00007fa066cab000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000653594388> (a
de.hybris.platform.util.threadpool.PoolableThread)
at java.lang.Object.wait(Object.java:503)
at
de.hybris.platform.util.threadpool.PoolableThread.resetAndReturnToPool(PoolableThread.java:246)
- locked <0x0000000653594388> (a
de.hybris.platform.util.threadpool.PoolableThread)
at
de.hybris.platform.util.threadpool.PoolableThread.run(PoolableThread.java:219)
5) "FD_SOCK client connection handler,hybris-broadcast,hybrisnode-2" daemon
prio=10 tid=0x00007fa0bc09d800 nid=0x67b7 runnable [0x00007fa06b9f8000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at
org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1120)
at java.lang.Thread.run(Thread.java:745)
Here's our AJP connector config on tomcat
<Connector protocol="AJP/1.3"
port="8009"
redirectPort="443"
useIPVHosts="true"
URIEncoding="UTF-8"
maxThreads="850"
minSpareThreads="100"
connectionTimeout="30000"
backlog="1000"
keepAliveTimeout="60000"
maxConnections="700"
acceptorThreadCount="4"/>
And below is apache conf
prefork.c
StartServers 20
MaxSpareServers 40
MinSpareServers 40
ServerLimit 700
MaxClients 700
IfModule
and lb.conf
BalancerMember ajp://<host_ip>:8009 route=app5 retry=60
ProxySet stickysession=JSESSIONID
ProxySet lbmethod=byrequests failonstatus=503,504
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 58511] apache-tomcat - APR does not understand this error code
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=58511
Nihil <ni...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
OS| |All
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 58511] apache-tomcat - APR does not understand this error code
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=58511
Nihil <ni...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Severity|normal |major
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 58511] apache-tomcat - APR does not understand this error code
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=58511
--- Comment #1 from Nihil <ni...@gmail.com> ---
anyone has idea as to what's wrong here?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 58511] apache-tomcat - APR does not understand this error code
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=58511
Yann Ylavic <yl...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Resolution|--- |INVALID
Status|NEW |RESOLVED
--- Comment #2 from Yann Ylavic <yl...@gmail.com> ---
This bugzilla is for reporting bugs in the Apache HTTP Server, not for support.
There is no evidence of any particular bug in httpd in your report, please ask
for support on the users mailing-list (users@httpd.apache.org) or/and the
Tomcat one (users@tomcat.apache.org).
If it proves to be a bug in httpd or tomcat, please reopen with the appropriate
Product selected above.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org