ACTIVEMQ Connection Reset error (Broker and Client)

[yogu13 <yo...@gmail.com>]

Hello,I am facing a weird connection reset errors being logged at debug level
randomly. I am using ACTIVEMQ 5.14.4 with JDK 8-141 deployed on a Windows
2012 R2 Machine. The ActiveMQ Broker has 4 GB heap defined with almost 14000
connections (Yeah we havent used any pooling yet it uses a different
connection for every queue/topic per thread) for queues and topics across
various java processes The error message seen is
07:09:55:740|1209-00246:DEBUG
[org.apache.activemq.broker.TransportConnection.Transport] - Transport
Connection to: tcp://172.31.26.125:50898 failed: java.net.SocketException:
Connection reset Thread {ActiveMQ Transport:
tcp:///172.31.26.125:50898@61616}java.net.SocketException: Connection reset               
at java.net.SocketInputStream.read(SocketInputStream.java:210)               
at java.net.SocketInputStream.read(SocketInputStream.java:141)               
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)               
at
org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634)               
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59)               
at
org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619)               
at java.io.DataInputStream.readInt(DataInputStream.java:387)               
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268)               
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)               
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)               
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)               
at java.lang.Thread.run(Thread.java:748)07:09:55:740|0324-00246:DEBUG
[org.apache.activemq.broker.jmx.ManagementContext] - Unregistering MBean
org.apache.activemq:type=Broker,brokerName=AMQ,connector=clientConnectors,connectorName=tcp,connectionViewType=clientId,connectionName=ID_WIN-31THCOHACHB-49421-1509705222509-4_1
Thread {ActiveMQ Transport:
tcp:///172.31.26.125:50898@61616}07:09:55:740|0312-00246:DEBUG
[org.apache.activemq.broker.jmx.ManagementContext] - Unregistering MBean
org.apache.activemq:type=Broker,brokerName=AMQ,connector=clientConnectors,connectorName=tcp,connectionViewType=remoteAddress,connectionName=tcp_//172.31.26.125_50898
Thread {ActiveMQ Transport:
tcp:///172.31.26.125:50898@61616}07:09:55:740|0155-00675:DEBUG
[org.apache.activemq.broker.TransportConnection] - Stopping connection:
tcp://172.31.26.125:50898 Thread {ActiveMQ BrokerService[AMQ]
Task-103}07:09:55:740|0160-00675:DEBUG
[org.apache.activemq.transport.tcp.TcpTransport] - Stopping transport
tcp:///172.31.26.125:50898@61616 Thread {ActiveMQ BrokerService[AMQ]
Task-103}07:09:55:740|0306-00675:DEBUG
[org.apache.activemq.thread.TaskRunnerFactory] - Initialized
TaskRunnerFactory[ActiveMQ Task] using ExecutorService:
java.util.concurrent.ThreadPoolExecutor@30b3bb2e[Running, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 0] Thread {ActiveMQ
BrokerService[AMQ] Task-103}07:09:55:741|0156-00714:DEBUG
[org.apache.activemq.transport.tcp.TcpTransport] - Closed socket
Socket[addr=/172.31.26.125,port=50898,localport=61616] Thread {ActiveMQ
Task-1}07:09:55:741|0271-00675:DEBUG
[org.apache.activemq.util.ThreadPoolUtils] - Forcing shutdown of
ExecutorService: java.util.concurrent.ThreadPoolExecutor@30b3bb2e[Running,
pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1]
Thread {ActiveMQ BrokerService[AMQ] Task-103}07:09:55:741|0153-00675:DEBUG
[org.apache.activemq.broker.TransportConnection] - Stopped transport:
tcp://172.31.26.125:50898 Thread {ActiveMQ BrokerService[AMQ]
Task-103}07:09:55:741|0168-00675:DEBUG
[org.apache.activemq.broker.TransportConnection] - Cleaning up connection
resources: tcp://172.31.26.125:50898 Thread {ActiveMQ BrokerService[AMQ]
Task-103}07:09:55:741|0173-00675:DEBUG
[org.apache.activemq.broker.TransportConnection] - remove connection id:
ID:WIN-31THCOHACHB-49421-1509705222509-5:1 Thread {ActiveMQ
BrokerService[AMQ] Task-103}Things that i have tried1. Setting TCP level
Keep Alive to true2. Disabling InactivityMonitorThread by setting Max
Inactivity Duration to 03. using connection pool for creating connections
which brought the entire connection count to 300-350 connections in all,
thus reducing load on OSBut sad part is nothing of the above changes have
worked for me to get rid of the problem.my entire infrastructure is hosted
on AWS and CPU usage not going beyond 15-20%. Also I am not using failover
protocol First are these exception traces considered to be harmful ? If yes
what could be causing this connection reset ?Let me know if any more info is
needed.Regards,-Yogesh



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html