You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Hemanathan <he...@gmail.com> on 2018/09/21 12:22:49 UTC
Stale connection hanging at broker during consumer crash
Stale connection hanging at broker during consumer crash :
Step1 :
When a consumer is getting attached with ActiveMQ broker, the consumer count
is getting increased with one , which is verified through “dstat” command .
Also the “netstat” from client machine shows one connection establishment
with broker.
Netstat:
tcp 0 0 x.x.x.x:61617 x.x.x.x:44332
ESTABLISHED 18625/java
dstat value is:
Name Queue
Size Producer # Consumer # Enqueue # Dequeue # Forward # Memory %
consumer.xyz.virtualTopic.abc.productUpdateNotificationEvent.1_0 8
0 1 10 2 0 0
Step2 :
Kill the client connection and no such connection in netstat from client
machine. But still the connection hanging with broker and dstat showing the
same result as before.
Netstat:
-NA-
dstat value is:
Name Queue
Size Producer # Consumer # Enqueue # Dequeue # Forward # Memory %
consumer.xyz.virtualTopic.abc.productUpdateNotificationEvent.1_0 8
0 1 10 2 0 0
Step 3:
Few of the message (multiples of 1000) not getting de-queue even after the
new client getting connected. So the number of messages and number of
consumers is getting increased on every attempt of client termination. When
we try to change the "prefetchPolicy.queuePrefetch" property on consumer
side, its reflecting so the number of messages are hanging (as configured in
property . Eg. If we set value as "prefetchPolicy.queuePrefetch=10", then 10
messages are hanging on the queue)
Step 4 :
After re-starting the broker , all the pending messages are getting
de-queued, Where one client is up and running .
Please suggest how to solve this problem.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Stale connection hanging at broker during consumer crash
Posted by Hemanathan <he...@gmail.com>.
I am getting this exception on broker side when consumer crashes.
Is there anyway to close the connection completely on broker side.?
2018-10-01T09:50:27.589+0200 [ActiveMQ Transport: ssl:///x.x.x.x:42188]
DEBUG o.a.a.t.logwriters.DefaultLogWriter:65 logReceivedException RECEIVED
Exception: java.io.EOFException
java.io.EOFException: null
at java.io.DataInputStream.readInt(DataInputStream.java:392)
~[na:1.8.0_152]
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268)
~[activemq-client-5.14.0.jar:5.14.0]
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)
~[activemq-client-5.14.0.jar:5.14.0]
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)
~[activemq-client-5.14.0.jar:5.14.0]
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
~[activemq-client-5.14.0.jar:5.14.0]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]
2018-10-01T09:50:27.592+0200 [ActiveMQ Transport: ssl:///x.x.x.x:42188]
DEBUG o.a.a.broker.TransportConnection:238 serviceTransportException
Transport Connection to: tcp://x.x.x.x:42188 failed: java.io.EOFException
java.io.EOFException: null
at java.io.DataInputStream.readInt(DataInputStream.java:392)
~[na:1.8.0_152]
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268)
~[activemq-client-5.14.0.jar:5.14.0]
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)
~[activemq-client-5.14.0.jar:5.14.0]
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)
~[activemq-client-5.14.0.jar:5.14.0]
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
~[activemq-client-5.14.0.jar:5.14.0]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Stale connection hanging at broker during consumer crash
Posted by Hemanathan <he...@gmail.com>.
I am using RHEL 6.5.
I am not using any load balancers.
I tried with stopping iptables service in both broker and consumer, still
the result is same.
I use "dstat" to know the consumer count, "netstat" to know the live
connections in low level as I mentioned in the thread earlier.
It is a direct connection only.
Firewall information:
# iptables -L -n
Chain INPUT (policy DROP)
target prot opt source destination
RH-Firewall-1-INPUT all -- 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy ACCEPT)
target prot opt source destination
RH-Firewall-1-INPUT all -- 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain RH-Firewall-1-INPUT (2 references)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport
dports 1090
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport
dports 8000
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state
RELATED,ESTABLISHED
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
ACCEPT 2 -- 0.0.0.0/0 0.0.0.0/0
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport
dports 22,705,2992,6666,7666,8666,8800,8801,9010,9011
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport
dports 13000:13002,80,115,123,443,1090,1099,1389,1636,1638
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport
dports 2181,2281,5672,8161,8543,13000,13001,13002,9443,19443
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport
dports 61616,61617,61619,8080
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 multiport
dports 161,7500,8162,22,80,123,443
REJECT udp -- 0.0.0.0/0 0.0.0.0/0 multiport
dports 137,138,139,445 reject-with icmp-host-prohibited
REJECT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport
dports 137,138,139,445 reject-with icmp-host-prohibited
LOG all -- 0.0.0.0/0 0.0.0.0/0 limit: avg
25/hour burst 5 LOG flags 0 level 6 prefix `REJECT '
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with
icmp-host-prohibited
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Stale connection hanging at broker during consumer crash
Posted by Tim Bain <tb...@alumni.duke.edu>.
Days?? That doesn't sound like the TCP timeout behavior after all; that's
tunable at the O/S level and via the parameter I mentioned, but it's
usually measured in seconds to minutes.
What O/S are you using? And can you show the commands you ran to produce
the various output?
Oh, and is there a load balancer or some other form of proxy between your
client and the broker, or does it connect directly? Might some intermediary
be keeping a connection alive even though the client has closed theirs?
Tim
On Tue, Sep 25, 2018, 7:21 AM Hemanathan <he...@gmail.com>
wrote:
> staying active for days. I haven't observed the removal at least once.
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: Stale connection hanging at broker during consumer crash
Posted by Hemanathan <he...@gmail.com>.
staying active for days. I haven't observed the removal at least once.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Stale connection hanging at broker during consumer crash
Posted by Tim Bain <tb...@alumni.duke.edu>.
How long do they stay active? Are we talking seconds, minutes, hours,
days...?
Tim
On Mon, Sep 24, 2018, 12:22 PM Hemanathan <he...@gmail.com>
wrote:
> I tried with "soTimeout" parameter. Still connections are active in
> broker.
>
> My transport connector configuration is
>
>
> ssl://x.x.x.x:61617?transport.needClientAuth=true&transport.soTimeout=20000&transport.trace=true&maximumConnections=1000&wireformat.maxFrameSize=104857600
>
> please let me know what is wrongly configured here.
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: Stale connection hanging at broker during consumer crash
Posted by Hemanathan <he...@gmail.com>.
I tried with "soTimeout" parameter. Still connections are active in broker.
My transport connector configuration is
ssl://x.x.x.x:61617?transport.needClientAuth=true&transport.soTimeout=20000&transport.trace=true&maximumConnections=1000&wireformat.maxFrameSize=104857600
please let me know what is wrongly configured here.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Stale connection hanging at broker during consumer crash
Posted by Hemanathan <he...@gmail.com>.
I tried with soTimeout parameter. Still connections are active in broker.
My transport connector configuration is
ssl://x.x.x.x:61617?transport.needClientAuth=true&transport.soTimeout=20000&transport.trace=true&maximumConnections=1000&wireformat.maxFrameSize=104857600
please let me know what is wrongly configured here.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Stale connection hanging at broker during consumer crash
Posted by Tim Bain <tb...@alumni.duke.edu>.
The simplest way to do this is to set the soTimeout parameter on the TCP
transport (http://activemq.apache.org/tcp-transport-reference.html).
Because this is a protocol parameter, it can be set from either end of the
connection (or both), but if it were me I'd just set it on the broker's
transportConnector since that lets you set it once and have it always apply.
Tim
On Mon, Sep 24, 2018, 1:04 AM Hemanathan <he...@gmail.com>
wrote:
> Thanks for the answer Tim.
> Please suggest what are all the ways to configure timeout to close the
> stale
> connections.
> Need to configure on broker side or consumer side?
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: Stale connection hanging at broker during consumer crash
Posted by Hemanathan <he...@gmail.com>.
Thanks for the answer Tim.
Please suggest what are all the ways to configure timeout to close the stale
connections.
Need to configure on broker side or consumer side?
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Stale connection hanging at broker during consumer crash
Posted by Tim Bain <tb...@alumni.duke.edu>.
To me, this sounds like the expected behavior when dealing with TCP
connections that are not closed gracefully. TCP can only detect broken
connections by timing them out, so till the timeout elapses the broker will
think that a connection is still open.
Tim
On Fri, Sep 21, 2018, 7:16 AM Hemanathan <he...@gmail.com>
wrote:
> Stale connection hanging at broker during consumer crash :
>
> Step1 :
> When a consumer is getting attached with ActiveMQ broker, the consumer
> count
> is getting increased with one , which is verified through “dstat” command .
> Also the “netstat” from client machine shows one connection establishment
> with broker.
>
> Netstat:
> tcp 0 0 x.x.x.x:61617 x.x.x.x:44332
> ESTABLISHED 18625/java
>
> dstat value is:
> Name Queue
> Size Producer # Consumer # Enqueue # Dequeue # Forward # Memory
> %
> consumer.xyz.virtualTopic.abc.productUpdateNotificationEvent.1_0
> 8
> 0 1 10 2 0 0
>
>
> Step2 :
> Kill the client connection and no such connection in netstat from client
> machine. But still the connection hanging with broker and dstat showing the
> same result as before.
>
> Netstat:
> -NA-
>
> dstat value is:
> Name Queue
> Size Producer # Consumer # Enqueue # Dequeue # Forward # Memory
> %
> consumer.xyz.virtualTopic.abc.productUpdateNotificationEvent.1_0
> 8
> 0 1 10 2 0 0
>
> Step 3:
> Few of the message (multiples of 1000) not getting de-queue even after the
> new client getting connected. So the number of messages and number of
> consumers is getting increased on every attempt of client termination. When
> we try to change the "prefetchPolicy.queuePrefetch" property on consumer
> side, its reflecting so the number of messages are hanging (as configured
> in
> property . Eg. If we set value as "prefetchPolicy.queuePrefetch=10", then
> 10
> messages are hanging on the queue)
>
> Step 4 :
> After re-starting the broker , all the pending messages are getting
> de-queued, Where one client is up and running .
>
> Please suggest how to solve this problem.
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>