large number of connections from the same client

[khandelwalanuj <kh...@gmail.com>]

Hi,

I am seeing large number of connections ( more than 100) from the same
client to ActiveMQ broker. I am not able to understand the reason behind
this.
 
Will someone from ActiveMQ dev team help me out here ( what might cause a
client to have so many connections) ??

I am seeing this log message very frequently in broker log:

--------------------------------------
20131224 00:01:09:468 (ActiveMQ InactivityMonitor Worker)
org.apache.activemq.broker.TransportConnection#serviceTransportException 238
WARN] - Transport Connection to: tcp://IP_ADDRESS:31819 failed:
org.apache.activemq.transport.InactivityIOException: Channel was inactive
for too (>30000) long: tcp://IP_ADDRESS
--------------------------------------

I suspect some connection leak is there in ActiveMQ inactivity monitoring.

Thanks,
Anuj 



--
View this message in context: http://activemq.2283324.n4.nabble.com/large-number-of-connections-from-the-same-client-tp4675871.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Re: large number of connections from the same client

[artnaseef <ar...@artnaseef.com>]

What is the warning message?



--
View this message in context: http://activemq.2283324.n4.nabble.com/large-number-of-connections-from-the-same-client-tp4675871p4676007.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Re: large number of connections from the same client

[xabhi <xa...@gmail.com>]

Hey,

I have also seen this warning in my broker logs. Please let me know why this
is coming ?





--
View this message in context: http://activemq.2283324.n4.nabble.com/large-number-of-connections-from-the-same-client-tp4675871p4675966.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Re: large number of connections from the same client

[khandelwalanuj <kh...@gmail.com>]

Hi,

Waiting for some help from ActiveMQ development team.

Thanks,
Anuj



--
View this message in context: http://activemq.2283324.n4.nabble.com/large-number-of-connections-from-the-same-client-tp4675871p4675927.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Re: large number of connections from the same client

[artnaseef <ar...@artnaseef.com>]

Given the netstat results, the problem does not appear to be related to the
broker - the clients have opened the connections.

The pooled connection factory snippet looks OK.

So, the next step is to track down the source of the connections in the
client.  A more complete review of the configuration may help.  Analyzing a
heap dump may help as well - looking for the references to the connections
to understand why they exist.  Running the application in a debugger with a
breakpoint on the constructors for ActiveMQConnection could do the trick as
well.





--
View this message in context: http://activemq.2283324.n4.nabble.com/large-number-of-connections-from-the-same-client-tp4675871p4676008.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Re: large number of connections from the same client

[khandelwalanuj <kh...@gmail.com>]

Hey,

I have checked with all the netstat commands and found that all the
connections are ESTABLISHED.
--> lsof -p 27139 | grep TCP | wc -l
--> netstat commands you specified.

Both are giving me the same number of connections, all are ESTABLISHED.


Clients are also configured using pooledconnection factory:

  <bean id="jmsConnectionFactory"
        class="org.apache.activemq.pool.PooledConnectionFactory"
        destroy-method="stop">
        <property name="connectionFactory" ref="connectionFactory" />
        <property name="maxConnections" value="3" /> 
    </bean>


Configuration also looks fine to me.
But I am seeing again lot of connections from the same client. Not able to
figure out what is going wrong with broker ?

Thanks,
Anuj









--
View this message in context: http://activemq.2283324.n4.nabble.com/large-number-of-connections-from-the-same-client-tp4675871p4675921.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Re: large number of connections from the same client

[artnaseef <Ar...@Ticketmaster.com>]

First I would check a couple of things to confirm the connections:

1. Check the web console and/or MBeans (jconsole or visualvm) to confirm the
number of connections from the client.
2. Check the O/S for the number of connections using netstat (something like
this on Linux: netstat -an | grep 61616.  Or this: netstat -an | grep 6161 |
awk '{ print $5 }' | sed -e 's,::ffff:,,' -e 's,:[0-9]*$,,' | sort | uniq
-c)
3. Check the O/S state of the connections (are they all open?)
4. Check the client system for the same.

If the connects are in fact active from the client, then solving the case is
a matter of looking at the client configuration.  A misconfigured client can
easily cause this problem.  It's easy to create large numbers of connections
with connection pools.

Inactive connections from the client (especially sockets in CLOSED_WAIT
state) indicate either the client or server failed to close the socket
properly.  There's a known issue with the same using SSL in java 1.6 (I
think it's fixed in 1.7).

On the other hand, if the connections exist in the broker but not at the O/S
level, then it's time to look for a leak.



--
View this message in context: http://activemq.2283324.n4.nabble.com/large-number-of-connections-from-the-same-client-tp4675871p4675919.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.