You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by cooshal <ku...@gmail.com> on 2019/06/11 08:20:43 UTC

Too Many TCP Connections, ultimately causing the connections to be dropped by the broker

Hi:

I have a difficult situation here, and I am not able to comprehend this
issue.

I have the following error, and I understand that it has reached max.
connections, and I need to change the value for max. connections.


| ERROR | Could not accept connection  : {} |
org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server
Thread Handler:
tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
org.apache.activemq.transport.tcp.ExceededMaximumConnectionsException:
Exceeded the maximum number of allowed client connections. See the
'maximumConnections' property on the TCP transport configuration URI in the
ActiveMQ configuration file (e.g., activemq.xml)
	at
org.apache.activemq.transport.tcp.TcpTransportServer.doHandleSocket(TcpTransportServer.java:567)[activemq-client-5.15.4.jar:5.15.4]
	at
org.apache.activemq.transport.tcp.TcpTransportServer.handleSocket(TcpTransportServer.java:556)[activemq-client-5.15.4.jar:5.15.4]
	at
org.apache.activemq.transport.tcp.TcpTransportServer$1.run(TcpTransportServer.java:482)[activemq-client-5.15.4.jar:5.15.4]
	at java.lang.Thread.run(Unknown Source)[:1.8.0_172]
2019-06-10 21:22:03,075 | ERROR | Could not accept connection  : {} |
org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server
Thread Handler:
tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600

Now, there are two scenarios from here on.

One is my development environment, where I tried deploying my broker using
docker, or simply running an activemq broker from the downloaded
distribution (on Windows), and the version is : 5.14.3. This works
completely fine, and I see something like below, in the ActiveMQ console:

Name  	Remote Address  	Active  	Slow   
ID:MDK-10884-1559899975517-0:2	tcp://127.0.0.1:10942	true	false 
ID:MDK-10884-1559899975517-2:8	tcp://127.0.0.1:12448	true	false 

Clicking on one of those two connections, I can see all my consumer
sessions.

I have the same broker configuration in my staging environment (which is
Windows 2012 R2), and broker version is 5.15.4. I am bound to that OS for
the time being. 

From the ActiveMQ web console, I see hundreds of connections. The same is
from netstat as well.

On both cases, I am using this broker from Apache Karaf (which hosts camel
routes bundles). The bundles and karaf instances are same on both
environments. I have used pax-jms and also configured connection pooling.
But, neither of these settings seem to have an effect in the overloaded
number of connections.

I think, I have encountered this issue, after I switched to pax-jms.

Any pointers on this would be very helpful.

Regards,
Cooshal.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: Too Many TCP Connections, ultimately causing the connections to be dropped by the broker

Posted by cooshal <ku...@gmail.com>.
Hi:

yes.

Just figured out a way to get out of this issue.

Somehow, the broker was caching the connections (because restarting the
instance did not cause the instance to use the pooled connection), and there
were a lot of Temp Advisory Queues. I have no idea how and why. I stopped
all the instances, stopped the broker, and restarted all the instances
one-by-one. And now, I see that all my instances are using pooled
connection. Instead of hundreds of connections, there are just 5 connections
with their respective Consumer sessions inside them.

Strangely, I did not configure anything new, but restarting everything
(which I did yesterday as well :D) resolved this issue.

Regards,
Cooshal.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: Too Many TCP Connections, ultimately causing the connections to be dropped by the broker

Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
The ConnectionFactory is exactly the same configuration between the
instances ?

Regards
JB

On 12/06/2019 09:40, cooshal wrote:
> Hi:
> 
> This is a very weird issue that I am facing right now.
> 
> I have 5 different karaf instances, on the same machine, and all those
> instances have the same bundles installed. 4 out of 5 instances show just
> one connection in the activemq web console (multiple consumer sessions
> inside it). There is only one instance, that is giving trouble. I have fresh
> installed this instance, restarted the broker, and restarted all the
> instances. But, this strange issue persists.
> 
> And, I realized that, this is the cause of one of the issues that I am
> facing right now
> (http://activemq.2283324.n4.nabble.com/Sporadic-quot-Error-while-attempting-to-add-new-Connection-to-the-pool-nested-exception-is-javax-jms-td4741737.html).
> 
> I will look further to check if I might have misconfigured something.
> 
> Regards,
> Cooshal.
> 
> 
> 
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
> 

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Too Many TCP Connections, ultimately causing the connections to be dropped by the broker

Posted by cooshal <ku...@gmail.com>.
Hi:

This is a very weird issue that I am facing right now.

I have 5 different karaf instances, on the same machine, and all those
instances have the same bundles installed. 4 out of 5 instances show just
one connection in the activemq web console (multiple consumer sessions
inside it). There is only one instance, that is giving trouble. I have fresh
installed this instance, restarted the broker, and restarted all the
instances. But, this strange issue persists.

And, I realized that, this is the cause of one of the issues that I am
facing right now
(http://activemq.2283324.n4.nabble.com/Sporadic-quot-Error-while-attempting-to-add-new-Connection-to-the-pool-nested-exception-is-javax-jms-td4741737.html).

I will look further to check if I might have misconfigured something.

Regards,
Cooshal.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: Too Many TCP Connections, ultimately causing the connections to be dropped by the broker

Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi,

I would check your clients first, if we are talking about Camel routes,
maybe you are using a EIP creating a bunch of threads/connections.

Regards
JB

On 11/06/2019 14:46, cooshal wrote:
> Hi again:
> 
> I configured my client to use pooled jms connection, via pax-jms.
> 
> This is how my config looks like:
> 
> name = consumer
> jms.url = tcp://localhost:61616
> jms.username = system
> jms.password = manager
> type = activemq
> pool = pooledjms
> osgi.jndi.service.name = jms/consumer
> org.apache.karaf.features.configKey = org.ops4j.connectionfactory-consumer
> 
> But, I still see hundreds of connections. The same settings in my local
> system behaves completely normal, with one connection and multiple consumer
> sessions using that connection.
> 
> Also, I tried restarting my karaf instances. But, does not help much on this
> issue.
> 
> Regards,
> Cooshal.
> 
> 
> 
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
> 

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Too Many TCP Connections, ultimately causing the connections to be dropped by the broker

Posted by cooshal <ku...@gmail.com>.
Hi again:

I configured my client to use pooled jms connection, via pax-jms.

This is how my config looks like:

name = consumer
jms.url = tcp://localhost:61616
jms.username = system
jms.password = manager
type = activemq
pool = pooledjms
osgi.jndi.service.name = jms/consumer
org.apache.karaf.features.configKey = org.ops4j.connectionfactory-consumer

But, I still see hundreds of connections. The same settings in my local
system behaves completely normal, with one connection and multiple consumer
sessions using that connection.

Also, I tried restarting my karaf instances. But, does not help much on this
issue.

Regards,
Cooshal.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: Too Many TCP Connections, ultimately causing the connections to be dropped by the broker

Posted by cooshal <ku...@gmail.com>.
Hi:

thank you for your reply.

I will inspect those number and post back.

Regards,
Cooshal.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: Too Many TCP Connections, ultimately causing the connections to be dropped by the broker

Posted by Anna Jaworska <va...@gmail.com>.
Make sure your camel routes are using pooled connection factory. If not, it
might happen that each route run connects anew for each run.

Observe advisory topic Activemq. Advisory.Connection - each equeued
messages there is either connection open action or connection close action.
If amount of messages there is very high it should be alarming.  Not using
pooled connection factory will often cause this.


wt., 11 cze 2019, 10:20 użytkownik cooshal <ku...@gmail.com>
napisał:

> Hi:
>
> I have a difficult situation here, and I am not able to comprehend this
> issue.
>
> I have the following error, and I understand that it has reached max.
> connections, and I need to change the value for max. connections.
>
>
> | ERROR | Could not accept connection  : {} |
> org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server
> Thread Handler:
> tcp://
> 0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
> org.apache.activemq.transport.tcp.ExceededMaximumConnectionsException:
> Exceeded the maximum number of allowed client connections. See the
> 'maximumConnections' property on the TCP transport configuration URI in the
> ActiveMQ configuration file (e.g., activemq.xml)
>         at
>
> org.apache.activemq.transport.tcp.TcpTransportServer.doHandleSocket(TcpTransportServer.java:567)[activemq-client-5.15.4.jar:5.15.4]
>         at
>
> org.apache.activemq.transport.tcp.TcpTransportServer.handleSocket(TcpTransportServer.java:556)[activemq-client-5.15.4.jar:5.15.4]
>         at
>
> org.apache.activemq.transport.tcp.TcpTransportServer$1.run(TcpTransportServer.java:482)[activemq-client-5.15.4.jar:5.15.4]
>         at java.lang.Thread.run(Unknown Source)[:1.8.0_172]
> 2019-06-10 21:22:03,075 | ERROR | Could not accept connection  : {} |
> org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server
> Thread Handler:
> tcp://
> 0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
>
> Now, there are two scenarios from here on.
>
> One is my development environment, where I tried deploying my broker using
> docker, or simply running an activemq broker from the downloaded
> distribution (on Windows), and the version is : 5.14.3. This works
> completely fine, and I see something like below, in the ActiveMQ console:
>
> Name    Remote Address          Active          Slow
> ID:MDK-10884-1559899975517-0:2  tcp://127.0.0.1:10942   true    false
> ID:MDK-10884-1559899975517-2:8  tcp://127.0.0.1:12448   true    false
>
> Clicking on one of those two connections, I can see all my consumer
> sessions.
>
> I have the same broker configuration in my staging environment (which is
> Windows 2012 R2), and broker version is 5.15.4. I am bound to that OS for
> the time being.
>
> From the ActiveMQ web console, I see hundreds of connections. The same is
> from netstat as well.
>
> On both cases, I am using this broker from Apache Karaf (which hosts camel
> routes bundles). The bundles and karaf instances are same on both
> environments. I have used pax-jms and also configured connection pooling.
> But, neither of these settings seem to have an effect in the overloaded
> number of connections.
>
> I think, I have encountered this issue, after I switched to pax-jms.
>
> Any pointers on this would be very helpful.
>
> Regards,
> Cooshal.
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>