You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by pdudits <na...@dudits.net> on 2013/06/03 16:53:53 UTC
Endpoint factory / ClientID problems during deployment on Glassfish
Hello,
I've run into following issue with ActiveMQ 5.5.1 (fuse-10-16) on Glassfish
3.1.2.2,
two instances running embedded brokers that are connected over network
connector.
During startup of an application I'll get following exception for some of
the MDBs:
[#|2013-05-30T09:14:49.612+0200|WARNING|glassfish3.1.2|javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.inbound|_ThreadID=276;_ThreadName=Thread-2;|
RAR7004 : MDB deployment is still happening. Cannot create end point now.
|#]
[#|2013-05-30T09:14:49.596+0200|SEVERE|glassfish3.1.2|org.apache.activemq.ra.ActiveMQEndpointWorker|
_ThreadID=411;_ThreadName=Thread-2;|
Connection to broker failed: Endpoint factory did not allow creation of any
endpoints.
javax.jms.JMSException: Endpoint factory did not allow creation of any
endpoints.
at
org.apache.activemq.ra.ServerSessionPoolImpl.getServerSession(ServerSessionPoolImpl.java:123)
at
org.apache.activemq.ActiveMQConnectionConsumer.dispatch(ActiveMQConnectionConsumer.java:129)
at
org.apache.activemq.ActiveMQConnection$2.processMessageDispatch(ActiveMQConnection.java:1756)
at
org.apache.activemq.command.MessageDispatch.visit(MessageDispatch.java:109)
at
org.apache.activemq.ActiveMQConnection.onCommand(ActiveMQConnection.java:1737)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:233)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
|#]
So far so good - Glassfish will not allow any endpoints to be activated
while deployment is in progress.
It tries to wait 60 seconds, and if the deployment is still running, it will
fail with UnavailableException
(Application's startup time is around 3 minutes). But some time later, I'll
get this:
[#|2013-05-30T09:14:49.737+0200|WARNING|glassfish3.1.2|org.apache.activemq.broker.TransportConnection|
_ThreadID=265;_ThreadName=Thread-2;|
Failed to add Connection ID:S030A1193-60183-1369897804974-313:1, reason:
javax.jms.InvalidClientIDException: Broker: iconQR1 - Client: XXXX already
connected from vm://QR1#154
|#]
[#|2013-05-30T09:14:49.737+0200|FINE|glassfish3.1.2|org.apache.activemq.ra.ActiveMQEndpointWorker|
_ThreadID=296;_ThreadName=Thread-2;ClassName=org.apache.activemq.ra.ActiveMQEndpointWorker$1;MethodName=run;|
Failed to connect: Broker: iconQR1 - Client: XXXX already connected from
vm://QR1#154
javax.jms.InvalidClientIDException: Broker: QR1 - Client: XXXX already
connected from vm://QR1#154
at
org.apache.activemq.broker.region.RegionBroker.addConnection(RegionBroker.java:230)
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:85)
at
org.apache.activemq.advisory.AdvisoryBroker.addConnection(AdvisoryBroker.java:81)
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:85)
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:85)
at
org.apache.activemq.broker.MutableBrokerFilter.addConnection(MutableBrokerFilter.java:91)
at
org.apache.activemq.broker.TransportConnection.processAddConnection(TransportConnection.java:715)
at
org.apache.activemq.command.ConnectionInfo.visit(ConnectionInfo.java:139)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:233)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
|#]
These two entries likely do not correlate to the same MDB, but these are the
only two kinds of exceptions
I get during startup. The affected subscription is not always the same.
My understanding of this situation is, that a new endpoint is deployed, and
since it is already an existing
subscription with pending messages, ActiveMQ RA will try to deliver the
message. It will fail, because
deployment is not yet finished, worker closes the consumer and connection,
but broker will not get or
process remove connection message. Therefore, when ActiveMQEndpointWorker
tries to connect again,
the ClientID is already taken (although the connection is already used for
different purpose, or idle).
This situation will not resolve, the old connection will not get removed by
itself.
Is there a way to correct this behaviour?
My connection factory uses attribute jms.watchTopicAdvisories=false,
otherwise there were issues with
replying to temp queues from remote broker.
Thanks in advance
-----
---
Patrik Dudits
--
View this message in context: http://activemq.2283324.n4.nabble.com/Endpoint-factory-ClientID-problems-during-deployment-on-Glassfish-tp4667751.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Endpoint factory / ClientID problems during deployment on
Glassfish
Posted by sbarlabanov <co...@sergiy-barlabanov.de>.
Great! Thanks!
--
View this message in context: http://activemq.2283324.n4.nabble.com/Endpoint-factory-ClientID-problems-during-deployment-on-Glassfish-tp4667751p4691525.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Endpoint factory / ClientID problems during deployment on
Glassfish
Posted by pdudits <na...@dudits.net>.
I ended up patching ServerSessionPoolImpl, so it would block and wait for the
session instead of releasing the connection. I even managed to cherrypick
the commit for you on top of 5.10:
https://github.com/pdudits/activemq/tree/sync-session-pool-5.10.0
Good luck!
-----
---
Patrik Dudits
--
View this message in context: http://activemq.2283324.n4.nabble.com/Endpoint-factory-ClientID-problems-during-deployment-on-Glassfish-tp4667751p4691513.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Endpoint factory / ClientID problems during deployment on
Glassfish
Posted by sbarlabanov <co...@sergiy-barlabanov.de>.
We are experiencing the same problem now in the production with ActiveMQ
5.10.0 and Glassfish 3.1.2.2.
Does anybody know what the problem is and how to solve it?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Endpoint-factory-ClientID-problems-during-deployment-on-Glassfish-tp4667751p4691508.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Endpoint factory / ClientID problems during deployment on
Glassfish
Posted by pdudits <na...@dudits.net>.
I take my statement back, the error occurred again. Is there anything I can
do to troubleshoot the problem
further?
-----
---
Patrik Dudits
--
View this message in context: http://activemq.2283324.n4.nabble.com/Endpoint-factory-ClientID-problems-during-deployment-on-Glassfish-tp4667751p4667883.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Endpoint factory / ClientID problems during deployment on
Glassfish
Posted by pdudits <na...@dudits.net>.
It looks, that putting ?jms.watchTopicAdvisories=false in ServerUrl of
resource adapter was causing this.
I moved the setting to connection pool instead (as advisories caused
troubles only on sending messages to
temporary queues from remote brokers), and it appears that the server is now
able recover from this situation
correctly.
-----
---
Patrik Dudits
--
View this message in context: http://activemq.2283324.n4.nabble.com/Endpoint-factory-ClientID-problems-during-deployment-on-Glassfish-tp4667751p4667795.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.