You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Steven Janssens <St...@axi.be> on 2020/10/23 12:42:21 UTC
Connection stability issues? Any kind of help is greatly appreciated
Hi all,
I'm faced with stability issues in my java application which only occurs on the UAT environment and not (let's hope it stays that way) on PRD.
The issue I'm dealing with:
After some undetermined time frame the application grinds to a hold.
There doesn't seem to be any pattern on then functions used in the application when this happens.
At that time the issue occurs the log is full of stacktraces that there a no jdbc connections available in the pool within the given timeout period.
A thread dump from both JVM's (java application and ActiveMQ) makes me believe that the issue is related to ActiveMQ.
This is the reason why I post this question in this mailing list.
Distributed transactions are used to combine jdbc and jms transactions into one XA transaction.
The application is deployed in Karaf (Servicemix) so it leverages the Aries Transaction Manager to coordinate the global transaction.
The thread dump from Servicemix makes me believe that the Distributed transaction is blocked by the call to ActiveMQ waiting to receive a response from ActiveMQ.
This eventually results in exhausting the available jdbc connections.
I hope that someone on this mailing list can guide me in solving the issue (or at least give me some new pointers to investigate and diagnose).
Servicemix thread dump extract:
The thread dump is full of these WAITING (parked) threads all originated from the same line of code within the application (EventManagerServiceImpl) which posts a message to a Virtual Topic.
For me this reads like the thread is waiting for a response from ActiveMQ.
Is this a correct assumption?
"qtp1039116670-1733" #1733 prio=5 os_prio=0 tid=0x00007fa65c3d3800 nid=0x10970 waiting on condition [0x00007fa61e3b1000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e9f2ed18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:48)
at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:87)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1388)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1428)
at org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:2081)
at org.apache.activemq.ActiveMQMessageProducer.<init>(ActiveMQMessageProducer.java:124)
at org.apache.activemq.ActiveMQSession.createProducer(ActiveMQSession.java:1112)
at org.apache.activemq.jms.pool.SessionHolder.getOrCreateProducer(SessionHolder.java:62)
- locked <0x00000000e9f2d220> (a org.apache.activemq.jms.pool.SessionHolder)
at org.apache.activemq.jms.pool.PooledSession.getMessageProducer(PooledSession.java:393)
at org.apache.activemq.jms.pool.PooledSession.createProducer(PooledSession.java:368)
at org.springframework.jms.core.JmsTemplate.doCreateProducer(JmsTemplate.java:971)
at org.springframework.jms.core.JmsTemplate.createProducer(JmsTemplate.java:952)
at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.doSendToDestination(JmsConfiguration.java:480)
at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.access$100(JmsConfiguration.java:426)
at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate$1.doInJms(JmsConfiguration.java:440)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:466)
at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.send(JmsConfiguration.java:437)
at org.apache.camel.component.jms.JmsProducer.doSend(JmsProducer.java:413)
at org.apache.camel.component.jms.JmsProducer.processInOnly(JmsProducer.java:367)
at org.apache.camel.component.jms.JmsProducer.process(JmsProducer.java:153)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:196)
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
at org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:68)
at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:412)
at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:380)
at org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:270)
at org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:380)
at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:221)
at org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:124)
at org.apache.camel.impl.DefaultProducerTemplate.sendBodyAndHeaders(DefaultProducerTemplate.java:233)
at org.apache.camel.impl.DefaultProducerTemplate.sendBodyAndHeaders(DefaultProducerTemplate.java:229)
at eventManager.services.EventManagerServiceImpl.publishEvent(EventManagerServiceImpl.java:41)
ActiveMQ thread dump extract:
There a different thread...I extracted samples for the once I believe might be important to diagnose the issue I'm running into.
I'm not quite sure how I must interpret this thread dump. So any help is greatly appreciated!
"ActiveMQ BrokerService[localhost] Task-383" #522 daemon prio=5 os_prio=0 tid=0x00007fbd5c03d000 nid=0x17f98 waiting on condition [0x00007fbd38998000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c1c5a428> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"ActiveMQ JDBC PA Scheduled Task" #75 daemon prio=5 os_prio=0 tid=0x00007fbd2c002000 nid=0xce60 waiting on condition [0x00007fbd38897000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c0863d68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"ActiveMQ Transport: tcp:///172.20.10.31:20014@61616" #115 daemon prio=5 os_prio=0 tid=0x00007fbd50004000 nid=0xd2ac in Object.wait() [0x00007fbd278f6000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.postgresql.jdbc.PgStatement.killTimerTask(PgStatement.java:932)
- locked <0x00000000c43db7e0> (a org.postgresql.jdbc.PgConnection)
at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:796)
at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1659)
at org.apache.activemq.store.jdbc.TransactionContext.executeBatch(TransactionContext.java:130)
at org.apache.activemq.store.jdbc.TransactionContext.executeBatch(TransactionContext.java:108)
at org.apache.activemq.store.jdbc.TransactionContext.close(TransactionContext.java:156)
at org.apache.activemq.store.jdbc.JDBCMessageStore.addMessage(JDBCMessageStore.java:165)
at org.apache.activemq.store.memory.MemoryTransactionStore.addMessage(MemoryTransactionStore.java:352)
at org.apache.activemq.store.memory.MemoryTransactionStore$1.addMessage(MemoryTransactionStore.java:143)
at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:840)
at org.apache.activemq.broker.region.Queue.send(Queue.java:727)
at org.apache.activemq.broker.region.DestinationFilter.send(DestinationFilter.java:132)
at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:503)
at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:468)
at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:293)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:153)
at org.apache.activemq.broker.scheduler.SchedulerBroker.send(SchedulerBroker.java:195)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:153)
at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:293)
at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:158)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:153)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:153)
at org.apache.activemq.security.AuthorizationBroker.send(AuthorizationBroker.java:226)
at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:158)
at org.apache.activemq.util.BrokerSupport.doResend(BrokerSupport.java:69)
at org.apache.activemq.util.BrokerSupport.resendNoCopy(BrokerSupport.java:38)
at org.apache.activemq.broker.region.RegionBroker.sendToDeadLetterQueue(RegionBroker.java:799)
at org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:319)
at org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:319)
at org.apache.activemq.advisory.AdvisoryBroker.sendToDeadLetterQueue(AdvisoryBroker.java:721)
at org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:319)
at org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:319)
at org.apache.activemq.broker.MutableBrokerFilter.sendToDeadLetterQueue(MutableBrokerFilter.java:331)
at org.apache.activemq.broker.util.RedeliveryPlugin.sendToDeadLetterQueue(RedeliveryPlugin.java:150)
at org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:319)
at org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:319)
at org.apache.activemq.broker.MutableBrokerFilter.sendToDeadLetterQueue(MutableBrokerFilter.java:331)
at org.apache.activemq.broker.region.PrefetchSubscription.sendToDLQ(PrefetchSubscription.java:535)
at org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:394)
- locked <0x00000000c43a0198> (a java.lang.Object)
at org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:526)
at org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:484)
at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:88)
at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:88)
at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:88)
at org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:276)
at org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:98)
at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:88)
at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:88)
at org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:98)
at org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:590)
at org.apache.activemq.command.MessageAck.visit(MessageAck.java:245)
at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:336)
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:200)
at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)
at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:301)
at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:233)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
at java.lang.Thread.run(Thread.java:748)
Some useful background information:
UAT vs PRD
The operations team members have guaranteed that the UAT and PRD environments are exact copies of each other.
The only difference is that the UAT is running on less powerful infrastructure.
ActiveMQ
Version: 5.14.2
ActiveMQ is configured to use the jdbc peristence adapter which uses a org.postgresql.ds.PGSimpleDataSource DataSource.
<bean id="postgres-ds" class="org.postgresql.ds.PGSimpleDataSource">
<property name="serverName" value="xxxx"/>
<property name="databaseName" value="activemq"/>
<property name="portNumber" value="5432"/>
<property name="user" value="activemq"/>
<property name="password" value="xxxx"/>
</bean>
Thanks for your advice and guidance!
Steven.
Steven Janssens
Integration Architect
e. Steven.Janssens@axi.be
t. + 32 495 58 07 07
AXI nv, Molenweg 107, BE-2830 Willebroek - www.axi.be<https://www.axi.be/>
Follow us on LinkedIn<https://www.linkedin.com/company/axi> & Facebook<https://www.facebook.com/axinews>
[cid:AXIlogo_95c5e8fe-4564-4b01-8247-acc454872849.gif]
AXI Group members are Ground lion<https://www.groundlion.be/>, LoQutus<https://www.loqutus.com/> & Elegio<https://www.elegio.eu/>
[cid:AXI_Cloud_Event_Mailfooter_Banner_AXI_5bf7612c-2658-4f42-8342-c5ad5957d563.jpg]<https://www.axi.be/nl-be/evenementen/axi-cloud-event-how-become-cloud-native><https://www.axi.be/nl-be/evenementen/axi-cloud-event-how-become-cloud-native>