You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Raúl Kripalani (JIRA)" <ji...@apache.org> on 2014/06/09 13:29:01 UTC
[jira] [Created] (AMQ-5219) Deadlock between JMS Job Scheduler and
broker initialization
Raúl Kripalani created AMQ-5219:
-----------------------------------
Summary: Deadlock between JMS Job Scheduler and broker initialization
Key: AMQ-5219
URL: https://issues.apache.org/jira/browse/AMQ-5219
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.7.0
Environment: Apache SMX 4.5.0, JDK 1.6, RHEL.
Reporter: Raúl Kripalani
Priority: Critical
Thread dump exhibiting a deadlock shows that JMS scheduler is started too early in the broker boot procedure. If scheduled jobs are pending and fired straightaway after scheduler initialisation – while other broker elements are still being initialised – this can incur in a deadlock which stalls the broker completely.
{code}
Java stack information for the threads listed above:
===================================================
"JobScheduler:JMS":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006bca4ecb0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
at org.apache.activemq.broker.region.AbstractRegion.getDestinations(AbstractRegion.java:240)
at org.apache.activemq.broker.region.RegionBroker.getDestinations(RegionBroker.java:130)
at org.apache.activemq.store.kahadb.KahaDBStore.rollbackStatsOnDuplicate(KahaDBStore.java:295)
at org.apache.activemq.store.kahadb.MessageDatabase.upadateIndex(MessageDatabase.java:1165)
at org.apache.activemq.store.kahadb.MessageDatabase$14.execute(MessageDatabase.java:982)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:980)
at org.apache.activemq.store.kahadb.MessageDatabase$13.visit(MessageDatabase.java:927)
at org.apache.activemq.store.kahadb.data.KahaAddMessageCommand.visit(KahaAddMessageCommand.java:241)
at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:924)
at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:836)
at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:818)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.addMessage(KahaDBStore.java:433)
at org.apache.activemq.store.kahadb.KahaDBTransactionStore.addMessage(KahaDBTransactionStore.java:385)
at org.apache.activemq.store.kahadb.KahaDBTransactionStore$1.addMessage(KahaDBTransactionStore.java:157)
at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:746)
at org.apache.activemq.broker.region.Queue.send(Queue.java:717)
at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:407)
at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:503)
at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:311)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
at org.apache.activemq.broker.scheduler.SchedulerBroker.scheduledJob(SchedulerBroker.java:251)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.fireJob(JobSchedulerImpl.java:423)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:473)
- locked <0x00000006bca05f50> (a org.apache.activemq.broker.scheduler.JobSchedulerImpl)
at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
at java.lang.Thread.run(Thread.java:662)
"Start Level Event Dispatcher":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006bc5cd318> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.getMessageCount(KahaDBStore.java:477)
at org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:101)
at org.apache.activemq.broker.region.Queue.initialize(Queue.java:376)
at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:86)
at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532)
at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137)
at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283)
- locked <0x00000006bca82920> (a java.util.concurrent.ConcurrentHashMap)
at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175)
at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)
at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186)
at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121)
at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
at org.apache.activemq.broker.scheduler.SchedulerBroker.start(SchedulerBroker.java:89)
at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122)
at org.apache.activemq.broker.MutableBrokerFilter.start(MutableBrokerFilter.java:163)
at org.apache.activemq.broker.BrokerPluginSupport.start(BrokerPluginSupport.java:40)
at org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110)
at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662)
at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642)
at org.apache.activemq.broker.BrokerService.start(BrokerService.java:578)
at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1546)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1487)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1419)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:518)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:455)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
- locked <0x00000006bc3cb720> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
- locked <0x00000006bc3cb510> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$1600(AbstractDelegatedExecutionApplicationContext.java:69)
at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$4.run(AbstractDelegatedExecutionApplicationContext.java:355)
- locked <0x00000006bc3bccc0> (a java.lang.Object)
at org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.completeRefresh(AbstractDelegatedExecutionApplicationContext.java:320)
at org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$CompleteRefreshTask.run(DependencyWaiterApplicationContextExecutor.java:132)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:48)
at org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageTwo(DependencyWaiterApplicationContextExecutor.java:299)
at org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplicationContextExecutor.java:244)
at org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicationContextExecutor.java:169)
at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.java:175)
at org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:716)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:48)
at org.springframework.osgi.extender.internal.activator.ContextLoaderListener.maybeCreateApplicationContextFor(ContextLoaderListener.java:781)
at org.springframework.osgi.extender.internal.activator.ContextLoaderListener$ContextBundleListener.handleEvent(ContextLoaderListener.java:229)
at org.springframework.osgi.extender.internal.activator.ContextLoaderListener$BaseListener.bundleChanged(ContextLoaderListener.java:172)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:919)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:227)
at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:149)
at org.eclipse.osgi.framework.internal.core.Framework.publishBundleEventPrivileged(Framework.java:1349)
at org.eclipse.osgi.framework.internal.core.Framework.publishBundleEvent(Framework.java:1300)
at org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(BundleHost.java:380)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.resume(AbstractBundle.java:374)
at org.eclipse.osgi.framework.internal.core.Framework.resumeBundle(Framework.java:1067)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartLevelManager.java:561)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartLevelManager.java:546)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.incFWSL(StartLevelManager.java:459)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.doSetStartLevel(StartLevelManager.java:243)
- locked <0x00000006ba9f9338> (a java.lang.Object)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.dispatchEvent(StartLevelManager.java:440)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:227)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:337)
Found 1 deadlock.
{code}
Useful info:
* This was seen in an OSGi environment, Apache SMX 4.5.0.
* Broker is being initialised through Spring configuration. Original OOTB Blueprint activemq-broker.xml was replaced with Spring equivalent, with the following manifest option:
{code}
<manifest>
Spring-Context: *;publish-context:=false;create-asynchronously:=false
</manifest>
{code}
** This ensures that the broker is started before all other business bundles (by starting this Spring context synchronously, the OSGi framework waits until the bundle is started before continuing with starting the rest).
* No other broker plugins apart from the redeliveryPlugin are configured. Config of this one is as follows:
{code}
<redeliveryPlugin fallbackToDeadLetter="true" sendToDlqIfMaxRetriesExceeded="true">
<redeliveryPolicyMap>
<redeliveryPolicyMap>
<redeliveryPolicyEntries>
...
</redeliveryPolicyEntries>
</redeliveryPolicyMap>
</redeliveryPolicyMap>
</redeliveryPlugin>
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)