You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Timothy Bish (JIRA)" <ji...@apache.org> on 2014/06/09 19:01:05 UTC
[jira] [Updated] (AMQ-5219) Deadlock between JMS Job Scheduler and
broker initialization
[ https://issues.apache.org/jira/browse/AMQ-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Timothy Bish updated AMQ-5219:
------------------------------
Priority: Major (was: Critical)
> Deadlock between JMS Job Scheduler and broker initialization
> ------------------------------------------------------------
>
> Key: AMQ-5219
> URL: https://issues.apache.org/jira/browse/AMQ-5219
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.7.0
> Environment: Apache SMX 4.5.0, JDK 1.6, RHEL.
> Reporter: Raúl Kripalani
>
> Thread dump exhibiting a deadlock shows that JMS scheduler is started too early in the broker boot procedure. If scheduled jobs are pending and fired straightaway after scheduler initialisation – while other broker elements are still being initialised – this can incur in a deadlock which stalls the broker completely.
> {code}
> Java stack information for the threads listed above:
> ===================================================
> "JobScheduler:JMS":
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000006bca4ecb0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
> at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
> at org.apache.activemq.broker.region.AbstractRegion.getDestinations(AbstractRegion.java:240)
> at org.apache.activemq.broker.region.RegionBroker.getDestinations(RegionBroker.java:130)
> at org.apache.activemq.store.kahadb.KahaDBStore.rollbackStatsOnDuplicate(KahaDBStore.java:295)
> at org.apache.activemq.store.kahadb.MessageDatabase.upadateIndex(MessageDatabase.java:1165)
> at org.apache.activemq.store.kahadb.MessageDatabase$14.execute(MessageDatabase.java:982)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
> at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:980)
> at org.apache.activemq.store.kahadb.MessageDatabase$13.visit(MessageDatabase.java:927)
> at org.apache.activemq.store.kahadb.data.KahaAddMessageCommand.visit(KahaAddMessageCommand.java:241)
> at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:924)
> at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:836)
> at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:818)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.addMessage(KahaDBStore.java:433)
> at org.apache.activemq.store.kahadb.KahaDBTransactionStore.addMessage(KahaDBTransactionStore.java:385)
> at org.apache.activemq.store.kahadb.KahaDBTransactionStore$1.addMessage(KahaDBTransactionStore.java:157)
> at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:746)
> at org.apache.activemq.broker.region.Queue.send(Queue.java:717)
> at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:407)
> at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:503)
> at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:311)
> at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
> at org.apache.activemq.broker.scheduler.SchedulerBroker.scheduledJob(SchedulerBroker.java:251)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.fireJob(JobSchedulerImpl.java:423)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:473)
> - locked <0x00000006bca05f50> (a org.apache.activemq.broker.scheduler.JobSchedulerImpl)
> at org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
> at java.lang.Thread.run(Thread.java:662)
> "Start Level Event Dispatcher":
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000006bc5cd318> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
> at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.getMessageCount(KahaDBStore.java:477)
> at org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:101)
> at org.apache.activemq.broker.region.Queue.initialize(Queue.java:376)
> at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:86)
> at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532)
> at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
> at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137)
> at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283)
> - locked <0x00000006bca82920> (a java.util.concurrent.ConcurrentHashMap)
> at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175)
> at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
> at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
> at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)
> at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186)
> at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121)
> at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> at org.apache.activemq.broker.scheduler.SchedulerBroker.start(SchedulerBroker.java:89)
> at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122)
> at org.apache.activemq.broker.MutableBrokerFilter.start(MutableBrokerFilter.java:163)
> at org.apache.activemq.broker.BrokerPluginSupport.start(BrokerPluginSupport.java:40)
> at org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110)
> at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662)
> at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642)
> at org.apache.activemq.broker.BrokerService.start(BrokerService.java:578)
> at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:58)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1546)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1487)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1419)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:518)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:455)
> at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
> at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
> - locked <0x00000006bc3cb720> (a java.util.concurrent.ConcurrentHashMap)
> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
> at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
> - locked <0x00000006bc3cb510> (a java.util.concurrent.ConcurrentHashMap)
> at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
> at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$1600(AbstractDelegatedExecutionApplicationContext.java:69)
> at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$4.run(AbstractDelegatedExecutionApplicationContext.java:355)
> - locked <0x00000006bc3bccc0> (a java.lang.Object)
> at org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
> at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.completeRefresh(AbstractDelegatedExecutionApplicationContext.java:320)
> at org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$CompleteRefreshTask.run(DependencyWaiterApplicationContextExecutor.java:132)
> at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:48)
> at org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageTwo(DependencyWaiterApplicationContextExecutor.java:299)
> at org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplicationContextExecutor.java:244)
> at org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicationContextExecutor.java:169)
> at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.java:175)
> at org.springframework.osgi.extender.internal.activator.ContextLoaderListener$2.run(ContextLoaderListener.java:716)
> at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:48)
> at org.springframework.osgi.extender.internal.activator.ContextLoaderListener.maybeCreateApplicationContextFor(ContextLoaderListener.java:781)
> at org.springframework.osgi.extender.internal.activator.ContextLoaderListener$ContextBundleListener.handleEvent(ContextLoaderListener.java:229)
> at org.springframework.osgi.extender.internal.activator.ContextLoaderListener$BaseListener.bundleChanged(ContextLoaderListener.java:172)
> at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:919)
> at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:227)
> at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:149)
> at org.eclipse.osgi.framework.internal.core.Framework.publishBundleEventPrivileged(Framework.java:1349)
> at org.eclipse.osgi.framework.internal.core.Framework.publishBundleEvent(Framework.java:1300)
> at org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(BundleHost.java:380)
> at org.eclipse.osgi.framework.internal.core.AbstractBundle.resume(AbstractBundle.java:374)
> at org.eclipse.osgi.framework.internal.core.Framework.resumeBundle(Framework.java:1067)
> at org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartLevelManager.java:561)
> at org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartLevelManager.java:546)
> at org.eclipse.osgi.framework.internal.core.StartLevelManager.incFWSL(StartLevelManager.java:459)
> at org.eclipse.osgi.framework.internal.core.StartLevelManager.doSetStartLevel(StartLevelManager.java:243)
> - locked <0x00000006ba9f9338> (a java.lang.Object)
> at org.eclipse.osgi.framework.internal.core.StartLevelManager.dispatchEvent(StartLevelManager.java:440)
> at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:227)
> at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:337)
> Found 1 deadlock.
> {code}
> Useful info:
> * This was seen in an OSGi environment, Apache SMX 4.5.0.
> * Broker is being initialised through Spring configuration. Original OOTB Blueprint activemq-broker.xml was replaced with Spring equivalent, with the following manifest option:
> {code}
> <manifest>
> Spring-Context: *;publish-context:=false;create-asynchronously:=false
> </manifest>
> {code}
> ** This ensures that the broker is started before all other business bundles (by starting this Spring context synchronously, the OSGi framework waits until the bundle is started before continuing with starting the rest).
> * No other broker plugins apart from the redeliveryPlugin are configured. Config of this one is as follows:
> {code}
> <redeliveryPlugin fallbackToDeadLetter="true" sendToDlqIfMaxRetriesExceeded="true">
> <redeliveryPolicyMap>
> <redeliveryPolicyMap>
> <redeliveryPolicyEntries>
> ...
> </redeliveryPolicyEntries>
> </redeliveryPolicyMap>
> </redeliveryPolicyMap>
> </redeliveryPlugin>
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)