You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/03/04 20:06:45 UTC
[jira] [Commented] (AMQ-6601) AMQ with master and slave broker,
shutting down slave causes dead-lock
[ https://issues.apache.org/jira/browse/AMQ-6601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15895846#comment-15895846 ]
ASF GitHub Bot commented on AMQ-6601:
-------------------------------------
GitHub user catull opened a pull request:
https://github.com/apache/activemq/pull/228
[AMQ-6601] AMQ with master and slave broker, shutting down slave causes a dead-lock
This PR addresses the issue https://issues.apache.org/jira/browse/AMQ-6601.
The proposed solution is to start the broker asynchronously, but only wait for its starting up if it is the master (i.e. non-slave).
Closes / fixes issue #6601.
Credit goes to @lichtin for coming up with the solution.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/catull/activemq fix-issue-6601-shutdown-slave-broker
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/activemq/pull/228.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #228
----
commit 5de4f788d17f406c713d42a1bd77c15b973ac0de
Author: Carlo Dapor <ca...@gmail.com>
Date: 2017-03-04T20:01:34Z
[AMQ-6601] AMQ with master and slave broker, shutting down slave causes dead-lock.
----
> AMQ with master and slave broker, shutting down slave causes dead-lock
> ----------------------------------------------------------------------
>
> Key: AMQ-6601
> URL: https://issues.apache.org/jira/browse/AMQ-6601
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.14.0
> Reporter: Carlo Dapor
> Attachments: jstack-28657
>
>
> We have 2 karaf instances configured to be activemq brokers, broker-amq (b1) and broker2-amq (b2).
> They run on the same machine, use KahaDB with file locking.
> It does not matter if b1 or b2 is started first, it becomes the master.
> The other one, the slave, when shutting down - while the master is running - hits a dead-lock; it must be `kill -9`'d manually in the end.
> We have a classic dead-lock scenario. I have attached a `jstack` output when the slave broker is shutting down.
> The race is on between thread #20 and thread #17.
> Thread #17 is in
> {code:java}
> ActiveMQServiceFactory.destroy(ActiveMQServiceFactory.java:173)
> {code}
> and thread #20 is in
> {code:java}
> ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:140)
> {code}
> {code}
> "CM Configuration Updater (ManagedServiceFactory Update: factoryPid=[org.apache.activemq.server])" #20 daemon prio=5 os_prio=0 tid=0x00007f793c160800 nid=0x7084 waiting on condition [0x00007f799819f000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at java.lang.Thread.sleep(Thread.java:340)
> at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
> at org.apache.activemq.store.SharedFileLocker.doStart(SharedFileLocker.java:83)
> at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
> at org.apache.activemq.broker.LockableServiceSupport.preStart(LockableServiceSupport.java:94)
> at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:54)
> at org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:674)
> at org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:658)
> at org.apache.activemq.broker.BrokerService.start(BrokerService.java:622)
> at org.apache.activemq.osgi.ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:140)
> - locked <0x000000072bd74db0> (a org.apache.activemq.osgi.ActiveMQServiceFactory)
> at Proxy8890d2d1_e3a3_4b71_a7a0_88810df56856.updated(Unknown Source)
> at org.apache.felix.cm.impl.helper.ManagedServiceFactoryTracker.updated(ManagedServiceFactoryTracker.java:159)
> at org.apache.felix.cm.impl.helper.ManagedServiceFactoryTracker.provideConfiguration(ManagedServiceFactoryTracker.java:93)
> at org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.provide(ConfigurationManager.java:1597)
> at org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.run(ConfigurationManager.java:1540)
> at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:103)
> at java.lang.Thread.run(Thread.java:745)
>
> "Thread-4" #19 daemon prio=5 os_prio=0 tid=0x00007f7940002800 nid=0x7081 runnable [0x00007f79984b4000]
> java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at org.apache.karaf.main.ShutdownSocketThread.run(ShutdownSocketThread.java:56)
>
> "Thread-3" #18 prio=5 os_prio=0 tid=0x00007f79d0c48800 nid=0x7080 waiting on condition [0x00007f79985b5000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.karaf.main.Main.doMonitor(Main.java:299)
> at org.apache.karaf.main.Main.access$100(Main.java:65)
> at org.apache.karaf.main.Main$1.run(Main.java:275)
>
> "FelixStartLevel" #17 daemon prio=5 os_prio=0 tid=0x00007f79d0c48000 nid=0x707f waiting for monitor entry [0x00007f79986b5000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.activemq.osgi.ActiveMQServiceFactory.destroy(ActiveMQServiceFactory.java:173)
> - waiting to lock <0x000000072bd74db0> (a org.apache.activemq.osgi.ActiveMQServiceFactory)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:299)
> at org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:980)
> at org.apache.aries.blueprint.container.BeanRecipe.destroy(BeanRecipe.java:887)
> at org.apache.aries.blueprint.container.BlueprintRepository.destroy(BlueprintRepository.java:329)
> at org.apache.aries.blueprint.container.BlueprintContainerImpl.destroyComponents(BlueprintContainerImpl.java:765)
> at org.apache.aries.blueprint.container.BlueprintContainerImpl.tidyupComponents(BlueprintContainerImpl.java:964)
> at org.apache.aries.blueprint.container.BlueprintContainerImpl.destroy(BlueprintContainerImpl.java:909)
> at org.apache.aries.blueprint.container.BlueprintExtender$3.run(BlueprintExtender.java:325)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.aries.blueprint.container.BlueprintExtender.destroyContainer(BlueprintExtender.java:346)
> at org.apache.aries.blueprint.container.BlueprintExtender.modifiedBundle(BlueprintExtender.java:238)
> at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:500)
> at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:433)
> at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$AbstractTracked.track(BundleHookBundleTracker.java:725)
> at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.bundleChanged(BundleHookBundleTracker.java:463)
> at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$BundleEventHook.event(BundleHookBundleTracker.java:422)
> at org.apache.felix.framework.util.SecureAction.invokeBundleEventHook(SecureAction.java:1103)
> at org.apache.felix.framework.util.EventDispatcher.createWhitelistFromHooks(EventDispatcher.java:695)
> at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:483)
> at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4403)
> at org.apache.felix.framework.Felix.stopBundle(Felix.java:2520)
> at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1309)
> at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:304)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)