You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Andrei Shakirin (JIRA)" <ji...@apache.org> on 2017/05/31 11:45:04 UTC
[jira] [Commented] (AMQ-6589) Broker hangs by shutdown after
loosing exclusive lock
[ https://issues.apache.org/jira/browse/AMQ-6589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16031034#comment-16031034 ]
Andrei Shakirin commented on AMQ-6589:
--------------------------------------
The issue is happens time to time again.
By last occurance broker needs ~ 1 hour to stop:
{code}
2017-05-29 15:33:37,209 | INFO | Lock file /CE/activemq/ESB_K2/lock, locked at Sun May 28 17:50:14 CEST 2017, has been modified at Mon May 29 15:33:36 CEST 2017 | org.apache.activemq.util.LockFile | ActiveMQ Lock KeepAlive Timer
2017-05-29 15:33:37,210 | ERROR | degtlun5214, no longer able to keep the exclusive lock so giving up being a master | org.apache.activemq.broker.LockableServiceSupport | ActiveMQ Lock KeepAlive Timer
2017-05-29 15:33:37,224 | INFO | Apache ActiveMQ 5.13.4 (degtlun5214, ID:degtlun5214-48167-1495986717775-1:1) is shutting down | org.apache.activemq.broker.BrokerService | ActiveMQ Lock KeepAlive Timer
2017-05-29 15:33:42,227 | INFO | The connection to 'tcp://172.26.244.150:40032' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Lock KeepAlive Timer
2017-05-29 15:33:47,227 | INFO | The connection to 'tcp://172.26.244.150:40032' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Lock KeepAlive Timer
2017-05-29 15:33:52,228 | INFO | The connection to 'tcp://172.26.244.150:40032' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Lock KeepAlive Timer
2017-05-29 15:33:57,228 | INFO | The connection to 'tcp://172.26.244.150:40032' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Lock KeepAlive Timer
2017-05-29 15:34:02,229 | INFO | The connection to 'tcp://172.26.244.150:40032' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Lock KeepAlive Timer
2017-05-29 15:34:07,229 | INFO | The connection to 'tcp://172.26.244.150:40032' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Lock KeepAlive Timer
2017-05-29 15:34:12,229 | INFO | The connection to 'tcp://172.26.244.150:40032' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Lock KeepAlive Timer
...
2017-05-29 16:24:07,453 | INFO | The connection to 'tcp://172.26.244.150:40032' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Lock KeepAlive Timer
2017-05-29 16:24:12,453 | INFO | The connection to 'tcp://172.26.244.150:40032' is taking a long time to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ Lock KeepAlive Timer
2017-05-29 16:24:38,999 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@27c20538: startup date [Mon May 29 16:24:38 CEST 2017]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2017-05-29 16:24:40,025 | INFO | Using Persistence Adapter: MultiKahaDBPersistenceAdapter[/CE/activemq/ESB_K2][] | org.apache.activemq.broker.BrokerService | main
{code}
Any chance to investigate and fix it?
> Broker hangs by shutdown after loosing exclusive lock
> -----------------------------------------------------
>
> Key: AMQ-6589
> URL: https://issues.apache.org/jira/browse/AMQ-6589
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.13.4
> Environment: Linux
> Reporter: Andrei Shakirin
> Attachments: activemq-old-master-anonymized.log, activemq.xml
>
>
> 1) Configuration: ActiveMQ brokers are configured with shared store (NFS) and shared file locker with configured lockAcquireSleepInterval:
> {code:xml}
> <persistenceAdapter>
> <mKahaDB directory="amq-share/" lockKeepAlivePeriod="10000">
> <locker>
> <shared-file-locker lockAcquireSleepInterval="15000"/>
> </locker>
> <filteredPersistenceAdapters>
> <filteredKahaDB perDestination="true">
> <persistenceAdapter>
> <kahaDB checkForCorruptJournalFiles="true" checksumJournalFiles="true" />
> </persistenceAdapter>
> </filteredKahaDB>
> </filteredPersistenceAdapters>
> </mKahaDB>
> </persistenceAdapter>
> {code}
> 2) Workflow
> The master broker looses exclusive lock and tries to shutdown, it is reported in the log file:
> {code}
> 2017-01-31 16:30:45,921 | INFO | Lock file /CE/activemq/lock, locked at Thu Jan 05 01:14:11 CET 2017, has been modified at Tue Jan 31 16:30:45 CET 2017 | org.apache.activemq.util.LockFile | ActiveMQ Lock KeepAlive Timer
> 2017-01-31 16:30:45,921 | ERROR | hostXXX, no longer able to keep the exclusive lock so giving up being a master | org.apache.activemq.broker.LockableServiceSupport | ActiveMQ Lock KeepAlive Timer
> 2017-01-31 16:30:45,924 | INFO | Apache ActiveMQ 5.13.4 (hostXXX, ID:xxx-36540-1483575278364-0:1) is shutting down | org.apache.activemq.broker.BrokerService | ActiveMQ Lock KeepAlive Timer
> {code}
> 3) Problem
> Broker hangs during shutdown, I see a lot of messages like:
> {code}
> The connection to 'tcp://xxx:55057' is taking a long time to shutdown.
> {code}
> The problem happens only in case of shutdown after loosing exclusive log, normal shutdown works fine.
> I see some other defects with this problem: AMQ-3435, AMQ-3293, AMQ-4073, but all of them have to be fixed in AMQ 5.13.4.
> What can be the reason of this problem?
> Note: the connections were created from "old" AMQ clients (5.7.0). Could the problem related with that?
> Complete log file and configuration are attached.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)