[jira] [Created] (AMQ-7147) Stale DefaultIOExceptionHandler Thread stuck in while loop

["sunil (JIRA)" <ji...@apache.org>]

sunil created AMQ-7147:
--------------------------

             Summary: Stale DefaultIOExceptionHandler Thread stuck in while loop
                 Key: AMQ-7147
                 URL: https://issues.apache.org/jira/browse/AMQ-7147
             Project: ActiveMQ
          Issue Type: Bug
    Affects Versions: 5.14.3
            Reporter: sunil
         Attachments: activemq.log

Hi,

We are currently using activemq version 5.14.3 with Master/Slave/Slave architecture with JDBC Persistence adapter , LeaseDatabaseLocker and IoException handler.
Below are the configuration:

Persistence Adapter:
{code:java}
<persistenceAdapter>
 <jdbcPersistenceAdapter dataDirectory="${activemq.data}" dataSource="#mysql-ds" lockKeepAlivePeriod="3000" createTablesOnStartup="false">
 <locker>
 <lease-database-locker lockAcquireSleepInterval="20000"/>
 </locker>
 </jdbcPersistenceAdapter>
 </persistenceAdapter>{code}
IoException Handler:
{code:java}
<ioExceptionHandler>
 <leaseLockerIOExceptionHandler ignoreSQLExceptions="false" stopStartConnectors="true" systemExitOnShutdown="true"/>
 </ioExceptionHandler>
 
{code}

We had a network patition event on the master node and after it was removed from the partition eventough it stopped the transports and stopped the broker , it failed to exit the JVM .
But the new broker instance came up and it entered in to slave mode which is good. But when we looked at the logs we saw that every 5 secs:
2019-02-05 16:29:11,098 | INFO | waiting for broker persistence adapter checkpoint to succeed before restarting transports | org.apache.activemq.util.DefaultIOExceptionHandler | IOExceptionHandler: restart transports


When we took the thread dump to see what the ioExceptionHandler thread is hung at :
{code:java}
"IOExceptionHandler: restart transports@6055" daemon prio=5 tid=0x1d9 nid=NA sleeping
 java.lang.Thread.State: TIMED_WAITING
 at java.lang.Thread.sleep(Unknown Source:-1)
 at java.lang.Thread.sleep(Unknown Source:-1)
 at java.util.concurrent.TimeUnit.sleep(Unknown Source:-1)
 at org.apache.activemq.util.DefaultIOExceptionHandler$1$1.run(DefaultIOExceptionHandler.java:111){code}
It is stuck in the while loop:
{code:java}
while (hasLockOwnership() && isPersistenceAdapterDown()) {
 LOG.info("waiting for broker persistence adapter checkpoint to succeed before restarting transports");
 TimeUnit.MILLISECONDS.sleep(resumeCheckSleepPeriod);
 }{code}
The broker persistenceAdapter value is null and it fails to exit the while loop and the thread is hung there. 
What is the scenario when the persistence Adapter is set to null? and is it bug in a code since there is a stale thread running and also reference to the old brokerInstance which has the persistence Adapter set to null.

Also can anyone clarify why the systemExitOnShutdown didn't kill the JVM when the stop on broker was initiated due to ExceptionHandler code.

Any Help will be appreciated.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)