DefaultIOExceptionHandler Thread hung in while loop because of broker.getPersistance is null

[sunilvp <vi...@gmail.com>]

Hi,

We are currently using activemq version 5.14.3 with Master/Slave/Slave
architecture with JDBC Persistence adapter , LeaseDatabaseLocker and
IoException handler.
Below are the configuration:

Persistence Adapter:
        <persistenceAdapter>
            <jdbcPersistenceAdapter dataDirectory="${activemq.data}"
dataSource="#mysql-ds" lockKeepAlivePeriod="3000"
createTablesOnStartup="false">
                <locker>
                    <lease-database-locker
lockAcquireSleepInterval="20000"/>
                </locker>
            </jdbcPersistenceAdapter>
        </persistenceAdapter>

IoException Handler:
        <ioExceptionHandler>
            <leaseLockerIOExceptionHandler ignoreSQLExceptions="false"
stopStartConnectors="true" systemExitOnShutdown="true"/>
        </ioExceptionHandler>


We had a network patition event on the master node and after it was removed
from the partition eventough it stopped the transports and stopped the
broker , it failed to exit the JVM .
But the new broker instance came up and it entered in to slave mode which is
good. But when we looked at the logs we saw that every 5 secs:
2019-02-05 16:29:11,098 | INFO  | waiting for broker persistence adapter
checkpoint to succeed before restarting transports |
org.apache.activemq.util.DefaultIOExceptionHandler | IOExceptionHandler:
restart transports


When we took the thread dump to see what the ioExceptionHandler thread is
hung at :
"IOExceptionHandler: restart transports@6055" daemon prio=5 tid=0x1d9 nid=NA
sleeping
  java.lang.Thread.State: TIMED_WAITING
	  at java.lang.Thread.sleep(Unknown Source:-1)
	  at java.lang.Thread.sleep(Unknown Source:-1)
	  at java.util.concurrent.TimeUnit.sleep(Unknown Source:-1)
	  at
org.apache.activemq.util.DefaultIOExceptionHandler$1$1.run(DefaultIOExceptionHandler.java:111)

It is stuck in the while loop:
 while (hasLockOwnership() && isPersistenceAdapterDown()) {
                                            LOG.info("waiting for broker
persistence adapter checkpoint to succeed before restarting transports");
                                           
TimeUnit.MILLISECONDS.sleep(resumeCheckSleepPeriod);
                                        }

The broker persistenceAdapter value is null and it fails to exit the while
loop and the thread is hung there. 
What is the scenario when the persistenceAdapter is set to null? and is it
bug in a code since there is a stale thread running and also reference to
the old brokerInstance which has the persistenceAdapter set to null. 

Also can anyone clarify why the systemExitOnShutdown didn't kill the JVM
when the stop on broker was initiated due to ExceptionHandler code. 

Attaching the log file for reference:  activemq.log
<http://activemq.2283324.n4.nabble.com/file/t378677/activemq.log>  
Any Help will be appricitated, 

Thank You






--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html