[jira] [Commented] (AMQ-6865) JDBC Master/Slave: Master did not leave lock when the master loose database connection.

["Rohan Mars (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AMQ-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16418074#comment-16418074 ] 

Rohan Mars commented on AMQ-6865:
---------------------------------

I've been able to easily reproduce this error on 5.15.3. Use a packet loss strategy between amq and db and the problem will appear instantly.
{code:java}
 tc qdisc add dev enp0s9 root netem loss 100%
{code}
 

1) You need to configure the mysql jdbc connection string to timeout before the lease check interval does. The default jdbc mysql connection doesn't timeout, which equivalent timeout settings will be required for any alternative database connection string.
{code:java}
jdbc:mysql://dbhost/activemq?relaxAutoCommit=true&useSSL=false&useJDBCCompliantTimezoneShift=true&connectTimeout=3000&socketTimeout=3000"{code}
 

2) The use of the LeaseLockerIOExceptionHandler is not enabled by default when you use a lease-database-locker, major documentation gap. You need to declare the bean and inject it. Additionally, you need to set back StopStartConnectors to false, so a full restart of the broker occurs (which if it can't reconnect will actually shut down the process). I wasn't convinced that keeping it as "true" property shuts down the broker and/or cleans up it's internal state to be a slave mode trying to reconnect, which only occurs in the logs when you restore the packet loss after the original salve has become master.

 
{code:java}
<bean id="ioExceptionHandler" class="org.apache.activemq.util.LeaseLockerIOExceptionHandler">
     <property name="stopStartConnectors"><value>false</value></property>
</bean>

...

<broker xmlns="http://activemq.apache.org/schema/core" brokerName="amq1" dataDirectory="${activemq.data}" ioExceptionHandler="#ioExceptionHandler">

....
{code}
Ideally, ActiveMQ would support starting up in a "retrying warm mode" when the persistence adapter is using JDBC by going in a loop trying to startup, i.e. connecting to the database. Shutting down the whole process requires some additional operational overhead. But it appears that would not be a straight forward change based on the current code. 

 

> JDBC Master/Slave: Master did not leave lock when the master loose database connection.
> ---------------------------------------------------------------------------------------
>
>                 Key: AMQ-6865
>                 URL: https://issues.apache.org/jira/browse/AMQ-6865
>             Project: ActiveMQ
>          Issue Type: Bug
>            Reporter: Pranjal Bathia
>            Priority: Major
>
> We have JBoss AMQ instance of  jboss-a-mq-6.3.0.redhat-187, and we use JDBC for master/slave configuration. 
> As per our observation, during networking disruption for connecting to the database, a broker was unable to recover cleanly. 
> In hawtio, both master/slave was showing as master, and ActiveMQ tab with amq connection was available on both broker and which resulted in a situation where one app was sending messages to broker01 (which was acting as producer) and all consumers were connected to broker02. 
> We have below setting :
> <persistenceAdapter>
>              <jdbcPersistenceAdapter dataSource="#mysql-ds" lockKeepAlivePeriod="5000">
>                 <locker>
>                    <lease-database-locker lockAcquireSleepInterval="10000"/>
>                 </locker>
>              </jdbcPersistenceAdapter>
>   </persistenceAdapter>
> We had seen this issue in production, we did not get a chance to look at an activemq_lock table.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)