[jira] [Created] (AMQ-5435) Persistence Adapter Starting Thread is still alive after stopping a slave broker with lease database locker

["Shi Lei (JIRA)" <ji...@apache.org>]

Shi Lei created AMQ-5435:
----------------------------

             Summary: Persistence Adapter Starting Thread is still alive after stopping a slave broker with lease database locker
                 Key: AMQ-5435
                 URL: https://issues.apache.org/jira/browse/AMQ-5435
             Project: ActiveMQ
          Issue Type: Bug
          Components: Broker
    Affects Versions: 5.10.0
         Environment: Windows, JDK7
            Reporter: Shi Lei


I am using jdbc master/slave with lease database lock.
I found if I call broker.stop to stop a slave broker service (which means it tries to get a lease locker and has not got yet), its Persistence Adapter Starting Thread is still alive. If I create and start a new broker in the same java VM, there will be 2 Persistence Adapter Starting Threads inside the same java VM. At this time, if the master broker is down, the stopped broker will get the database lease locker, but somehow it cannot start broker. Now I have 2 broker service in the same VM. One has got the locker, but cannot start broker, the other one is still requesting the locker.

The root cause is that after stopping broker, LeaseDatabaseLocker.isStopping() is false, LeaseDatabaseLocker.isStopped() is true,
In LeaseDatabaseLocker.doStart

 while (!isStopping()) {
            Connection connection = null;
            PreparedStatement statement = null;
            try {
                connection = getConnection();
                initTimeDiff(connection);

                statement = connection.prepareStatement(sql);
                setQueryTimeout(statement);

                now = System.currentTimeMillis() + diffFromCurrentTime;
                statement.setString(1, getLeaseHolderId());
                statement.setLong(2, now + lockAcquireSleepInterval);
                statement.setLong(3, now);

                int result = statement.executeUpdate();
                if (result == 1) {
                    // we got the lease, verify we still have it
                    if (keepAlive()) {
                        break;
                    }
                }

                reportLeasOwnerShipAndDuration(connection);

            } catch (Exception e) {
                LOG.debug(getLeaseHolderId() + " lease acquire failure: "+ e, e);
                if (isStopping()) {
                    throw new Exception(
                            "Cannot start broker as being asked to shut down. "
                                    + "Interrupted attempt to acquire lock: "
                                    + e, e);
                }
                if (handleStartException) {
                    lockable.getBrokerService().handleIOException(IOExceptionSupport.create(e));
                }
            } finally {
                close(statement);
                close(connection);
            }

            LOG.info(getLeaseHolderId() + " failed to acquire lease.  Sleeping for " + lockAcquireSleepInterval + " milli(s) before trying again...");
            TimeUnit.MILLISECONDS.sleep(lockAcquireSleepInterval);
        }
        if (isStopping()) {
            throw new RuntimeException(getLeaseHolderId() + " failing lease acquire due to stop");
        }

        LOG.info(getLeaseHolderId() + ", becoming master with lease expiry " + new Date(now) + " on dataSource: " + dataSource);
    }

I think we should replace isStopping() with isStopping() or isStopped().




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)