[jira] [Updated] (ARTEMIS-6159) A lock coordinator with an unreachable ZooKeeper hangs broker startup and shutdown indefinitely

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARTEMIS-6159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated ARTEMIS-6159:
------------------------------------
    Labels: pull-request-available  (was: )

> A lock coordinator with an unreachable ZooKeeper hangs broker startup and shutdown indefinitely 
> ------------------------------------------------------------------------------------------------
>
>                 Key: ARTEMIS-6159
>                 URL: https://issues.apache.org/jira/browse/ARTEMIS-6159
>             Project: Artemis
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 2.51.0, 2.52.0, 2.54.0, 2.53.0, 2.55.0
>            Reporter: Michael Lukyanov
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> h3. Summary
>   When a broker is configured with a ZooKeeper-based lock coordinator
>   ({{CuratorDistributedLockManager}}) and ZooKeeper is unreachable, the lock
>   coordinator's periodic task blocks forever trying to connect. This occupies
>   the coordinator's ordered-executor thread permanently, and broker
>   stop/restart hangs indefinitely as a result. The broker becomes
>   unmanageable and the log fills with an endless ZooKeeper reconnect loop.
>   Also reproduces when the lock coordinator is defined but not referenced by
>   any acceptor or broker-connection - every configured {{<lock-coordinator>}}
>   gets started unconditionally, so an unused one can hang the broker just as
>   well.
>   h3. Root cause
>   Confirmed against {{main}} (2.56.0-SNAPSHOT), present since 2.51.0.
>   {{LockCoordinator.run()}} starts the manager with the unbounded overload,
>   {{lockManager.start()}} (LockCoordinator.java:316). {{CuratorDistributedLockManager.start()}}
>   delegates to {{start(-1, null)}} (line 260), which calls
>   {{client.blockUntilConnected((int) timeout, unit)}} with {{unit == null}}
>   (line 244). With a null {{TimeUnit}}, Curator's {{blockUntilConnected}}
>   waits indefinitely - even though Curator itself keeps retrying in the
>   background, hence the endless "Opening socket connection.../SASL
>   configuration failed/Authentication failed" loop in the logs.
>   {{LockCoordinator}} extends {{ActiveMQScheduledComponent}} and its scheduled
>   task runs on the coordinator's own ordered executor ({{startLockCoordinators()}}
>   does {{new LockCoordinator(scheduledPool, executorFactory.getExecutor(), ...)}}).
>   Once {{run()}} is stuck in {{blockUntilConnected}}, that executor thread is
>   gone for good.
>   Then on stop/restart, {{LockCoordinator.stop()}} (line 168) queues its
>   cleanup on the same ordered executor and blocks on {{simpleFuture.get()}}
>   with no timeout (line 197). That cleanup task never gets to run, so
>   {{ActiveMQServerImpl.stopLockCoordinators()}} -> {{LockCoordinator.stop()}}
>   never returns and the broker can't be stopped or restarted.
>   Separately, {{ActiveMQServerImpl.startLockCoordinators()}} (lines 824-844)
>   starts every configured coordinator regardless of whether anything
>   references it via {{lock-coordinator="..."}}. Not the cause of the hang by
>   itself, but it means a coordinator nobody uses can still take the broker
>   down.
>   h3. Steps to reproduce
>   Configure a coordinator pointing at an address nothing is listening on:
>   {code:xml}
>   <lock-coordinators>
>      <lock-coordinator name="zk">
>         <class-name>org.apache.activemq.artemis.lockmanager.zookeeper.CuratorDistributedLockManager</class-name>
>         <properties>
>            <property key="connect-string" value="localhost:2181"/>
>            <property key="namespace" value="test"/>
>         </properties>
>      </lock-coordinator>
>   </lock-coordinators>
>   {code}
>   Don't start ZooKeeper, start the broker, then try to stop or restart it. It
>   doesn't come back. Assigning the coordinator to an acceptor isn't necessary
>   to trigger this - an unreferenced one hangs the broker just the same.
>   h3. Fix
>   I wrote two tests to pin this down before touching anything: one in
>   artemis-server with a fake {{DistributedLockManager}} whose {{start()}}
>   just blocks (same shape as Curator against a dead ensemble), asserting
>   {{LockCoordinator.stop()}} returns; one in integration-tests with a real
>   {{CuratorDistributedLockManager}} pointed at a closed port and the
>   coordinator left unassigned, asserting {{server.stop()}} returns. Both hang
>   and fail on {{main}} as-is (~31s and ~72s respectively before the assertion
>   gives up), and both pass once the fix below is applied (under a second, and
>   a couple seconds).
>   The fix is just bounding the start call in {{LockCoordinator.run()}}:
>   {code:java}
>   if (!lockManager.isStarted()) {
>      if (!lockManager.start(getPeriod(), getTimeUnit())) {
>         logger.debug("Not able to start the lock manager for {}, lockID={}", name, lockID);
>         return;
>      }
>   }
>   {code}
>   {{getPeriod()}}/{{getTimeUnit()}} are just the coordinator's own
>   {{checkPeriod}}, inherited from {{ActiveMQScheduledComponent}} - so the
>   connect attempt is bounded to one scheduling tick, and on timeout it's
>   retried on the next one at the same cadence it already runs at. No new
>   config knob needed.
>   Turns out that's all that's needed - I didn't touch {{stop()}}. Once
>   {{run()}} can't block forever, the cleanup task queued by {{stop()}} gets
>   its turn on the executor within one {{checkPeriod}}, so
>   {{simpleFuture.get()}} returns on its own without a separate timeout. Both
>   tests above confirm this.
>   Left the unreferenced-coordinator behavior alone since it's not needed to
>   fix the hang - up to whoever picks this up whether it's worth addressing
>   separately.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@artemis.apache.org
For additional commands, e-mail: issues-help@artemis.apache.org