You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "amaechler (via GitHub)" <gi...@apache.org> on 2026/10/02 20:52:47 UTC
[PR] fix: re-announce a restarted service once old ZooKeeper entry expires (druid)
amaechler opened a new pull request, #20476:
URL: https://github.com/apache/druid/pull/20476
### Description
When a Druid process is killed (for example by SIGKILL after a container OOM) and restarts on the same host and port within `druid.zk.service.sessionTimeoutMs`, the ephemeral announcement node of its previous ZooKeeper session still exists. The new process's announcement `create` fails with `NODEEXISTS`, and the result was dropped because the create was fire-and-forget. Recovery relied on the announcer's cache reporting the stale node's removal once the old session expires.
`PathChildrenCache` (`CHILD_REMOVED`) and `CuratorCache` (`NODE_DELETED`) only report removals of nodes they have already loaded. If ZooKeeper deletes the stale node before the cache's first `getData`, no event fires and the announcement is never re-created. The process keeps running and can still win leader election, which uses a separate path, but stays invisible to internal discovery until it restarts. We hit this in production at: a restarted Coordinator's stale node was deleted about 30ms after it announced itself, and when that Coordinator later became leader, other services failed with `issued redirect to unknown URL`. Both `PathChildrenAnnouncer` (the default) and `NodeAnnouncer` have this window.
Related: #20339, where a discovery node that is removed and re-added right away, the reinstatement this PR relies on, blocks the Overlord from acquiring leadership. This PR neither fixes nor causes that issue.
#### Handle `NODEEXISTS` when creating an announcement
`createAnnouncement()` in both announcers now passes a background callback. On `NODEEXISTS` it sets an exists watch on the path, mirroring Curator's `PersistentNode`:
- `NONODE`: the stale node is already gone, so the announcement is re-created right away.
- The node exists: its owning session is logged, and `NodeDeleted` re-creates the announcement. `NodeDataChanged` re-arms the one-shot watch.
- Any other failure of the create or the exists check is logged as a warning instead of being dropped.
All re-create paths, including the existing cache listeners and `PathChildrenAnnouncer`'s reconnect handler, now go through `reinstateAnnouncement(path)`. It uses the current payload and skips paths that were unannounced in the meantime; the reconnect handler previously passed a null payload to the create for such paths. Each announcer shares one watcher instance, so repeated registrations on a path are deduplicated. When the cache did see the stale node, both the cache and the watch fire; the second create gets `NODEEXISTS` on our own node, which is harmless.
Alternatives considered:
- Deleting the existing node when its `ephemeralOwner` is another session: the payload includes `startTime`, so it never matches the old process's payload, and two live processes sharing a host and port could keep deleting each other's announcement.
- Switching to `PersistentNode`: it handles this case, but adopting it means reworking both announcers (compression, parent cleanup, `update()`), which is more than this fix needs.
#### Tests
New tests in `PathChildrenAnnouncerTest` and `NodeAnnouncerTest` reproduce the race deterministically, using a second ZooKeeper session that owns the stale node:
- `testReinstatesWhenStaleNodeIsRemovedBeforeCacheSeesIt` blocks the announcer client's event thread, so the stale node is gone before the exists check runs.
- `testReinstatesWhenWatchedStaleNodeIsRemoved` removes the stale node after the watch is set while the announcer's executor is blocked, so only the watch can recover the announcement.
Without the fix all four tests time out; the second one also fails when only the watcher's `NodeDeleted` branch is disabled. With the fix, each passed 25 of 25 repeated runs.
The existing tests waited for `CuratorListener` `CREATE` events, which Curator no longer delivers once a create has a callback. They now poll until the node holds the expected payload and belongs to the announcer's session, using new helpers in `CuratorTestBase`. Because `KillSession` only expires the session on the client side, `testSessionKilled` now waits about one session timeout (about 15 s) for the killed session's nodes to go away; the old latch passed on the failed `NODEEXISTS` create instead.
#### Release note
Fixed a race in ZooKeeper-based service discovery where a service that restarted within the ZooKeeper session timeout could stay unregistered until its next restart.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md)
- [x] a release note entry in the PR description.
- [x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fix: re-announce a restarted service once old ZooKeeper entry expires (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #20476:
URL: https://github.com/apache/druid/pull/20476#discussion_r4173130107
##########
server/src/main/java/org/apache/druid/curator/announcement/NodeAnnouncer.java:
##########
@@ -94,6 +96,17 @@ public class NodeAnnouncer implements ServiceAnnouncer
@GuardedBy("toAnnounce")
private final List<String> parentsIBuilt = new CopyOnWriteArrayList<>();
+ /**
+ * Watches a node that already existed when we tried to create our announcement, see {@link #watchExistingNode}.
+ */
+ private final CuratorWatcher existingNodeWatcher = event -> {
Review Comment:
[P1] Re-arm the stale-node watch after reconnect
**Finding:** This is a one-shot ZooKeeper watch tied to the announcer's current session. If that session expires after the create returns NODEEXISTS but before the old ephemeral node expires, ZooKeeper removes the watch; if the old node is then deleted while the client is disconnected, CuratorCache can still have no stored entry in exactly the cache-miss race this change targets, so reconnecting produces no deletion event and the service remains undiscoverable until another restart.
**Suggestion:** Register and remove a connection-state listener that reissues the existence watch on every reconnect, or use a managed persistent watcher that covers the session-expiration gap.
##########
server/src/main/java/org/apache/druid/curator/announcement/PathChildrenAnnouncer.java:
##########
@@ -388,7 +392,49 @@ public void update(final String path, final byte[] bytes)
private void createAnnouncement(final String path, byte[] value) throws Exception
{
- curator.create().compressed().withMode(CreateMode.EPHEMERAL).inBackground().forPath(path, value);
+ curator.create().compressed().withMode(CreateMode.EPHEMERAL).inBackground(
+ (client, event) -> {
+ if (event.getResultCode() == KeeperException.Code.NODEEXISTS.intValue()) {
+ watchExistingNode(path);
+ } else if (event.getResultCode() != KeeperException.Code.OK.intValue()) {
+ log.warn("Failed to announce node[%s]: %s", path, KeeperException.Code.get(event.getResultCode()));
+ }
+ }
+ ).forPath(path, value);
+ }
+
+ /**
+ * Watches a node that was already present when we tried to create our announcement at its path, for example one
+ * left behind by a previous session of this process. The announcement is re-created once that node is removed,
+ * even if the {@link PathChildrenCache} never saw it.
+ */
+ private void watchExistingNode(final String path) throws Exception
+ {
+ curator.checkExists().usingWatcher(existingNodeWatcher).inBackground(
+ (client, event) -> {
+ if (event.getResultCode() == KeeperException.Code.NONODE.intValue()) {
+ reinstateAnnouncement(path);
+ } else if (event.getResultCode() == KeeperException.Code.OK.intValue()) {
+ final long owner = event.getStat().getEphemeralOwner();
+ if (owner != curator.getZookeeperClient().getZooKeeper().getSessionId()) {
+ log.info("Node[%s] already exists with session[0x%x], will reinstate once it is removed.", path, owner);
+ }
+ } else {
+ log.warn("Failed to watch existing node[%s]: %s", path, KeeperException.Code.get(event.getResultCode()));
+ }
+ }
+ ).forPath(path);
+ }
+
+ private void reinstateAnnouncement(final String path) throws Exception
+ {
+ final ZKPaths.PathAndNode pathAndNode = ZKPaths.getPathAndNode(path);
+ final ConcurrentMap<String, byte[]> subPaths = announcements.get(pathAndNode.getPath());
Review Comment:
[P2] Guard reannounce against concurrent unannounce
**Finding:** The payload is read without the lifecycle lock and the create is queued afterward. If a NodeDeleted callback is already in flight while unannounce or stop removes this path, the old node can be deleted before this create runs, leaving a new ephemeral node after the caller has successfully unannounced or stopped it.
**Suggestion:** Synchronize the payload and started-state check with unannounce and stop, then revalidate immediately before issuing the create.
##########
server/src/main/java/org/apache/druid/curator/announcement/NodeAnnouncer.java:
##########
@@ -327,7 +336,47 @@ public void update(final String path, final byte[] bytes)
private void createAnnouncement(final String path, byte[] value) throws Exception
{
- curator.create().compressed().withMode(CreateMode.EPHEMERAL).inBackground().forPath(path, value);
+ curator.create().compressed().withMode(CreateMode.EPHEMERAL).inBackground(
+ (client, event) -> {
+ if (event.getResultCode() == KeeperException.Code.NODEEXISTS.intValue()) {
+ watchExistingNode(path);
+ } else if (event.getResultCode() != KeeperException.Code.OK.intValue()) {
+ log.warn("Failed to announce node[%s]: %s", path, KeeperException.Code.get(event.getResultCode()));
+ }
+ }
+ ).forPath(path, value);
+ }
+
+ /**
+ * Watches a node that was already present when we tried to create our announcement at its path, for example one
+ * left behind by a previous session of this process. The announcement is re-created once that node is removed,
+ * even if the {@link CuratorCache} never saw it.
+ */
+ private void watchExistingNode(final String path) throws Exception
+ {
+ curator.checkExists().usingWatcher(existingNodeWatcher).inBackground(
+ (client, event) -> {
+ if (event.getResultCode() == KeeperException.Code.NONODE.intValue()) {
+ reinstateAnnouncement(path);
+ } else if (event.getResultCode() == KeeperException.Code.OK.intValue()) {
+ final long owner = event.getStat().getEphemeralOwner();
+ if (owner != curator.getZookeeperClient().getZooKeeper().getSessionId()) {
+ log.info("Node[%s] already exists with session[0x%x], will reinstate once it is removed.", path, owner);
+ }
+ } else {
+ log.warn("Failed to watch existing node[%s]: %s", path, KeeperException.Code.get(event.getResultCode()));
+ }
+ }
+ ).forPath(path);
+ }
+
+ private void reinstateAnnouncement(final String path) throws Exception
+ {
+ final byte[] value = announcedPaths.get(path);
Review Comment:
[P2] Guard reannounce against concurrent unannounce
**Finding:** The payload is read without the lifecycle lock and the create is queued afterward. If a NodeDeleted callback is already in flight while unannounce or stop removes this path, the old node can be deleted before this create runs, leaving a new ephemeral node after the caller has successfully unannounced or stopped it; the identical race exists in PathChildrenAnnouncer.
**Suggestion:** Synchronize the payload and started-state check with unannounce and stop, then revalidate immediately before issuing the create in both announcers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fix: re-announce a restarted service once old ZooKeeper entry expires (druid)
Posted by "amaechler (via GitHub)" <gi...@apache.org>.
amaechler commented on code in PR #20476:
URL: https://github.com/apache/druid/pull/20476#discussion_r4174752803
##########
server/src/main/java/org/apache/druid/curator/announcement/NodeAnnouncer.java:
##########
@@ -94,6 +96,17 @@ public class NodeAnnouncer implements ServiceAnnouncer
@GuardedBy("toAnnounce")
private final List<String> parentsIBuilt = new CopyOnWriteArrayList<>();
+ /**
+ * Watches a node that already existed when we tried to create our announcement, see {@link #watchExistingNode}.
+ */
+ private final CuratorWatcher existingNodeWatcher = event -> {
Review Comment:
Fixed in 90ba8da901: `NodeAnnouncer` now registers a connection-state listener that re-announces every path on `RECONNECTED`, so the create hits `NODEEXISTS` again and the watch is set again from the new session. Added `testReinstatesWhenOwnSessionExpiresWhileWatchingStaleNode`, which fails without the listener. In practice the cache usually recovers here because it still holds the stale node after reconnecting; this only matters when the cache missed it too. `PathChildrenAnnouncer` already re-creates its paths after a session loss.
##########
server/src/main/java/org/apache/druid/curator/announcement/NodeAnnouncer.java:
##########
@@ -327,7 +336,47 @@ public void update(final String path, final byte[] bytes)
private void createAnnouncement(final String path, byte[] value) throws Exception
{
- curator.create().compressed().withMode(CreateMode.EPHEMERAL).inBackground().forPath(path, value);
+ curator.create().compressed().withMode(CreateMode.EPHEMERAL).inBackground(
+ (client, event) -> {
+ if (event.getResultCode() == KeeperException.Code.NODEEXISTS.intValue()) {
+ watchExistingNode(path);
+ } else if (event.getResultCode() != KeeperException.Code.OK.intValue()) {
+ log.warn("Failed to announce node[%s]: %s", path, KeeperException.Code.get(event.getResultCode()));
+ }
+ }
+ ).forPath(path, value);
+ }
+
+ /**
+ * Watches a node that was already present when we tried to create our announcement at its path, for example one
+ * left behind by a previous session of this process. The announcement is re-created once that node is removed,
+ * even if the {@link CuratorCache} never saw it.
+ */
+ private void watchExistingNode(final String path) throws Exception
+ {
+ curator.checkExists().usingWatcher(existingNodeWatcher).inBackground(
+ (client, event) -> {
+ if (event.getResultCode() == KeeperException.Code.NONODE.intValue()) {
+ reinstateAnnouncement(path);
+ } else if (event.getResultCode() == KeeperException.Code.OK.intValue()) {
+ final long owner = event.getStat().getEphemeralOwner();
+ if (owner != curator.getZookeeperClient().getZooKeeper().getSessionId()) {
+ log.info("Node[%s] already exists with session[0x%x], will reinstate once it is removed.", path, owner);
+ }
+ } else {
+ log.warn("Failed to watch existing node[%s]: %s", path, KeeperException.Code.get(event.getResultCode()));
+ }
+ }
+ ).forPath(path);
+ }
+
+ private void reinstateAnnouncement(final String path) throws Exception
+ {
+ final byte[] value = announcedPaths.get(path);
Review Comment:
Fixed in 90ba8da901: the create is now queued inside `computeIfPresent` on the announcement's entry, so a concurrent `unannounce` either removes the entry first (no create) or waits until the create is queued, and its delete follows the create (one session's requests are handled in order while connected). This race already existed in the cache's removal path; I reproduced it with a forced interleaving before the fix and confirmed it no longer happens after.
##########
server/src/main/java/org/apache/druid/curator/announcement/PathChildrenAnnouncer.java:
##########
@@ -388,7 +392,49 @@ public void update(final String path, final byte[] bytes)
private void createAnnouncement(final String path, byte[] value) throws Exception
{
- curator.create().compressed().withMode(CreateMode.EPHEMERAL).inBackground().forPath(path, value);
+ curator.create().compressed().withMode(CreateMode.EPHEMERAL).inBackground(
+ (client, event) -> {
+ if (event.getResultCode() == KeeperException.Code.NODEEXISTS.intValue()) {
+ watchExistingNode(path);
+ } else if (event.getResultCode() != KeeperException.Code.OK.intValue()) {
+ log.warn("Failed to announce node[%s]: %s", path, KeeperException.Code.get(event.getResultCode()));
+ }
+ }
+ ).forPath(path, value);
+ }
+
+ /**
+ * Watches a node that was already present when we tried to create our announcement at its path, for example one
+ * left behind by a previous session of this process. The announcement is re-created once that node is removed,
+ * even if the {@link PathChildrenCache} never saw it.
+ */
+ private void watchExistingNode(final String path) throws Exception
+ {
+ curator.checkExists().usingWatcher(existingNodeWatcher).inBackground(
+ (client, event) -> {
+ if (event.getResultCode() == KeeperException.Code.NONODE.intValue()) {
+ reinstateAnnouncement(path);
+ } else if (event.getResultCode() == KeeperException.Code.OK.intValue()) {
+ final long owner = event.getStat().getEphemeralOwner();
+ if (owner != curator.getZookeeperClient().getZooKeeper().getSessionId()) {
+ log.info("Node[%s] already exists with session[0x%x], will reinstate once it is removed.", path, owner);
+ }
+ } else {
+ log.warn("Failed to watch existing node[%s]: %s", path, KeeperException.Code.get(event.getResultCode()));
+ }
+ }
+ ).forPath(path);
+ }
+
+ private void reinstateAnnouncement(final String path) throws Exception
+ {
+ final ZKPaths.PathAndNode pathAndNode = ZKPaths.getPathAndNode(path);
+ final ConcurrentMap<String, byte[]> subPaths = announcements.get(pathAndNode.getPath());
Review Comment:
Same fix as in `NodeAnnouncer` (90ba8da901): the create is queued inside `computeIfPresent` on the entry, so a concurrent `unannounce` either prevents it or deletes the node afterwards.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fix: re-announce a restarted service once old ZooKeeper entry expires (druid)
Posted by "amaechler (via GitHub)" <gi...@apache.org>.
amaechler merged PR #20476:
URL: https://github.com/apache/druid/pull/20476
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org