You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "Denovo1998 (via GitHub)" <gi...@apache.org> on 2026/10/09 13:19:53 UTC

Re: [PR] [fix][metadata] Deliver the session events on the oxia metadata store [pulsar]

Denovo1998 commented on code in PR #26712:
URL: https://github.com/apache/pulsar/pull/26712#discussion_r4230507187


##########
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/oxia/OxiaSessionWatcher.java:
##########
@@ -0,0 +1,401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.metadata.impl.oxia;
+
+import io.netty.util.concurrent.DefaultThreadFactory;
+import io.oxia.client.api.AsyncOxiaClient;
+import io.oxia.client.api.Notification;
+import io.oxia.client.api.options.PutOption;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.Executors;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import lombok.CustomLog;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+/**
+ * Delivers the metadata store session events on the oxia backend by watching a private ephemeral
+ * canary record.
+ *
+ * <p>Oxia does not expose session lifecycle events: the session machinery is deliberately kept
+ * internal to the client, and the only public observation surface is the notification stream.
+ * The per-record {@code KeyDeleted} notification that the server emits for every ephemeral record
+ * swept by a session expiry is therefore the intended way for an embedding application to learn
+ * about a session loss. This watcher applies exactly that mechanism: each store instance owns a
+ * canary record, under the reserved {@link #CANARY_KEY_PREFIX} namespace, that nobody but the
+ * session itself deletes. The deletion of the canary is a server-authoritative signal that the
+ * session died, and maps to {@link SessionEvent#SessionLost}; a subsequently successful canary
+ * write means a fresh session was established and maps to
+ * {@link SessionEvent#SessionReestablished}.
+ *
+ * <p>Each loss moves to a fresh incarnation of the canary key, a monotonically increasing
+ * generation suffixed to the instance-unique prefix. Notifications and write completions are only
+ * honored for the current incarnation: a {@code KeyDeleted} of an expired incarnation delivered
+ * late — the notification stream resumes from its last offset after a reconnect — cannot flip a
+ * live session to lost, and the completion of a write issued before a loss cannot report a
+ * recovery. Recovery is only ever reported by a write issued after the loss.
+ *
+ * <p>While the session is considered established, a periodic existence check of the canary backs
+ * the notification stream up: a deletion whose notification was lost, or that was delivered as a
+ * range delete, is detected by the next check instead of being missed forever. The same check
+ * monitors connectivity: the first failed check reports {@link SessionEvent#ConnectionLost},
+ * check failures spanning a full session timeout report {@link SessionEvent#SessionLost}, and a
+ * successful check after a connection loss reports {@link SessionEvent#Reconnected}. A failing
+ * check counts as no answer whatever its cause; the window is the session timeout because that is
+ * the window after which the server reaps the session — it is the timeout the client sends in
+ * {@code CreateSessionRequest} — and after which the oxia client itself expires the session. If
+ * connectivity returns within the window and the session survived, the pessimistic SessionLost is
+ * followed by a prompt SessionReestablished once the canary is rewritten: the same trade-off
+ * {@code ZKSessionWatcher} makes at its monitor timeout.
+ *
+ * <p>No ordering is guaranteed between the synthesized session events and the per-key
+ * notifications of the same sweep: a notification batch is a map on the wire and carries no
+ * per-entry order, so consumers must not rely on SessionLost preceding the Deleted notifications
+ * of the records swept with the canary.
+ *
+ * <p>Known limitation: the oxia client keeps one session per shard and the canary lives on a
+ * single shard, so a session loss confined to a different shard — for example the clean session
+ * close of a shard rebalance, which sweeps that shard's ephemeral records without an expiry — is
+ * not reported by this watcher. The common loss of the session, a server restart or outage, kills
+ * the sessions of all shards together with the canary and is fully covered.
+ */
+@CustomLog
+final class OxiaSessionWatcher implements AutoCloseable {
+
+    /**
+     * Reserved namespace of the session canary records. It must not be used for anything else and
+     * it must keep sorting before every {@code '/'}-rooted path, so that the canaries stay
+     * outside the {@code '/'} hierarchy the store exposes to ordinary listings.
+     */
+    static final String CANARY_KEY_PREFIX = ".pulsar-oxia-session-canary-";
+
+    /** Fallback session timeout for store instances built directly around a client. */
+    static final long DEFAULT_SESSION_TIMEOUT_MILLIS = TimeUnit.SECONDS.toMillis(30);
+
+    private static final byte[] CANARY_VALUE = new byte[0];
+    private static final long MAX_CHECK_INTERVAL_MILLIS = TimeUnit.SECONDS.toMillis(30);
+    private static final long MIN_CHECK_INTERVAL_MILLIS = TimeUnit.SECONDS.toMillis(1);
+    private static final long WRITE_RETRY_MIN_DELAY_MILLIS = TimeUnit.SECONDS.toMillis(1);
+    private static final long WRITE_RETRY_MAX_DELAY_MILLIS = TimeUnit.SECONDS.toMillis(8);
+
+    private final AsyncOxiaClient client;
+    private final Consumer<SessionEvent> eventSink;
+
+    /** Instance-unique prefix of the canary keys; the generation of the moment is appended. */
+    private final String canaryKeyPrefix;
+
+    private final long sessionTimeoutMillis;
+    private final long checkIntervalMillis;
+
+    private final ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor(
+            new DefaultThreadFactory("oxia-metadata-session-watcher"));
+
+    /**
+     * Set while a canary write is scheduled or in flight, so that at most one write chain exists
+     * at any time.
+     */
+    private final AtomicBoolean canaryWritePending = new AtomicBoolean();
+
+    private final Object stateLock = new Object();
+    private State state = State.STARTING;
+
+    /**
+     * Generation of the current canary incarnation and its key. Guarded by {@link #stateLock};
+     * the key is additionally readable without the lock for notification matching, where a stale
+     * read can only funnel into a loss transition that a newer generation already made a no-op.
+     */
+    private int generation;
+    private volatile String canaryKey;
+
+    /**
+     * Set under {@link #stateLock} when the watcher closes, so that no state transition — and
+     * with it no listener dispatch — can interleave with the close. Volatile for the executor
+     * fast paths that do not dispatch.
+     */
+    private volatile boolean closed;
+
+    /** Retry counter of the current canary write chain; only accessed on the executor. */
+    private int writeFailures;
+
+    /**
+     * When the current streak of failing existence checks began, or zero outside a streak. Only
+     * accessed on the executor.
+     */
+    private long checkFailedSinceMillis;
+
+    private enum State {
+        STARTING, ESTABLISHED, DISCONNECTED, LOST
+    }
+
+    OxiaSessionWatcher(AsyncOxiaClient client, Consumer<SessionEvent> eventSink,
+            long sessionTimeoutMillis) {
+        this.client = client;
+        this.eventSink = eventSink;
+        this.canaryKeyPrefix = CANARY_KEY_PREFIX + UUID.randomUUID() + "-";
+        this.sessionTimeoutMillis = sessionTimeoutMillis;
+        this.checkIntervalMillis = Math.max(MIN_CHECK_INTERVAL_MILLIS,
+                Math.min(MAX_CHECK_INTERVAL_MILLIS, sessionTimeoutMillis / 3));
+        this.canaryKey = canaryKeyPrefix + generation;
+    }
+
+    /**
+     * Whether the notification belongs to the reserved canary namespace and must not be forwarded
+     * as an ordinary metadata store notification.
+     */
+    static boolean isCanaryKey(Notification notification) {
+        return notification.key().startsWith(CANARY_KEY_PREFIX);
+    }
+
+    void start() {
+        ensureCanaryWrite();
+        try {
+            executor.scheduleWithFixedDelay(this::checkCanarySafely, checkIntervalMillis,
+                    checkIntervalMillis, TimeUnit.MILLISECONDS);
+        } catch (RejectedExecutionException ignore) {
+            // The watcher was closed concurrently with the store construction.
+        }
+    }
+
+    /**
+     * Routes a notification from the canary namespace: only the deletion of the canary of the
+     * current incarnation means the session was lost, every other canary notification is ignored.
+     */
+    void handleNotification(Notification notification) {
+        if (closed) {
+            return;
+        }
+        if (notification instanceof Notification.KeyDeleted && notification.key().equals(canaryKey)) {
+            onCanaryGone("deleted");
+        }
+    }

Review Comment:
   There's a small TOCTOU race in the canary generation check.
   
   We compare the notification key with the volatile `canaryKey` outside `stateLock`, but `onCanaryGone()` doesn't verify that the deletion still belongs to the current generation.
   
   A notification thread can match generation N, pause, and then enter `onCanaryGone()` after another thread has already reported the loss and recreated generation N+1. Since the state is `ESTABLISHED` again, the old deletion can trigger another false `SessionLost`.
   
   Could we pass the expected canary key or generation into `onCanaryGone()` and validate it under the same lock as the state transition?



##########
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/oxia/OxiaMetadataStore.java:
##########
@@ -98,17 +118,34 @@ public OxiaMetadataStore(
             oxiaClientBuilder.loadConfig(metadataStoreConfig.getConfigFilePath());
         }
         client = oxiaClientBuilder.asyncClient().get();
+        this.sessionWatcher = enableSessionWatcher
+                ? new OxiaSessionWatcher(client, this::receivedSessionEvent,
+                        metadataStoreConfig.getSessionTimeoutMillis())
+                : null;

Review Comment:
   The watcher and the Oxia client can end up using different session timeouts when a config file is provided.
   
   `loadConfig()` runs after `builder.sessionTimeout(...)` and can override that builder value, but the watcher still receives the original timeout from `MetadataStoreConfig`.
   
   For example, the client may use 60s while the watcher uses 30s, which breaks the assumption that the watcher's timeout matches the real Oxia session expiry.
   
   Could we resolve one effective timeout and use it for both?



##########
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/oxia/OxiaSessionWatcher.java:
##########
@@ -0,0 +1,401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.metadata.impl.oxia;
+
+import io.netty.util.concurrent.DefaultThreadFactory;
+import io.oxia.client.api.AsyncOxiaClient;
+import io.oxia.client.api.Notification;
+import io.oxia.client.api.options.PutOption;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.Executors;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import lombok.CustomLog;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+/**
+ * Delivers the metadata store session events on the oxia backend by watching a private ephemeral
+ * canary record.
+ *
+ * <p>Oxia does not expose session lifecycle events: the session machinery is deliberately kept
+ * internal to the client, and the only public observation surface is the notification stream.
+ * The per-record {@code KeyDeleted} notification that the server emits for every ephemeral record
+ * swept by a session expiry is therefore the intended way for an embedding application to learn
+ * about a session loss. This watcher applies exactly that mechanism: each store instance owns a
+ * canary record, under the reserved {@link #CANARY_KEY_PREFIX} namespace, that nobody but the
+ * session itself deletes. The deletion of the canary is a server-authoritative signal that the
+ * session died, and maps to {@link SessionEvent#SessionLost}; a subsequently successful canary
+ * write means a fresh session was established and maps to
+ * {@link SessionEvent#SessionReestablished}.
+ *
+ * <p>Each loss moves to a fresh incarnation of the canary key, a monotonically increasing
+ * generation suffixed to the instance-unique prefix. Notifications and write completions are only
+ * honored for the current incarnation: a {@code KeyDeleted} of an expired incarnation delivered
+ * late — the notification stream resumes from its last offset after a reconnect — cannot flip a
+ * live session to lost, and the completion of a write issued before a loss cannot report a
+ * recovery. Recovery is only ever reported by a write issued after the loss.
+ *
+ * <p>While the session is considered established, a periodic existence check of the canary backs
+ * the notification stream up: a deletion whose notification was lost, or that was delivered as a
+ * range delete, is detected by the next check instead of being missed forever. The same check
+ * monitors connectivity: the first failed check reports {@link SessionEvent#ConnectionLost},
+ * check failures spanning a full session timeout report {@link SessionEvent#SessionLost}, and a
+ * successful check after a connection loss reports {@link SessionEvent#Reconnected}. A failing
+ * check counts as no answer whatever its cause; the window is the session timeout because that is
+ * the window after which the server reaps the session — it is the timeout the client sends in
+ * {@code CreateSessionRequest} — and after which the oxia client itself expires the session. If
+ * connectivity returns within the window and the session survived, the pessimistic SessionLost is
+ * followed by a prompt SessionReestablished once the canary is rewritten: the same trade-off
+ * {@code ZKSessionWatcher} makes at its monitor timeout.
+ *
+ * <p>No ordering is guaranteed between the synthesized session events and the per-key
+ * notifications of the same sweep: a notification batch is a map on the wire and carries no
+ * per-entry order, so consumers must not rely on SessionLost preceding the Deleted notifications
+ * of the records swept with the canary.
+ *
+ * <p>Known limitation: the oxia client keeps one session per shard and the canary lives on a
+ * single shard, so a session loss confined to a different shard — for example the clean session
+ * close of a shard rebalance, which sweeps that shard's ephemeral records without an expiry — is
+ * not reported by this watcher. The common loss of the session, a server restart or outage, kills
+ * the sessions of all shards together with the canary and is fully covered.
+ */
+@CustomLog
+final class OxiaSessionWatcher implements AutoCloseable {
+
+    /**
+     * Reserved namespace of the session canary records. It must not be used for anything else and
+     * it must keep sorting before every {@code '/'}-rooted path, so that the canaries stay
+     * outside the {@code '/'} hierarchy the store exposes to ordinary listings.
+     */
+    static final String CANARY_KEY_PREFIX = ".pulsar-oxia-session-canary-";
+
+    /** Fallback session timeout for store instances built directly around a client. */
+    static final long DEFAULT_SESSION_TIMEOUT_MILLIS = TimeUnit.SECONDS.toMillis(30);
+
+    private static final byte[] CANARY_VALUE = new byte[0];
+    private static final long MAX_CHECK_INTERVAL_MILLIS = TimeUnit.SECONDS.toMillis(30);
+    private static final long MIN_CHECK_INTERVAL_MILLIS = TimeUnit.SECONDS.toMillis(1);
+    private static final long WRITE_RETRY_MIN_DELAY_MILLIS = TimeUnit.SECONDS.toMillis(1);
+    private static final long WRITE_RETRY_MAX_DELAY_MILLIS = TimeUnit.SECONDS.toMillis(8);
+
+    private final AsyncOxiaClient client;
+    private final Consumer<SessionEvent> eventSink;
+
+    /** Instance-unique prefix of the canary keys; the generation of the moment is appended. */
+    private final String canaryKeyPrefix;
+
+    private final long sessionTimeoutMillis;
+    private final long checkIntervalMillis;
+
+    private final ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor(
+            new DefaultThreadFactory("oxia-metadata-session-watcher"));
+
+    /**
+     * Set while a canary write is scheduled or in flight, so that at most one write chain exists
+     * at any time.
+     */
+    private final AtomicBoolean canaryWritePending = new AtomicBoolean();
+
+    private final Object stateLock = new Object();
+    private State state = State.STARTING;
+
+    /**
+     * Generation of the current canary incarnation and its key. Guarded by {@link #stateLock};
+     * the key is additionally readable without the lock for notification matching, where a stale
+     * read can only funnel into a loss transition that a newer generation already made a no-op.
+     */
+    private int generation;
+    private volatile String canaryKey;
+
+    /**
+     * Set under {@link #stateLock} when the watcher closes, so that no state transition — and
+     * with it no listener dispatch — can interleave with the close. Volatile for the executor
+     * fast paths that do not dispatch.
+     */
+    private volatile boolean closed;
+
+    /** Retry counter of the current canary write chain; only accessed on the executor. */
+    private int writeFailures;
+
+    /**
+     * When the current streak of failing existence checks began, or zero outside a streak. Only
+     * accessed on the executor.
+     */
+    private long checkFailedSinceMillis;
+
+    private enum State {
+        STARTING, ESTABLISHED, DISCONNECTED, LOST
+    }
+
+    OxiaSessionWatcher(AsyncOxiaClient client, Consumer<SessionEvent> eventSink,
+            long sessionTimeoutMillis) {
+        this.client = client;
+        this.eventSink = eventSink;
+        this.canaryKeyPrefix = CANARY_KEY_PREFIX + UUID.randomUUID() + "-";
+        this.sessionTimeoutMillis = sessionTimeoutMillis;
+        this.checkIntervalMillis = Math.max(MIN_CHECK_INTERVAL_MILLIS,
+                Math.min(MAX_CHECK_INTERVAL_MILLIS, sessionTimeoutMillis / 3));
+        this.canaryKey = canaryKeyPrefix + generation;
+    }
+
+    /**
+     * Whether the notification belongs to the reserved canary namespace and must not be forwarded
+     * as an ordinary metadata store notification.
+     */
+    static boolean isCanaryKey(Notification notification) {
+        return notification.key().startsWith(CANARY_KEY_PREFIX);
+    }
+
+    void start() {
+        ensureCanaryWrite();
+        try {
+            executor.scheduleWithFixedDelay(this::checkCanarySafely, checkIntervalMillis,
+                    checkIntervalMillis, TimeUnit.MILLISECONDS);
+        } catch (RejectedExecutionException ignore) {
+            // The watcher was closed concurrently with the store construction.
+        }
+    }
+
+    /**
+     * Routes a notification from the canary namespace: only the deletion of the canary of the
+     * current incarnation means the session was lost, every other canary notification is ignored.
+     */
+    void handleNotification(Notification notification) {
+        if (closed) {
+            return;
+        }
+        if (notification instanceof Notification.KeyDeleted && notification.key().equals(canaryKey)) {
+            onCanaryGone("deleted");
+        }
+    }
+
+    private void checkCanarySafely() {
+        try {
+            checkCanary();
+        } catch (Throwable t) {
+            log.warn().exception(t).log("Unexpected failure while checking the oxia session canary");
+        }
+    }
+
+    /**
+     * Existence check of the canary, backing the notification stream up while the session is
+     * considered established and monitoring the connectivity to the server. Package-private so
+     * that tests can drive it directly.
+     */
+    void checkCanary() {
+        String keyAtIssue;
+        int generationAtIssue;
+        synchronized (stateLock) {
+            if (state != State.ESTABLISHED && state != State.DISCONNECTED) {
+                return;
+            }
+            keyAtIssue = canaryKey;
+            generationAtIssue = generation;
+        }
+        try {
+            client.get(keyAtIssue).whenComplete((result, ex) -> hop(() -> {
+                if (ex != null) {
+                    onCheckFailed(ex);
+                } else if (result != null) {
+                    onCheckSucceeded();
+                } else {
+                    boolean staleRead;
+                    synchronized (stateLock) {
+                        staleRead = closed || generation != generationAtIssue;
+                    }
+                    // A read of an expired incarnation finds its key missing, but that loss was
+                    // already reported when the incarnation was replaced.
+                    if (!staleRead) {
+                        onCanaryGone("missing");
+                    }
+                }
+            }));
+        } catch (RejectedExecutionException | IllegalStateException ignore) {
+            // The client was closed concurrently.
+        }
+    }
+
+    private void onCheckFailed(Throwable ex) {
+        long now = System.currentTimeMillis();
+        boolean sessionTimedOut;
+        synchronized (stateLock) {
+            if (closed || state == State.LOST || state == State.STARTING) {
+                return;
+            }
+            if (state == State.ESTABLISHED) {
+                state = State.DISCONNECTED;
+                checkFailedSinceMillis = now;
+                log.warn().exception(ex).log("Oxia metadata store connection was lost");
+                eventSink.accept(SessionEvent.ConnectionLost);
+            }
+            sessionTimedOut = now - checkFailedSinceMillis >= sessionTimeoutMillis;
+        }
+        if (sessionTimedOut) {
+            // The checks could not reach the server for a full session timeout, by which point
+            // the server has reaped the session. Report the loss now instead of waiting for the
+            // connectivity to return and confirm it.
+            onCanaryGone("unreachable");
+        }
+    }
+
+    private void onCheckSucceeded() {
+        synchronized (stateLock) {
+            if (closed || state != State.DISCONNECTED) {
+                return;
+            }
+            state = State.ESTABLISHED;
+            checkFailedSinceMillis = 0L;
+            log.info().log("Oxia metadata store connection was re-established");
+            eventSink.accept(SessionEvent.Reconnected);
+        }
+    }
+
+    private void onCanaryGone(String cause) {
+        // The events are dispatched while holding the lock, so that the listeners observe them
+        // in exactly the order of the state transitions, and so that no transition can inter
+        // leave with the close. The dispatch is non-blocking.
+        synchronized (stateLock) {
+            if (closed || state == State.LOST) {
+                return;
+            }
+            state = State.LOST;
+            generation++;
+            canaryKey = canaryKeyPrefix + generation;
+            log.info().attr("canaryKey", canaryKey).attr("cause", cause)
+                    .log("Oxia metadata store session was lost");
+            eventSink.accept(SessionEvent.SessionLost);
+        }
+        ensureCanaryWrite();
+    }
+
+    /**
+     * Arms the canary write if no write chain is already scheduled or in flight: the first write
+     * of a chain retries with backoff until it succeeds, so re-arming is only needed when the
+     * chain came to rest on a successfully written canary.
+     */
+    private void ensureCanaryWrite() {
+        if (canaryWritePending.compareAndSet(false, true)) {
+            scheduleCanaryWrite(0, true);
+        }
+    }
+
+    private void scheduleCanaryWrite(long delayMillis, boolean resetFailures) {
+        try {
+            executor.schedule(() -> writeCanary(resetFailures), delayMillis, TimeUnit.MILLISECONDS);
+        } catch (RejectedExecutionException ignore) {
+            canaryWritePending.set(false);
+        }
+    }
+
+    private void writeCanary(boolean resetFailures) {
+        if (closed) {
+            canaryWritePending.set(false);
+            return;
+        }
+        if (resetFailures) {
+            writeFailures = 0;
+        }
+        String keyAtIssue;
+        int generationAtIssue;
+        synchronized (stateLock) {
+            keyAtIssue = canaryKey;
+            generationAtIssue = generation;
+        }

Review Comment:
   I think there's a shard-routing issue with the canary generations.
   
   Oxia 0.9.5 routes a put using the hash of the record key unless a `PartitionKey` is provided. Since we append the generation to the canary key, consecutive incarnations can end up on different shards.
   
   For example, generation N can be deleted when its shard session expires, but the next canary write might succeed on another shard. We'd then emit `SessionReestablished` even though the original shard's session hasn't been recreated.
   
   Could we use a stable `PartitionKey` (e.g. `canaryKeyPrefix`) for both the canary put and existence check? That would keep the generation-based stale notification protection while making sure all incarnations target the same shard.



##########
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/oxia/OxiaSessionWatcher.java:
##########
@@ -0,0 +1,401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.metadata.impl.oxia;
+
+import io.netty.util.concurrent.DefaultThreadFactory;
+import io.oxia.client.api.AsyncOxiaClient;
+import io.oxia.client.api.Notification;
+import io.oxia.client.api.options.PutOption;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.Executors;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import lombok.CustomLog;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+/**
+ * Delivers the metadata store session events on the oxia backend by watching a private ephemeral
+ * canary record.
+ *
+ * <p>Oxia does not expose session lifecycle events: the session machinery is deliberately kept
+ * internal to the client, and the only public observation surface is the notification stream.
+ * The per-record {@code KeyDeleted} notification that the server emits for every ephemeral record
+ * swept by a session expiry is therefore the intended way for an embedding application to learn
+ * about a session loss. This watcher applies exactly that mechanism: each store instance owns a
+ * canary record, under the reserved {@link #CANARY_KEY_PREFIX} namespace, that nobody but the
+ * session itself deletes. The deletion of the canary is a server-authoritative signal that the
+ * session died, and maps to {@link SessionEvent#SessionLost}; a subsequently successful canary
+ * write means a fresh session was established and maps to
+ * {@link SessionEvent#SessionReestablished}.
+ *
+ * <p>Each loss moves to a fresh incarnation of the canary key, a monotonically increasing
+ * generation suffixed to the instance-unique prefix. Notifications and write completions are only
+ * honored for the current incarnation: a {@code KeyDeleted} of an expired incarnation delivered
+ * late — the notification stream resumes from its last offset after a reconnect — cannot flip a
+ * live session to lost, and the completion of a write issued before a loss cannot report a
+ * recovery. Recovery is only ever reported by a write issued after the loss.
+ *
+ * <p>While the session is considered established, a periodic existence check of the canary backs
+ * the notification stream up: a deletion whose notification was lost, or that was delivered as a
+ * range delete, is detected by the next check instead of being missed forever. The same check
+ * monitors connectivity: the first failed check reports {@link SessionEvent#ConnectionLost},
+ * check failures spanning a full session timeout report {@link SessionEvent#SessionLost}, and a
+ * successful check after a connection loss reports {@link SessionEvent#Reconnected}. A failing
+ * check counts as no answer whatever its cause; the window is the session timeout because that is
+ * the window after which the server reaps the session — it is the timeout the client sends in
+ * {@code CreateSessionRequest} — and after which the oxia client itself expires the session. If
+ * connectivity returns within the window and the session survived, the pessimistic SessionLost is
+ * followed by a prompt SessionReestablished once the canary is rewritten: the same trade-off
+ * {@code ZKSessionWatcher} makes at its monitor timeout.
+ *
+ * <p>No ordering is guaranteed between the synthesized session events and the per-key
+ * notifications of the same sweep: a notification batch is a map on the wire and carries no
+ * per-entry order, so consumers must not rely on SessionLost preceding the Deleted notifications
+ * of the records swept with the canary.
+ *
+ * <p>Known limitation: the oxia client keeps one session per shard and the canary lives on a
+ * single shard, so a session loss confined to a different shard — for example the clean session
+ * close of a shard rebalance, which sweeps that shard's ephemeral records without an expiry — is
+ * not reported by this watcher. The common loss of the session, a server restart or outage, kills
+ * the sessions of all shards together with the canary and is fully covered.
+ */
+@CustomLog
+final class OxiaSessionWatcher implements AutoCloseable {
+
+    /**
+     * Reserved namespace of the session canary records. It must not be used for anything else and
+     * it must keep sorting before every {@code '/'}-rooted path, so that the canaries stay
+     * outside the {@code '/'} hierarchy the store exposes to ordinary listings.
+     */
+    static final String CANARY_KEY_PREFIX = ".pulsar-oxia-session-canary-";
+
+    /** Fallback session timeout for store instances built directly around a client. */
+    static final long DEFAULT_SESSION_TIMEOUT_MILLIS = TimeUnit.SECONDS.toMillis(30);
+
+    private static final byte[] CANARY_VALUE = new byte[0];
+    private static final long MAX_CHECK_INTERVAL_MILLIS = TimeUnit.SECONDS.toMillis(30);
+    private static final long MIN_CHECK_INTERVAL_MILLIS = TimeUnit.SECONDS.toMillis(1);
+    private static final long WRITE_RETRY_MIN_DELAY_MILLIS = TimeUnit.SECONDS.toMillis(1);
+    private static final long WRITE_RETRY_MAX_DELAY_MILLIS = TimeUnit.SECONDS.toMillis(8);
+
+    private final AsyncOxiaClient client;
+    private final Consumer<SessionEvent> eventSink;
+
+    /** Instance-unique prefix of the canary keys; the generation of the moment is appended. */
+    private final String canaryKeyPrefix;
+
+    private final long sessionTimeoutMillis;
+    private final long checkIntervalMillis;
+
+    private final ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor(
+            new DefaultThreadFactory("oxia-metadata-session-watcher"));
+
+    /**
+     * Set while a canary write is scheduled or in flight, so that at most one write chain exists
+     * at any time.
+     */
+    private final AtomicBoolean canaryWritePending = new AtomicBoolean();
+
+    private final Object stateLock = new Object();
+    private State state = State.STARTING;
+
+    /**
+     * Generation of the current canary incarnation and its key. Guarded by {@link #stateLock};
+     * the key is additionally readable without the lock for notification matching, where a stale
+     * read can only funnel into a loss transition that a newer generation already made a no-op.
+     */
+    private int generation;
+    private volatile String canaryKey;
+
+    /**
+     * Set under {@link #stateLock} when the watcher closes, so that no state transition — and
+     * with it no listener dispatch — can interleave with the close. Volatile for the executor
+     * fast paths that do not dispatch.
+     */
+    private volatile boolean closed;
+
+    /** Retry counter of the current canary write chain; only accessed on the executor. */
+    private int writeFailures;
+
+    /**
+     * When the current streak of failing existence checks began, or zero outside a streak. Only
+     * accessed on the executor.
+     */
+    private long checkFailedSinceMillis;
+
+    private enum State {
+        STARTING, ESTABLISHED, DISCONNECTED, LOST
+    }
+
+    OxiaSessionWatcher(AsyncOxiaClient client, Consumer<SessionEvent> eventSink,
+            long sessionTimeoutMillis) {
+        this.client = client;
+        this.eventSink = eventSink;
+        this.canaryKeyPrefix = CANARY_KEY_PREFIX + UUID.randomUUID() + "-";
+        this.sessionTimeoutMillis = sessionTimeoutMillis;
+        this.checkIntervalMillis = Math.max(MIN_CHECK_INTERVAL_MILLIS,
+                Math.min(MAX_CHECK_INTERVAL_MILLIS, sessionTimeoutMillis / 3));
+        this.canaryKey = canaryKeyPrefix + generation;
+    }
+
+    /**
+     * Whether the notification belongs to the reserved canary namespace and must not be forwarded
+     * as an ordinary metadata store notification.
+     */
+    static boolean isCanaryKey(Notification notification) {
+        return notification.key().startsWith(CANARY_KEY_PREFIX);
+    }
+
+    void start() {
+        ensureCanaryWrite();
+        try {
+            executor.scheduleWithFixedDelay(this::checkCanarySafely, checkIntervalMillis,
+                    checkIntervalMillis, TimeUnit.MILLISECONDS);
+        } catch (RejectedExecutionException ignore) {
+            // The watcher was closed concurrently with the store construction.
+        }
+    }
+
+    /**
+     * Routes a notification from the canary namespace: only the deletion of the canary of the
+     * current incarnation means the session was lost, every other canary notification is ignored.
+     */
+    void handleNotification(Notification notification) {
+        if (closed) {
+            return;
+        }
+        if (notification instanceof Notification.KeyDeleted && notification.key().equals(canaryKey)) {
+            onCanaryGone("deleted");
+        }
+    }
+
+    private void checkCanarySafely() {
+        try {
+            checkCanary();
+        } catch (Throwable t) {
+            log.warn().exception(t).log("Unexpected failure while checking the oxia session canary");
+        }
+    }
+
+    /**
+     * Existence check of the canary, backing the notification stream up while the session is
+     * considered established and monitoring the connectivity to the server. Package-private so
+     * that tests can drive it directly.
+     */
+    void checkCanary() {
+        String keyAtIssue;
+        int generationAtIssue;
+        synchronized (stateLock) {
+            if (state != State.ESTABLISHED && state != State.DISCONNECTED) {
+                return;
+            }
+            keyAtIssue = canaryKey;
+            generationAtIssue = generation;
+        }
+        try {
+            client.get(keyAtIssue).whenComplete((result, ex) -> hop(() -> {

Review Comment:
   I think the generation guard is still missing for successful and failed probe completions.
   
   Only the null-result branch checks `generationAtIssue`. A failed get from an older incarnation can arrive after the new canary has been established and incorrectly move the current state back to `DISCONNECTED`.
   
   Also, multiple checks can be in flight, so an older successful response can arrive after a newer failure and incorrectly emit `Reconnected`.
   
   Could we validate the generation for every completion, and either serialize probes or track their sequence so an older result cannot override a newer observation?



##########
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/oxia/OxiaSessionWatcher.java:
##########
@@ -0,0 +1,401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.metadata.impl.oxia;
+
+import io.netty.util.concurrent.DefaultThreadFactory;
+import io.oxia.client.api.AsyncOxiaClient;
+import io.oxia.client.api.Notification;
+import io.oxia.client.api.options.PutOption;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.Executors;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import lombok.CustomLog;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+/**
+ * Delivers the metadata store session events on the oxia backend by watching a private ephemeral
+ * canary record.
+ *
+ * <p>Oxia does not expose session lifecycle events: the session machinery is deliberately kept
+ * internal to the client, and the only public observation surface is the notification stream.
+ * The per-record {@code KeyDeleted} notification that the server emits for every ephemeral record
+ * swept by a session expiry is therefore the intended way for an embedding application to learn
+ * about a session loss. This watcher applies exactly that mechanism: each store instance owns a
+ * canary record, under the reserved {@link #CANARY_KEY_PREFIX} namespace, that nobody but the
+ * session itself deletes. The deletion of the canary is a server-authoritative signal that the
+ * session died, and maps to {@link SessionEvent#SessionLost}; a subsequently successful canary
+ * write means a fresh session was established and maps to
+ * {@link SessionEvent#SessionReestablished}.
+ *
+ * <p>Each loss moves to a fresh incarnation of the canary key, a monotonically increasing
+ * generation suffixed to the instance-unique prefix. Notifications and write completions are only
+ * honored for the current incarnation: a {@code KeyDeleted} of an expired incarnation delivered
+ * late — the notification stream resumes from its last offset after a reconnect — cannot flip a
+ * live session to lost, and the completion of a write issued before a loss cannot report a
+ * recovery. Recovery is only ever reported by a write issued after the loss.
+ *
+ * <p>While the session is considered established, a periodic existence check of the canary backs
+ * the notification stream up: a deletion whose notification was lost, or that was delivered as a
+ * range delete, is detected by the next check instead of being missed forever. The same check
+ * monitors connectivity: the first failed check reports {@link SessionEvent#ConnectionLost},
+ * check failures spanning a full session timeout report {@link SessionEvent#SessionLost}, and a
+ * successful check after a connection loss reports {@link SessionEvent#Reconnected}. A failing
+ * check counts as no answer whatever its cause; the window is the session timeout because that is
+ * the window after which the server reaps the session — it is the timeout the client sends in
+ * {@code CreateSessionRequest} — and after which the oxia client itself expires the session. If
+ * connectivity returns within the window and the session survived, the pessimistic SessionLost is
+ * followed by a prompt SessionReestablished once the canary is rewritten: the same trade-off
+ * {@code ZKSessionWatcher} makes at its monitor timeout.
+ *
+ * <p>No ordering is guaranteed between the synthesized session events and the per-key
+ * notifications of the same sweep: a notification batch is a map on the wire and carries no
+ * per-entry order, so consumers must not rely on SessionLost preceding the Deleted notifications
+ * of the records swept with the canary.
+ *
+ * <p>Known limitation: the oxia client keeps one session per shard and the canary lives on a
+ * single shard, so a session loss confined to a different shard — for example the clean session
+ * close of a shard rebalance, which sweeps that shard's ephemeral records without an expiry — is
+ * not reported by this watcher. The common loss of the session, a server restart or outage, kills
+ * the sessions of all shards together with the canary and is fully covered.
+ */
+@CustomLog
+final class OxiaSessionWatcher implements AutoCloseable {
+
+    /**
+     * Reserved namespace of the session canary records. It must not be used for anything else and
+     * it must keep sorting before every {@code '/'}-rooted path, so that the canaries stay
+     * outside the {@code '/'} hierarchy the store exposes to ordinary listings.
+     */
+    static final String CANARY_KEY_PREFIX = ".pulsar-oxia-session-canary-";
+
+    /** Fallback session timeout for store instances built directly around a client. */
+    static final long DEFAULT_SESSION_TIMEOUT_MILLIS = TimeUnit.SECONDS.toMillis(30);
+
+    private static final byte[] CANARY_VALUE = new byte[0];
+    private static final long MAX_CHECK_INTERVAL_MILLIS = TimeUnit.SECONDS.toMillis(30);
+    private static final long MIN_CHECK_INTERVAL_MILLIS = TimeUnit.SECONDS.toMillis(1);
+    private static final long WRITE_RETRY_MIN_DELAY_MILLIS = TimeUnit.SECONDS.toMillis(1);
+    private static final long WRITE_RETRY_MAX_DELAY_MILLIS = TimeUnit.SECONDS.toMillis(8);
+
+    private final AsyncOxiaClient client;
+    private final Consumer<SessionEvent> eventSink;
+
+    /** Instance-unique prefix of the canary keys; the generation of the moment is appended. */
+    private final String canaryKeyPrefix;
+
+    private final long sessionTimeoutMillis;
+    private final long checkIntervalMillis;
+
+    private final ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor(
+            new DefaultThreadFactory("oxia-metadata-session-watcher"));
+
+    /**
+     * Set while a canary write is scheduled or in flight, so that at most one write chain exists
+     * at any time.
+     */
+    private final AtomicBoolean canaryWritePending = new AtomicBoolean();
+
+    private final Object stateLock = new Object();
+    private State state = State.STARTING;
+
+    /**
+     * Generation of the current canary incarnation and its key. Guarded by {@link #stateLock};
+     * the key is additionally readable without the lock for notification matching, where a stale
+     * read can only funnel into a loss transition that a newer generation already made a no-op.
+     */
+    private int generation;
+    private volatile String canaryKey;
+
+    /**
+     * Set under {@link #stateLock} when the watcher closes, so that no state transition — and
+     * with it no listener dispatch — can interleave with the close. Volatile for the executor
+     * fast paths that do not dispatch.
+     */
+    private volatile boolean closed;
+
+    /** Retry counter of the current canary write chain; only accessed on the executor. */
+    private int writeFailures;
+
+    /**
+     * When the current streak of failing existence checks began, or zero outside a streak. Only
+     * accessed on the executor.
+     */
+    private long checkFailedSinceMillis;
+
+    private enum State {
+        STARTING, ESTABLISHED, DISCONNECTED, LOST
+    }
+
+    OxiaSessionWatcher(AsyncOxiaClient client, Consumer<SessionEvent> eventSink,
+            long sessionTimeoutMillis) {
+        this.client = client;
+        this.eventSink = eventSink;
+        this.canaryKeyPrefix = CANARY_KEY_PREFIX + UUID.randomUUID() + "-";
+        this.sessionTimeoutMillis = sessionTimeoutMillis;
+        this.checkIntervalMillis = Math.max(MIN_CHECK_INTERVAL_MILLIS,
+                Math.min(MAX_CHECK_INTERVAL_MILLIS, sessionTimeoutMillis / 3));
+        this.canaryKey = canaryKeyPrefix + generation;
+    }
+
+    /**
+     * Whether the notification belongs to the reserved canary namespace and must not be forwarded
+     * as an ordinary metadata store notification.
+     */
+    static boolean isCanaryKey(Notification notification) {
+        return notification.key().startsWith(CANARY_KEY_PREFIX);
+    }
+
+    void start() {
+        ensureCanaryWrite();
+        try {
+            executor.scheduleWithFixedDelay(this::checkCanarySafely, checkIntervalMillis,
+                    checkIntervalMillis, TimeUnit.MILLISECONDS);
+        } catch (RejectedExecutionException ignore) {
+            // The watcher was closed concurrently with the store construction.
+        }
+    }
+
+    /**
+     * Routes a notification from the canary namespace: only the deletion of the canary of the
+     * current incarnation means the session was lost, every other canary notification is ignored.
+     */
+    void handleNotification(Notification notification) {
+        if (closed) {
+            return;
+        }
+        if (notification instanceof Notification.KeyDeleted && notification.key().equals(canaryKey)) {
+            onCanaryGone("deleted");
+        }
+    }
+
+    private void checkCanarySafely() {
+        try {
+            checkCanary();
+        } catch (Throwable t) {
+            log.warn().exception(t).log("Unexpected failure while checking the oxia session canary");
+        }
+    }
+
+    /**
+     * Existence check of the canary, backing the notification stream up while the session is
+     * considered established and monitoring the connectivity to the server. Package-private so
+     * that tests can drive it directly.
+     */
+    void checkCanary() {
+        String keyAtIssue;
+        int generationAtIssue;
+        synchronized (stateLock) {
+            if (state != State.ESTABLISHED && state != State.DISCONNECTED) {
+                return;
+            }
+            keyAtIssue = canaryKey;
+            generationAtIssue = generation;
+        }
+        try {
+            client.get(keyAtIssue).whenComplete((result, ex) -> hop(() -> {
+                if (ex != null) {
+                    onCheckFailed(ex);
+                } else if (result != null) {
+                    onCheckSucceeded();
+                } else {
+                    boolean staleRead;
+                    synchronized (stateLock) {
+                        staleRead = closed || generation != generationAtIssue;
+                    }
+                    // A read of an expired incarnation finds its key missing, but that loss was
+                    // already reported when the incarnation was replaced.
+                    if (!staleRead) {
+                        onCanaryGone("missing");
+                    }
+                }
+            }));
+        } catch (RejectedExecutionException | IllegalStateException ignore) {
+            // The client was closed concurrently.
+        }
+    }
+
+    private void onCheckFailed(Throwable ex) {
+        long now = System.currentTimeMillis();
+        boolean sessionTimedOut;
+        synchronized (stateLock) {
+            if (closed || state == State.LOST || state == State.STARTING) {
+                return;
+            }
+            if (state == State.ESTABLISHED) {
+                state = State.DISCONNECTED;
+                checkFailedSinceMillis = now;
+                log.warn().exception(ex).log("Oxia metadata store connection was lost");
+                eventSink.accept(SessionEvent.ConnectionLost);
+            }
+            sessionTimedOut = now - checkFailedSinceMillis >= sessionTimeoutMillis;
+        }
+        if (sessionTimedOut) {
+            // The checks could not reach the server for a full session timeout, by which point
+            // the server has reaped the session. Report the loss now instead of waiting for the
+            // connectivity to return and confirm it.
+            onCanaryGone("unreachable");
+        }
+    }
+
+    private void onCheckSucceeded() {
+        synchronized (stateLock) {
+            if (closed || state != State.DISCONNECTED) {
+                return;
+            }
+            state = State.ESTABLISHED;
+            checkFailedSinceMillis = 0L;
+            log.info().log("Oxia metadata store connection was re-established");
+            eventSink.accept(SessionEvent.Reconnected);
+        }
+    }
+
+    private void onCanaryGone(String cause) {
+        // The events are dispatched while holding the lock, so that the listeners observe them
+        // in exactly the order of the state transitions, and so that no transition can inter
+        // leave with the close. The dispatch is non-blocking.
+        synchronized (stateLock) {
+            if (closed || state == State.LOST) {
+                return;
+            }
+            state = State.LOST;
+            generation++;
+            canaryKey = canaryKeyPrefix + generation;
+            log.info().attr("canaryKey", canaryKey).attr("cause", cause)
+                    .log("Oxia metadata store session was lost");
+            eventSink.accept(SessionEvent.SessionLost);
+        }
+        ensureCanaryWrite();
+    }
+
+    /**
+     * Arms the canary write if no write chain is already scheduled or in flight: the first write
+     * of a chain retries with backoff until it succeeds, so re-arming is only needed when the
+     * chain came to rest on a successfully written canary.
+     */
+    private void ensureCanaryWrite() {
+        if (canaryWritePending.compareAndSet(false, true)) {
+            scheduleCanaryWrite(0, true);
+        }
+    }
+
+    private void scheduleCanaryWrite(long delayMillis, boolean resetFailures) {
+        try {
+            executor.schedule(() -> writeCanary(resetFailures), delayMillis, TimeUnit.MILLISECONDS);
+        } catch (RejectedExecutionException ignore) {
+            canaryWritePending.set(false);
+        }
+    }
+
+    private void writeCanary(boolean resetFailures) {
+        if (closed) {
+            canaryWritePending.set(false);
+            return;
+        }
+        if (resetFailures) {
+            writeFailures = 0;
+        }
+        String keyAtIssue;
+        int generationAtIssue;
+        synchronized (stateLock) {
+            keyAtIssue = canaryKey;
+            generationAtIssue = generation;
+        }
+        try {
+            client.put(keyAtIssue, CANARY_VALUE, Set.of(PutOption.AsEphemeralRecord))
+                    .whenComplete((result, ex) -> hop(() -> onCanaryWriteCompleted(generationAtIssue, ex)));
+        } catch (RuntimeException e) {
+            hop(() -> onCanaryWriteCompleted(generationAtIssue, e));
+        }
+    }
+
+    private void onCanaryWriteCompleted(int generationAtIssue, Throwable ex) {
+        boolean appliesToCurrentIncarnation;
+        synchronized (stateLock) {
+            appliesToCurrentIncarnation = !closed && generation == generationAtIssue;
+            if (appliesToCurrentIncarnation && ex == null) {
+                boolean wasLost = state == State.LOST;
+                state = State.ESTABLISHED;
+                if (wasLost) {
+                    log.info().attr("canaryKey", canaryKey)
+                            .log("Oxia metadata store session was re-established");
+                    eventSink.accept(SessionEvent.SessionReestablished);
+                }
+            }
+        }
+        if (!appliesToCurrentIncarnation) {
+            // The write belongs to an incarnation that was lost in the meantime, or the watcher
+            // was closed: rest the chain and, if still open, arm a fresh write for the current
+            // incarnation.
+            canaryWritePending.set(false);
+            if (!closed) {
+                ensureCanaryWrite();
+            }
+            return;
+        }
+        if (ex == null) {
+            canaryWritePending.set(false);
+            return;

Review Comment:
   I think there's still a race between a successful canary write completion and `onCanaryGone()`.
   
   The completion moves the state to `ESTABLISHED` under `stateLock`, but clears `canaryWritePending` only after releasing the lock.
   
   If the notification thread processes a `KeyDeleted` in between, `onCanaryGone()` moves the watcher to `LOST` and increments the generation, but `ensureCanaryWrite()` sees `canaryWritePending == true` and doesn't schedule anything. The previous completion then clears the flag and returns.
   
   We end up in `LOST` with no pending canary write, and the periodic check skips `LOST`, so the watcher can remain stuck indefinitely.
   
   Could we coordinate clearing the pending flag with the state transition, or re-check whether the current generation still needs a write before completing the chain? A test forcing this interleaving would be useful.



##########
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/oxia/OxiaSessionWatcher.java:
##########
@@ -0,0 +1,401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.metadata.impl.oxia;
+
+import io.netty.util.concurrent.DefaultThreadFactory;
+import io.oxia.client.api.AsyncOxiaClient;
+import io.oxia.client.api.Notification;
+import io.oxia.client.api.options.PutOption;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.Executors;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import lombok.CustomLog;
+import org.apache.pulsar.metadata.api.extended.SessionEvent;
+
+/**
+ * Delivers the metadata store session events on the oxia backend by watching a private ephemeral
+ * canary record.
+ *
+ * <p>Oxia does not expose session lifecycle events: the session machinery is deliberately kept
+ * internal to the client, and the only public observation surface is the notification stream.
+ * The per-record {@code KeyDeleted} notification that the server emits for every ephemeral record
+ * swept by a session expiry is therefore the intended way for an embedding application to learn
+ * about a session loss. This watcher applies exactly that mechanism: each store instance owns a
+ * canary record, under the reserved {@link #CANARY_KEY_PREFIX} namespace, that nobody but the
+ * session itself deletes. The deletion of the canary is a server-authoritative signal that the
+ * session died, and maps to {@link SessionEvent#SessionLost}; a subsequently successful canary
+ * write means a fresh session was established and maps to
+ * {@link SessionEvent#SessionReestablished}.
+ *
+ * <p>Each loss moves to a fresh incarnation of the canary key, a monotonically increasing
+ * generation suffixed to the instance-unique prefix. Notifications and write completions are only
+ * honored for the current incarnation: a {@code KeyDeleted} of an expired incarnation delivered
+ * late — the notification stream resumes from its last offset after a reconnect — cannot flip a
+ * live session to lost, and the completion of a write issued before a loss cannot report a
+ * recovery. Recovery is only ever reported by a write issued after the loss.
+ *
+ * <p>While the session is considered established, a periodic existence check of the canary backs
+ * the notification stream up: a deletion whose notification was lost, or that was delivered as a
+ * range delete, is detected by the next check instead of being missed forever. The same check
+ * monitors connectivity: the first failed check reports {@link SessionEvent#ConnectionLost},
+ * check failures spanning a full session timeout report {@link SessionEvent#SessionLost}, and a
+ * successful check after a connection loss reports {@link SessionEvent#Reconnected}. A failing
+ * check counts as no answer whatever its cause; the window is the session timeout because that is
+ * the window after which the server reaps the session — it is the timeout the client sends in
+ * {@code CreateSessionRequest} — and after which the oxia client itself expires the session. If
+ * connectivity returns within the window and the session survived, the pessimistic SessionLost is
+ * followed by a prompt SessionReestablished once the canary is rewritten: the same trade-off
+ * {@code ZKSessionWatcher} makes at its monitor timeout.
+ *
+ * <p>No ordering is guaranteed between the synthesized session events and the per-key
+ * notifications of the same sweep: a notification batch is a map on the wire and carries no
+ * per-entry order, so consumers must not rely on SessionLost preceding the Deleted notifications
+ * of the records swept with the canary.
+ *
+ * <p>Known limitation: the oxia client keeps one session per shard and the canary lives on a
+ * single shard, so a session loss confined to a different shard — for example the clean session
+ * close of a shard rebalance, which sweeps that shard's ephemeral records without an expiry — is
+ * not reported by this watcher. The common loss of the session, a server restart or outage, kills
+ * the sessions of all shards together with the canary and is fully covered.
+ */
+@CustomLog
+final class OxiaSessionWatcher implements AutoCloseable {
+
+    /**
+     * Reserved namespace of the session canary records. It must not be used for anything else and
+     * it must keep sorting before every {@code '/'}-rooted path, so that the canaries stay
+     * outside the {@code '/'} hierarchy the store exposes to ordinary listings.
+     */
+    static final String CANARY_KEY_PREFIX = ".pulsar-oxia-session-canary-";
+
+    /** Fallback session timeout for store instances built directly around a client. */
+    static final long DEFAULT_SESSION_TIMEOUT_MILLIS = TimeUnit.SECONDS.toMillis(30);
+
+    private static final byte[] CANARY_VALUE = new byte[0];
+    private static final long MAX_CHECK_INTERVAL_MILLIS = TimeUnit.SECONDS.toMillis(30);
+    private static final long MIN_CHECK_INTERVAL_MILLIS = TimeUnit.SECONDS.toMillis(1);
+    private static final long WRITE_RETRY_MIN_DELAY_MILLIS = TimeUnit.SECONDS.toMillis(1);
+    private static final long WRITE_RETRY_MAX_DELAY_MILLIS = TimeUnit.SECONDS.toMillis(8);
+
+    private final AsyncOxiaClient client;
+    private final Consumer<SessionEvent> eventSink;
+
+    /** Instance-unique prefix of the canary keys; the generation of the moment is appended. */
+    private final String canaryKeyPrefix;
+
+    private final long sessionTimeoutMillis;
+    private final long checkIntervalMillis;
+
+    private final ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor(
+            new DefaultThreadFactory("oxia-metadata-session-watcher"));
+
+    /**
+     * Set while a canary write is scheduled or in flight, so that at most one write chain exists
+     * at any time.
+     */
+    private final AtomicBoolean canaryWritePending = new AtomicBoolean();
+
+    private final Object stateLock = new Object();
+    private State state = State.STARTING;
+
+    /**
+     * Generation of the current canary incarnation and its key. Guarded by {@link #stateLock};
+     * the key is additionally readable without the lock for notification matching, where a stale
+     * read can only funnel into a loss transition that a newer generation already made a no-op.
+     */
+    private int generation;
+    private volatile String canaryKey;
+
+    /**
+     * Set under {@link #stateLock} when the watcher closes, so that no state transition — and
+     * with it no listener dispatch — can interleave with the close. Volatile for the executor
+     * fast paths that do not dispatch.
+     */
+    private volatile boolean closed;
+
+    /** Retry counter of the current canary write chain; only accessed on the executor. */
+    private int writeFailures;
+
+    /**
+     * When the current streak of failing existence checks began, or zero outside a streak. Only
+     * accessed on the executor.
+     */
+    private long checkFailedSinceMillis;
+
+    private enum State {
+        STARTING, ESTABLISHED, DISCONNECTED, LOST
+    }
+
+    OxiaSessionWatcher(AsyncOxiaClient client, Consumer<SessionEvent> eventSink,
+            long sessionTimeoutMillis) {
+        this.client = client;
+        this.eventSink = eventSink;
+        this.canaryKeyPrefix = CANARY_KEY_PREFIX + UUID.randomUUID() + "-";
+        this.sessionTimeoutMillis = sessionTimeoutMillis;
+        this.checkIntervalMillis = Math.max(MIN_CHECK_INTERVAL_MILLIS,
+                Math.min(MAX_CHECK_INTERVAL_MILLIS, sessionTimeoutMillis / 3));
+        this.canaryKey = canaryKeyPrefix + generation;
+    }
+
+    /**
+     * Whether the notification belongs to the reserved canary namespace and must not be forwarded
+     * as an ordinary metadata store notification.
+     */
+    static boolean isCanaryKey(Notification notification) {
+        return notification.key().startsWith(CANARY_KEY_PREFIX);
+    }
+
+    void start() {
+        ensureCanaryWrite();
+        try {
+            executor.scheduleWithFixedDelay(this::checkCanarySafely, checkIntervalMillis,
+                    checkIntervalMillis, TimeUnit.MILLISECONDS);
+        } catch (RejectedExecutionException ignore) {
+            // The watcher was closed concurrently with the store construction.
+        }
+    }
+
+    /**
+     * Routes a notification from the canary namespace: only the deletion of the canary of the
+     * current incarnation means the session was lost, every other canary notification is ignored.
+     */
+    void handleNotification(Notification notification) {
+        if (closed) {
+            return;
+        }
+        if (notification instanceof Notification.KeyDeleted && notification.key().equals(canaryKey)) {
+            onCanaryGone("deleted");
+        }
+    }
+
+    private void checkCanarySafely() {
+        try {
+            checkCanary();
+        } catch (Throwable t) {
+            log.warn().exception(t).log("Unexpected failure while checking the oxia session canary");
+        }
+    }
+
+    /**
+     * Existence check of the canary, backing the notification stream up while the session is
+     * considered established and monitoring the connectivity to the server. Package-private so
+     * that tests can drive it directly.
+     */
+    void checkCanary() {
+        String keyAtIssue;
+        int generationAtIssue;
+        synchronized (stateLock) {
+            if (state != State.ESTABLISHED && state != State.DISCONNECTED) {
+                return;
+            }
+            keyAtIssue = canaryKey;
+            generationAtIssue = generation;
+        }
+        try {
+            client.get(keyAtIssue).whenComplete((result, ex) -> hop(() -> {
+                if (ex != null) {
+                    onCheckFailed(ex);
+                } else if (result != null) {
+                    onCheckSucceeded();
+                } else {
+                    boolean staleRead;
+                    synchronized (stateLock) {
+                        staleRead = closed || generation != generationAtIssue;
+                    }
+                    // A read of an expired incarnation finds its key missing, but that loss was
+                    // already reported when the incarnation was replaced.
+                    if (!staleRead) {
+                        onCanaryGone("missing");
+                    }
+                }
+            }));
+        } catch (RejectedExecutionException | IllegalStateException ignore) {
+            // The client was closed concurrently.
+        }
+    }
+
+    private void onCheckFailed(Throwable ex) {
+        long now = System.currentTimeMillis();
+        boolean sessionTimedOut;
+        synchronized (stateLock) {
+            if (closed || state == State.LOST || state == State.STARTING) {
+                return;
+            }
+            if (state == State.ESTABLISHED) {
+                state = State.DISCONNECTED;
+                checkFailedSinceMillis = now;
+                log.warn().exception(ex).log("Oxia metadata store connection was lost");
+                eventSink.accept(SessionEvent.ConnectionLost);
+            }
+            sessionTimedOut = now - checkFailedSinceMillis >= sessionTimeoutMillis;

Review Comment:
   One concern with the timeout calculation: we're starting the disconnect window when the first get completes exceptionally, rather than when connectivity was actually lost.
   
   Oxia's `get()` has its own request timeout (30s by default). With a 30s session timeout, a black-holed request could already have been pending for 30 seconds before we start another 30-second window for `SessionLost`.
   
   Could we track the last successful probe using a monotonic clock, and account for the time an outstanding probe has already spent without a response? Otherwise the loss notification can arrive significantly later than the configured session timeout.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org