You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "lucasbru (via GitHub)" <gi...@apache.org> on 2026/06/05 18:54:37 UTC

[PR] KAFKA-XXXXX: AdminClient partition-leader APIs hang until default.api.timeout.ms when a cached leader has left the cluster [kafka]

lucasbru opened a new pull request, #22493:
URL: https://github.com/apache/kafka/pull/22493

   Since the partition-leader cache fast path was introduced, `AdminApiDriver` routes a cached leader straight to the fulfillment stage with a `ConstantNodeIdProvider` for that broker id, skipping the lookup stage. If the cached broker id is no longer present in the admin client's cluster metadata — for example because the broker left the cluster and was not replaced under the same id — the call can never be assigned a node. `ConstantNodeIdProvider.provide()` returns null and the call sits in `pendingCalls` issuing only broker-info metadata refreshes, which never re-resolve the partition leader, so the request stays stuck until `default.api.timeout.ms` expires and then fails with "Timed out waiting for a node assignment". This affects all `PartitionLeaderStrategy`-based APIs: `listOffsets`, `deleteRecords`, `describeProducers` and `abortTransaction`.
   
   The cache's existing staleness recovery only covers leader changes that surface as per-partition errors such as `NOT_LEADER_OR_FOLLOWER`. There is no path that handles a cached leader id simply disappearing from the cluster without any such error reaching the admin client, so the stale entry is believed until the deadline.
   
   This change detects, on the admin client thread where metadata access is safe, that a fulfillment call's target broker is absent from ready metadata, and sends the affected keys back to the lookup stage so the leader is re-resolved with a fresh topic metadata request. The liveness check is performed on the admin client thread rather than where the cache is read: the cache is consulted in the `AdminApiDriver` constructor on the calling thread, where `AdminMetadataManager` (which is confined to the admin client thread) cannot be accessed safely.
   
   The behaviour is covered by a driver-level test that verifies a cached fulfillment request is sent back to the lookup stage, and by an end-to-end test in `KafkaAdminClientTest` that seeds the cache, removes the cached leader from the cluster, and asserts that the subsequent `listOffsets` re-resolves the leader and completes instead of timing out.
   
   The JIRA ticket is still to be filed; the title will be updated with the KAFKA number.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20673: AdminClient partition-leader APIs hang when a cached leader has left the cluster [kafka]

Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #22493:
URL: https://github.com/apache/kafka/pull/22493#discussion_r3387344721


##########
clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java:
##########
@@ -5151,6 +5163,27 @@ void maybeRetry(long currentTimeMs, Throwable throwable) {
                     super.maybeRetry(currentTimeMs, throwable);
                 }
             }
+
+            @Override
+            boolean handleNodeUnavailable(long currentTimeMs) {
+                OptionalInt brokerId = spec.scope.destinationBrokerId();
+                // The fulfillment target broker is no longer present in the cluster metadata. This
+                // happens when a stale entry in the partition leader cache points at a broker that
+                // has since left the cluster. Send the keys back to the lookup stage so the leader
+                // can be re-resolved, rather than waiting for the request deadline to expire without
+                // ever issuing another lookup. maybeRetryLookup is a no-op (returns false) for
+                // strategies that target a fixed broker id, in which case we leave the call pending.
+                if (brokerId.isPresent()
+                        && metadataManager.isReady()
+                        && metadataManager.nodeById(brokerId.getAsInt()) == null
+                        && driver.maybeRetryLookup(currentTimeMs, spec)) {
+                    log.debug("Broker {} for {} is no longer in the cluster metadata; retrying lookup.",
+                        brokerId.getAsInt(), spec.name);
+                    maybeSendRequests(driver, currentTimeMs);

Review Comment:
   Good catch, thanks! You're right - close() sets the hard shutdown time up front, so routing the re-lookup through runnable.call() gets it rejected immediately. I've added a guard so handleNodeUnavailable does nothing while the client is shutting down: the call stays in pendingCalls and follows the normal close(timeout) handling (retried within the grace period, or assigned if the broker reappears) instead of failing instantly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20673: AdminClient partition-leader APIs hang when a cached leader has left the cluster [kafka]

Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #22493:
URL: https://github.com/apache/kafka/pull/22493#discussion_r3380094780


##########
clients/src/main/java/org/apache/kafka/clients/admin/internals/AdminApiDriver.java:
##########
@@ -188,6 +188,33 @@ private void retryLookup(Collection<K> keys) {
         keys.forEach(this::unmap);
     }
 
+    /**
+     * Send the keys of a fulfillment request back to the Lookup stage. This is invoked when a
+     * fulfillment request cannot be routed because its target broker is no longer present in the
+     * cluster metadata, for example when a stale entry in the partition leader cache pointed at a
+     * broker that has since left the cluster. Re-running the lookup gives us a chance to discover
+     * the current leader. Without this, such a request would remain unassignable until the request
+     * deadline expires.
+     *
+     * <p>This is a no-op for lookup strategies that target a fixed broker (i.e. the lookup scope
+     * carries a destination broker id, as with {@link StaticBrokerStrategy}), since {@link #unmap}
+     * would simply remap the keys straight back to the same fulfillment broker without making any
+     * progress. In that case this returns {@code false} and the request is left untouched, so the
+     * caller can leave it pending rather than retrying in a loop and exhausting the retry budget.
+     *
+     * @return true if the keys were moved back to the Lookup stage, false if no lookup is possible
+     */
+    public boolean retryLookup(long currentTimeMs, RequestSpec<K> spec) {

Review Comment:
   Good point - renamed it to `maybeRetryLookup` to distinguish it from the private `retryLookup`, and the `maybe` also signals it can be a no-op (when the keys target a fixed broker and can't go back to lookup).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20673: AdminClient partition-leader APIs hang when a cached leader has left the cluster [kafka]

Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #22493:
URL: https://github.com/apache/kafka/pull/22493#discussion_r3374911670


##########
clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java:
##########
@@ -5151,6 +5163,26 @@ void maybeRetry(long currentTimeMs, Throwable throwable) {
                     super.maybeRetry(currentTimeMs, throwable);
                 }
             }
+
+            @Override
+            boolean handleNodeUnavailable(long currentTimeMs) {
+                OptionalInt brokerId = spec.scope.destinationBrokerId();
+                if (brokerId.isPresent()
+                        && metadataManager.isReady()
+                        && metadataManager.nodeById(brokerId.getAsInt()) == null) {
+                    // The fulfillment target broker is no longer present in the cluster metadata.
+                    // This happens when a stale entry in the partition leader cache points at a
+                    // broker that has since left the cluster. Send the keys back to the lookup
+                    // stage so the leader can be re-resolved, rather than waiting for the request
+                    // deadline to expire without ever issuing another lookup.
+                    log.debug("Broker {} for {} is no longer in the cluster metadata; retrying lookup.",
+                        brokerId.getAsInt(), spec.name);
+                    driver.retryLookup(currentTimeMs, spec);
+                    maybeSendRequests(driver, currentTimeMs);
+                    return true;
+                }
+                return false;

Review Comment:
   Good catch. Fixed: handleNodeUnavailable now only retries the lookup when retryLookup reports it could move the keys back. For StaticBrokerStrategy (describeProducers with an explicit broker id) it's a no-op, so the call is left pending until the deadline as before, rather than looping and exhausting the retry budget. Added testRetryLookupIsNoOpForFixedBrokerStrategy to cover it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20673: AdminClient partition-leader APIs hang when a cached leader has left the cluster [kafka]

Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #22493:
URL: https://github.com/apache/kafka/pull/22493#discussion_r3374910494


##########
clients/src/main/java/org/apache/kafka/clients/admin/internals/AdminApiDriver.java:
##########
@@ -188,6 +188,19 @@ private void retryLookup(Collection<K> keys) {
         keys.forEach(this::unmap);
     }
 
+    /**
+     * Send the keys of a fulfillment request back to the Lookup stage. This is invoked when a
+     * fulfillment request cannot be routed because its target broker is no longer present in the
+     * cluster metadata, for example when a stale entry in the partition leader cache pointed at a
+     * broker that has since left the cluster. Re-running the lookup gives us a chance to discover
+     * the current leader. Without this, such a request would remain unassignable until the request
+     * deadline expires.
+     */
+    public void retryLookup(long currentTimeMs, RequestSpec<K> spec) {
+        clearInflightRequest(currentTimeMs, spec);
+        retryLookup(spec.keys);
+    }

Review Comment:
   Done — retryLookup now returns a boolean and is a no-op for fixed-broker lookup strategies, so the caller only takes corrective action when the keys actually moved back to the lookup stage.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20673: AdminClient partition-leader APIs hang when a cached leader has left the cluster [kafka]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #22493:
URL: https://github.com/apache/kafka/pull/22493#discussion_r3374763096


##########
clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java:
##########
@@ -5151,6 +5163,26 @@ void maybeRetry(long currentTimeMs, Throwable throwable) {
                     super.maybeRetry(currentTimeMs, throwable);
                 }
             }
+
+            @Override
+            boolean handleNodeUnavailable(long currentTimeMs) {
+                OptionalInt brokerId = spec.scope.destinationBrokerId();
+                if (brokerId.isPresent()
+                        && metadataManager.isReady()
+                        && metadataManager.nodeById(brokerId.getAsInt()) == null) {
+                    // The fulfillment target broker is no longer present in the cluster metadata.
+                    // This happens when a stale entry in the partition leader cache points at a
+                    // broker that has since left the cluster. Send the keys back to the lookup
+                    // stage so the leader can be re-resolved, rather than waiting for the request
+                    // deadline to expire without ever issuing another lookup.
+                    log.debug("Broker {} for {} is no longer in the cluster metadata; retrying lookup.",
+                        brokerId.getAsInt(), spec.name);
+                    driver.retryLookup(currentTimeMs, spec);
+                    maybeSendRequests(driver, currentTimeMs);
+                    return true;
+                }
+                return false;

Review Comment:
   `handleNodeUnavailable` unconditionally calls `driver.retryLookup(...)` for any driver-based request with a destination broker missing from metadata. For `StaticBrokerStrategy` (e.g., `describeProducers` when `options.brokerId` is set), retryLookup is a no-op (it remaps straight back to the same fulfillment broker), so this path will repeatedly recreate calls and increment the driver/call `tries` without ever sending a request. That can cause premature failure via `maxRetries` ("Exceeded maxRetries") rather than waiting for `default.api.timeout.ms` / deadline, which is a behavior regression for APIs that intentionally target a specific broker id.



##########
clients/src/main/java/org/apache/kafka/clients/admin/internals/AdminApiDriver.java:
##########
@@ -188,6 +188,19 @@ private void retryLookup(Collection<K> keys) {
         keys.forEach(this::unmap);
     }
 
+    /**
+     * Send the keys of a fulfillment request back to the Lookup stage. This is invoked when a
+     * fulfillment request cannot be routed because its target broker is no longer present in the
+     * cluster metadata, for example when a stale entry in the partition leader cache pointed at a
+     * broker that has since left the cluster. Re-running the lookup gives us a chance to discover
+     * the current leader. Without this, such a request would remain unassignable until the request
+     * deadline expires.
+     */
+    public void retryLookup(long currentTimeMs, RequestSpec<K> spec) {
+        clearInflightRequest(currentTimeMs, spec);
+        retryLookup(spec.keys);
+    }

Review Comment:
   `retryLookup(long, RequestSpec)` always clears the inflight request and then calls `unmap` for each key, but `unmap` may immediately remap back into the fulfillment stage when the lookup strategy’s `lookupScope(key).destinationBrokerId()` is present (e.g., `StaticBrokerStrategy`). In that case, callers like `KafkaAdminClient.Call#handleNodeUnavailable` can end up in a loop that recreates fulfillment specs/calls and consumes retry counters without ever making progress. Consider making this method return whether it actually moved any keys into the lookup stage (destination broker id empty) so callers can decide whether to take corrective action or leave the call pending.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20673: AdminClient partition-leader APIs hang when a cached leader has left the cluster [kafka]

Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru merged PR #22493:
URL: https://github.com/apache/kafka/pull/22493


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20673: AdminClient partition-leader APIs hang when a cached leader has left the cluster [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #22493:
URL: https://github.com/apache/kafka/pull/22493#discussion_r3386849247


##########
clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java:
##########
@@ -5151,6 +5163,27 @@ void maybeRetry(long currentTimeMs, Throwable throwable) {
                     super.maybeRetry(currentTimeMs, throwable);
                 }
             }
+
+            @Override
+            boolean handleNodeUnavailable(long currentTimeMs) {
+                OptionalInt brokerId = spec.scope.destinationBrokerId();
+                // The fulfillment target broker is no longer present in the cluster metadata. This
+                // happens when a stale entry in the partition leader cache points at a broker that
+                // has since left the cluster. Send the keys back to the lookup stage so the leader
+                // can be re-resolved, rather than waiting for the request deadline to expire without
+                // ever issuing another lookup. maybeRetryLookup is a no-op (returns false) for
+                // strategies that target a fixed broker id, in which case we leave the call pending.
+                if (brokerId.isPresent()
+                        && metadataManager.isReady()
+                        && metadataManager.nodeById(brokerId.getAsInt()) == null
+                        && driver.maybeRetryLookup(currentTimeMs, spec)) {
+                    log.debug("Broker {} for {} is no longer in the cluster metadata; retrying lookup.",
+                        brokerId.getAsInt(), spec.name);
+                    maybeSendRequests(driver, currentTimeMs);

Review Comment:
   @lucasbru Thanks for bringing this fix. It looks good to me! However, I have a small question about this approach. Does it introduces a side effect for retriable requests during the shutdown phase? This approach drain the pending calls and then enqueues new ones. If the `AdminClient` is closing, the new calls will be rejected immediately due to the error "Cannot accept new calls when AdminClient is closing". Hence, those calls will be killed instantly and cannot be retried, even if users define a large timeout for the `admin.close(timeout)` phase. Let me know your thoughts!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-XXXXX: AdminClient partition-leader APIs hang until default.api.timeout.ms when a cached leader has left the cluster [kafka]

Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on PR #22493:
URL: https://github.com/apache/kafka/pull/22493#issuecomment-4649478156

   @hgeraldino If the problem in your case is also that the broker ID disappeared, we should be able refresh the metadata directly, instead of waiting for `metadata.max.age.ms` to expire? So I think I'd prefer this solution, as it will recover more quickly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20673: AdminClient partition-leader APIs hang when a cached leader has left the cluster [kafka]

Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #22493:
URL: https://github.com/apache/kafka/pull/22493#discussion_r3379517093


##########
clients/src/main/java/org/apache/kafka/clients/admin/internals/AdminApiDriver.java:
##########
@@ -188,6 +188,33 @@ private void retryLookup(Collection<K> keys) {
         keys.forEach(this::unmap);
     }
 
+    /**
+     * Send the keys of a fulfillment request back to the Lookup stage. This is invoked when a
+     * fulfillment request cannot be routed because its target broker is no longer present in the
+     * cluster metadata, for example when a stale entry in the partition leader cache pointed at a
+     * broker that has since left the cluster. Re-running the lookup gives us a chance to discover
+     * the current leader. Without this, such a request would remain unassignable until the request
+     * deadline expires.
+     *
+     * <p>This is a no-op for lookup strategies that target a fixed broker (i.e. the lookup scope
+     * carries a destination broker id, as with {@link StaticBrokerStrategy}), since {@link #unmap}
+     * would simply remap the keys straight back to the same fulfillment broker without making any
+     * progress. In that case this returns {@code false} and the request is left untouched, so the
+     * caller can leave it pending rather than retrying in a loop and exhausting the retry budget.
+     *
+     * @return true if the keys were moved back to the Lookup stage, false if no lookup is possible
+     */
+    public boolean retryLookup(long currentTimeMs, RequestSpec<K> spec) {

Review Comment:
   I find the use of `retryLookup` as the name of this method confusing. It calls the real `retryLookup`, which is called from other places too.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-XXXXX: AdminClient partition-leader APIs hang until default.api.timeout.ms when a cached leader has left the cluster [kafka]

Posted by "hgeraldino (via GitHub)" <gi...@apache.org>.
hgeraldino commented on PR #22493:
URL: https://github.com/apache/kafka/pull/22493#issuecomment-4643372858

   hey @lucasbru,
   
   https://github.com/apache/kafka/pull/22390 was my attempt to address this issue, which affects Kafka Connect session keys (issue documented in https://issues.apache.org/jira/browse/KAFKA-20628)
   
   My approach was to expire the cached brokers using the metadata timeout, which I think can complement this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20673: AdminClient partition-leader APIs hang when a cached leader has left the cluster [kafka]

Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #22493:
URL: https://github.com/apache/kafka/pull/22493#discussion_r3387723156


##########
clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java:
##########
@@ -5151,6 +5163,27 @@ void maybeRetry(long currentTimeMs, Throwable throwable) {
                     super.maybeRetry(currentTimeMs, throwable);
                 }
             }
+
+            @Override
+            boolean handleNodeUnavailable(long currentTimeMs) {
+                OptionalInt brokerId = spec.scope.destinationBrokerId();
+                // The fulfillment target broker is no longer present in the cluster metadata. This
+                // happens when a stale entry in the partition leader cache points at a broker that
+                // has since left the cluster. Send the keys back to the lookup stage so the leader
+                // can be re-resolved, rather than waiting for the request deadline to expire without
+                // ever issuing another lookup. maybeRetryLookup is a no-op (returns false) for
+                // strategies that target a fixed broker id, in which case we leave the call pending.
+                if (brokerId.isPresent()
+                        && metadataManager.isReady()
+                        && metadataManager.nodeById(brokerId.getAsInt()) == null
+                        && driver.maybeRetryLookup(currentTimeMs, spec)) {
+                    log.debug("Broker {} for {} is no longer in the cluster metadata; retrying lookup.",
+                        brokerId.getAsInt(), spec.name);
+                    maybeSendRequests(driver, currentTimeMs);

Review Comment:
   https://github.com/apache/kafka/pull/22529/changes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org