You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "devtrace404 (via GitHub)" <gi...@apache.org> on 2025/12/21 01:14:40 UTC
[PR] KAFKA-16799 Add exponential backoff to NetworkClientDelegate when node is not ready [kafka]
devtrace404 opened a new pull request, #21198:
URL: https://github.com/apache/kafka/pull/21198
### Problem
During stress testing, AsyncKafkaConsumer's NetworkClientDelegate did not back off when a node was not ready, causing excessive "Node is not ready" log messages. The Network client retried immediately, leading to log spam and unnecessary CPU usage.
### Solution
Implements exponential backoff in NetworkClientDelegate to prevent immediate retries when a node is not ready:
1. Tracks backoff state per node: Uses a `ConcurrentHashMap<Node, BackoffState>` to maintain separate backoff state for each node, allowing independent backoff tracking.
2. Applies exponential backoff: When client.ready(node) returns false, the client now:
- Records the failure using `recordNodeNotReady()`
- Calculates the next retry time using ExponentialBackoff with the configured retry.backoff.ms (default 100ms) and retry.backoff.max.ms (default 1000ms)
- Skips retry attempts until the backoff period has elapsed
3. Resets backoff on success: When a request is successfully sent to a node, the backoff state for that node is reset, ensuring the next failure starts with the initial backoff interval.
### Implementation Details
#### NetworkClientDelegate.java:
1. Added fields: `retryBackoffMs`, `retryBackoffMaxMs`, `exponentialBackoff`, `nodeBackoffStates`
2. Modified `doSend()` to check backoff delay before sending and record failures
3. Added helper methods: `getBackoffDelay()`, `recordNodeNotReady()`, `resetBackoff()`
4. Added BackoffState inner class to track attempt count and next retry time per node
#### NetworkClientDelegateTest.java:
1. Added testExponentialBackoffReducesRetryAttemptsWhenNodeNotReady(): Verifies backoff reduces retry attempts and validates exponential timing sequence (100ms, 200ms, 400ms, 800ms, capped at 1000ms)
2. Added testBackoffResetsAfterSuccessfulSend(): Ensures backoff state resets after successful send
3. Updated newNetworkClientDelegate() helper to set default backoff properties for testing
### Configuration
The backoff behavior uses existing consumer configuration:
retry.backoff.ms (default: 100ms): Initial backoff interval
retry.backoff.max.ms (default: 1000ms): Maximum backoff interval
The exponential backoff uses a base of 2 with 20% jitter, as defined in CommonClientConfigs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16799: Reduce log level for 'Node is not ready' message to TRACE [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21198:
URL: https://github.com/apache/kafka/pull/21198#issuecomment-4531655481
This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.
<p>
If you are having difficulty finding a reviewer, please reach out on the
[mailing list](https://kafka.apache.org/contact).
<p>
If this PR is no longer valid or desired, please feel free to close it. If no activity
occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16799 Add exponential backoff to NetworkClientDelegate when node is not ready [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on PR #21198:
URL: https://github.com/apache/kafka/pull/21198#issuecomment-3705592496
@devtrace404 Thanks for taking on this issue. Please can you assign https://issues.apache.org/jira/browse/KAFKA-16799 to yourself in JIRA and mark it as in-progress so we can see it's being worked on. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16799: Reduce log level for 'Node is not ready' message to TRACE [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21198:
URL: https://github.com/apache/kafka/pull/21198#issuecomment-4796000226
This PR has been closed since it has not had any activity in 120 days. If you feel like this
was a mistake, or you would like to continue working on it, please feel free to re-open the
PR and ask for a review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16799 Add exponential backoff to NetworkClientDelegate when node is not ready [kafka]
Posted by "devtrace404 (via GitHub)" <gi...@apache.org>.
devtrace404 commented on PR #21198:
URL: https://github.com/apache/kafka/pull/21198#issuecomment-3795808265
> @devtrace404 Thanks for taking on this issue. Please can you assign https://issues.apache.org/jira/browse/KAFKA-16799 to yourself in JIRA and mark it as in-progress so we can see it's being worked on. Thanks.
I don't have permission to Edit or re-assign. Is there a way to request the permission?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16799 Add exponential backoff to NetworkClientDelegate when node is not ready [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21198:
URL: https://github.com/apache/kafka/pull/21198#issuecomment-3694421120
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16799 Add exponential backoff to NetworkClientDelegate when node is not ready [kafka]
Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on PR #21198:
URL: https://github.com/apache/kafka/pull/21198#issuecomment-3721465740
From the PR description:
> The Network client retried immediately, leading to log spam and unnecessary CPU usage.
I'd love to know more detail about the CPU usage spike. How large was it? Is the CPU load usage increase _caused_ by the excess logging, or are they separate concerns?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16799 Add exponential backoff to NetworkClientDelegate when node is not ready [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #21198:
URL: https://github.com/apache/kafka/pull/21198#discussion_r2662028745
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/NetworkClientDelegate.java:
##########
@@ -221,13 +238,24 @@ boolean doSend(final UnsentRequest r, final long currentTimeMs) {
log.debug("No broker available to send the request: {}. Retrying.", r);
return false;
}
+
+ // Check if we're still in backoff period for this node
+ long backoffDelay = getBackoffDelay(node, currentTimeMs);
+ if (backoffDelay > 0) {
+ // Still in backoff period, skip retry
+ return false;
Review Comment:
This change means that we won't be checking the client readiness on every poll of the network client anymore, right? I'm not sure that would be right because client readiness to send can indeed change on any poll, and we don't want to add unneeded delays in sending requests if the node becomes ready (before the backoff expires)
Node readiness is about 2 things: being connected and being able to send more requests (depending on in-flights and metadata requests, which are prioritized based on this check). So on any call to poll, a client that was not ready may become ready (ex. not ready because too many in-flights or metadata needed + poll => ready).
I do see that the original Jira suggested backoff because of the noisy log "Node is not ready" showing continuously, it's confusing. The continuous check on client readiness, without backoff, is something we do want imo (we could review the log level if we think it's too much). Thoughts? (I could be missing something)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16799: Reduce log level for 'Node is not ready' message to TRACE [kafka]
Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on PR #21198:
URL: https://github.com/apache/kafka/pull/21198#issuecomment-3948322969
> > @devtrace404 Thanks for taking on this issue. Please can you assign [issues.apache.org/jira/browse/KAFKA-16799](https://issues.apache.org/jira/browse/KAFKA-16799) to yourself in JIRA and mark it as in-progress so we can see it's being worked on. Thanks.
>
> I don't have permission to Edit or re-assign. Is there a way to request the permission?
Please see [the instructions](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=50859233#KafkaImprovementProposals-Started) on how to self-assign Jira tickets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16799: Reduce log level for 'Node is not ready' message to TRACE [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #21198: KAFKA-16799: Reduce log level for 'Node is not ready' message to TRACE
URL: https://github.com/apache/kafka/pull/21198
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16799: Reduce log level for 'Node is not ready' message to TRACE [kafka]
Posted by "devtrace404 (via GitHub)" <gi...@apache.org>.
devtrace404 commented on code in PR #21198:
URL: https://github.com/apache/kafka/pull/21198#discussion_r2724783913
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/NetworkClientDelegate.java:
##########
@@ -221,13 +238,24 @@ boolean doSend(final UnsentRequest r, final long currentTimeMs) {
log.debug("No broker available to send the request: {}. Retrying.", r);
return false;
}
+
+ // Check if we're still in backoff period for this node
+ long backoffDelay = getBackoffDelay(node, currentTimeMs);
+ if (backoffDelay > 0) {
+ // Still in backoff period, skip retry
+ return false;
Review Comment:
@kirktrue @lianetm That is a fair point. Since the `ready()` method returns false for various conditions (metadata update in progress, too many in-flight requests, connecting state) and these shouldn't all be treated uniformly with backoff. They could be transient and must not have delay. I have eliminated the log spam with TRACE logs. Let me know if that makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16799 Add exponential backoff to NetworkClientDelegate when node is not ready [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #21198:
URL: https://github.com/apache/kafka/pull/21198#discussion_r2662028745
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/NetworkClientDelegate.java:
##########
@@ -221,13 +238,24 @@ boolean doSend(final UnsentRequest r, final long currentTimeMs) {
log.debug("No broker available to send the request: {}. Retrying.", r);
return false;
}
+
+ // Check if we're still in backoff period for this node
+ long backoffDelay = getBackoffDelay(node, currentTimeMs);
+ if (backoffDelay > 0) {
+ // Still in backoff period, skip retry
+ return false;
Review Comment:
This change means that we won't be checking the client readiness on every poll of the network client anymore, right? I'm not sure that would be right because client readiness to send can indeed change on any poll, and we don't want to add unneeded delays in sending requests if the node becomes ready (before the backoff expires)
Node readiness is about 2 things: being connected and being able to send more requests (depending on in-flights and metadata requests, which are prioritized based on this check). So on any call to poll, a node that was not ready may become ready (ex. not ready because too many in-flights or metadata needed + poll => ready).
I do see that the original Jira suggested backoff because of the noisy log "Node is not ready" showing continuously, it's confusing. The continuous check on client readiness, without backoff, is something we do want imo (we could review the log level if we think it's too much). Thoughts? (I could be missing something)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16799 Add exponential backoff to NetworkClientDelegate when node is not ready [kafka]
Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #21198:
URL: https://github.com/apache/kafka/pull/21198#discussion_r2670522553
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/NetworkClientDelegate.java:
##########
@@ -221,13 +238,24 @@ boolean doSend(final UnsentRequest r, final long currentTimeMs) {
log.debug("No broker available to send the request: {}. Retrying.", r);
return false;
}
+
+ // Check if we're still in backoff period for this node
+ long backoffDelay = getBackoffDelay(node, currentTimeMs);
+ if (backoffDelay > 0) {
+ // Still in backoff period, skip retry
+ return false;
Review Comment:
`ready()` has several conditions in which it could return `false`. For some of them (e.g. connecting state), it may make sense to back off, but for others (e.g. busy sending a metadata request), not so much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org