You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "chickenchickenlove (via GitHub)" <gi...@apache.org> on 2026/01/19 14:09:31 UTC
[PR] KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted [kafka]
chickenchickenlove opened a new pull request, #21332:
URL: https://github.com/apache/kafka/pull/21332
### Description
This PR addresses KAFKA-17397 by ensuring deterministic behavior in `ClassicKafkaConsumer.close()` under interruption for the classic group protocol.
In CI, `PlaintextConsumerTest.testCloseLeavesGroupOnInterrupt()` can fail for `ClassicKafkaConsumer` because `LeaveGroupRequest` is sometimes blocked by `NetworkClient.isReady()` when a metadata update is due. Since isReady prioritizes metadata requests, the pending `LeaveGroupRequest` can remain unsent. When the calling thread is already interrupted, `ConsumerNetworkClient` may throw an `InterruptException` before the pending request gets a chance to be sent. This causes the member to leave only via `session.timeout.ms`, resulting in test flakiness.
### Fix
Allow `LeaveGroupRequest` to bypass the `metadata-update` gating in `NetworkClient.isReady()` (while still respecting `canSendRequest`), ensuring the request is sent during `close()` even when a metadata update is due.
### Sequence Diagram
<img width="7840" height="5910" alt="seq-1" src="https://github.com/user-attachments/assets/1849b465-fc9e-4b9b-a76a-66aeab17188e" />
- Steps 9, 10, and 11 describe non-deterministic behavior that prevents the `ClassicKafkaConsumer` from successfully sending a `LEAVE_GROUP` request when it is interrupted.
- This PR makes the sending of the LEAVE_GROUP request deterministic by bypassing the metadata update step specifically for LEAVE_GROUP. (See, Step 9, 15, 16, 17, 18)
### Fixes
- https://issues.apache.org/jira/browse/KAFKA-17397
- https://issues.apache.org/jira/browse/KAFKA-18031
- Flaky Test - https://develocity.apache.org/scans/tests?search.rootProjectNames=kafka&search.timeZoneId=Asia%2FTaipei&tests.container=kafka.api.PlaintextConsumerTest&tests.sortField=FLAKY&tests.test=testCloseLeavesGroupOnInterrupt(String)%5B1%5D
### In local re-produce
- With current trunk branch, the test failed 1~2 times out of 20 runs.
- With this PR, all 100 test runs succeeded (although there were a few build failures due to busy CPU and full memory.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #21332:
URL: https://github.com/apache/kafka/pull/21332#issuecomment-4305128934
@chickenchickenlove thanks for opening the ticket. We could close [KAFKA-17397](https://issues.apache.org/jira/browse/KAFKA-17397) as won't fix after resolving KAFKA-20522
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted [kafka]
Posted by "chickenchickenlove (via GitHub)" <gi...@apache.org>.
chickenchickenlove commented on PR #21332:
URL: https://github.com/apache/kafka/pull/21332#issuecomment-4109740098
@chia7712
Thanks for your suggestion! 🙇♂️
I’ll follow your guidance.
I have one question:
Should we close JIRA KAFKA-17397 as `WON'T FIX`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #21332:
URL: https://github.com/apache/kafka/pull/21332#issuecomment-3916889457
Hey @chickenchickenlove ! Sorry for the delay.
The current approach on both consumers is that the leave request on close is sent on a best-effort (e.g., not retried), so I'm not sure we can we really guarantee what this test is claiming, right? (e.g, what if the request is lost? or fails for other reasons on the broker side?)
On top of that, changing the logic that prioritized metadata request seems very risky. E.g, could we be filling up the max_in_flight with this leave request that we want to allow to go before metadata, so delaying metadata in new unintended ways? Not sure, needs more thought, but I wouldn't change this without a good motivation.
I wonder if our option is to test this at different levels, to test what we can guarantee:
- `consumer.close` should trigger the actions that are expected to send the leave even when interrupted (enqueue `LeaveGroupOnCloseEvent` for the async, `LeaveGroupRequest` for classic) - not sure if this is covered. If not, we could add tests in this same PR
- `LeaveGroupOnCloseEvent` processing should generate the HB to leave - this is tested in the membership mgr transitions on `leaveGroup` I expect
- remove this integration test for leave on interrupt :), we cannot really guarantee this imo
Then I do think we should review the best-effort approach to send the leave group. Can we do better and retry/wait while there is time? (not to address this interrupt scenario, here there's not much we can do, but to ensure we retry leaving when unsubscribing or close with time maybe). I had filed a jira for this a long time ago actually https://issues.apache.org/jira/browse/KAFKA-15954 , we can maybe resurface that separately.
What do you think? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #21332:
URL: https://github.com/apache/kafka/pull/21332#issuecomment-4106112980
@chickenchickenlove I suggest keeping the current behavior, but we can modify the verification in the flaky test to ensure it does try to send the request :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #21332:
URL: https://github.com/apache/kafka/pull/21332#issuecomment-4290910790
I agree with @lianetm. It is fine with me to replace it with a unit test, or we could check the log messages to ensure it tries to leave the group on a best-effort basis
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #21332:
URL: https://github.com/apache/kafka/pull/21332#issuecomment-4290830890
Hey @chickenchickenlove , so seems we agree there is no low-impact change we can do? How do we want to push this forward??
I agree with keeping the current leave behaviour as @chia7712 suggested, and then I don't see how we can guarantee anything at the integration test level, so wonder if we should consider removing the flaky IT and just ensure we test with unit tests. This flaky test is annoying, mostly because I totally agree with the flakiness :) to me this is testing something we cannot guarantee. Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted [kafka]
Posted by "chickenchickenlove (via GitHub)" <gi...@apache.org>.
chickenchickenlove commented on PR #21332:
URL: https://github.com/apache/kafka/pull/21332#issuecomment-4304931227
Hi, @lianetm @chia7712 !
I agree with your thoughts.
I created ticket for this (https://issues.apache.org/jira/browse/KAFKA-20522) and I’ll explore possible ways to improve it, whether by moving it to a unit test or by refining the integration test itself.
I have a question.
Should we close KAFKA-17397 as `Won't fix` or leave it as it is? should we also close this PR
Please let me know your thoughts! 🙇♂️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21332:
URL: https://github.com/apache/kafka/pull/21332#issuecomment-4317962118
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21332:
URL: https://github.com/apache/kafka/pull/21332#issuecomment-4301666205
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #21332:
URL: https://github.com/apache/kafka/pull/21332#issuecomment-3916922654
Oh and btw, I just noticed we have the same test in java and scala https://github.com/apache/kafka/blob/f568932e45dd055bc7b95b3f2f0e3ca09ca0ee57/clients/clients-integration-tests/src/test/java/org/apache/kafka/clients/consumer/PlaintextConsumerTest.java#L1547-L1571
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm closed pull request #21332: KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted
URL: https://github.com/apache/kafka/pull/21332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21332:
URL: https://github.com/apache/kafka/pull/21332#issuecomment-4115215311
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #21332:
URL: https://github.com/apache/kafka/pull/21332#issuecomment-4722379682
Hey here:
> We could close [KAFKA-17397](https://issues.apache.org/jira/browse/KAFKA-17397) as won't fix after resolving [KAFKA-20522](https://issues.apache.org/jira/browse/KAFKA-20522)
Both done, closing this. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21332:
URL: https://github.com/apache/kafka/pull/21332#issuecomment-3802902272
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted [kafka]
Posted by "chickenchickenlove (via GitHub)" <gi...@apache.org>.
chickenchickenlove commented on PR #21332:
URL: https://github.com/apache/kafka/pull/21332#issuecomment-3913040346
@lucasbru @lianetm
Gently ping and sorry to bother you again... 🙇♂️
When you get a chance, please take a look!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted [kafka]
Posted by "chickenchickenlove (via GitHub)" <gi...@apache.org>.
chickenchickenlove commented on PR #21332:
URL: https://github.com/apache/kafka/pull/21332#issuecomment-4090155704
@chia7712
Thanks for your comments! 🙇♂️
> Going back to this flaky test, instead of verifying the broker state, we could just check if the consumer intends to send the request. For example, we could verify an internal flag, or check the log output to ensure the leave group logic was triggered.
Sound good!
I’ll basically update this ticket for with the flaky test fix changes. (https://issues.apache.org/jira/browse/KAFKA-18031)
> It is still a best-effort policy, right? We could dig into improving this best-effort behavior in a separate Jira or KIP, since adding more retries/polls is a behavior change (we might even need a new config/flag for it).
Yes, AFAIK, it is still considered best-effort.
In most general cases, the current behavior appears to provide a reasonable level of best-effort, though in the scenario mentioned above, it may be somewhat insufficient. If you believe that this behavior should be improved, I would be happy to prepare a KIP as you suggested.
IMHO, although the Kafka Classic Consumer is expected to be deprecated in the 5.x line, such an improvement could still be meaningful for the 4.x line.
At the same time, since the issue would eventually be resolved automatically once `session.timeout.ms` expires, others may reasonably view it as a relatively minor issue and suggest that it may be preferable to leave the current behavior unchanged throughout the 4.x line.
I would be very interested to hear your thoughts on this point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted [kafka]
Posted by "chickenchickenlove (via GitHub)" <gi...@apache.org>.
chickenchickenlove commented on PR #21332:
URL: https://github.com/apache/kafka/pull/21332#issuecomment-3802360816
@lucasbru @lianetm
I’ve created a PR that fixes the root cause we identified in the issue you guys reported.
When you have bandwidth, could you please take a look? 🙇♂️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted [kafka]
Posted by "chickenchickenlove (via GitHub)" <gi...@apache.org>.
chickenchickenlove commented on PR #21332:
URL: https://github.com/apache/kafka/pull/21332#issuecomment-3918157708
@lianetm
Thanks a lot for the comment!
Everything you pointed out sounds reasonable to me.
I’d like to discuss a couple of points a bit further.
> On top of that, changing the logic that prioritized metadata request seems very risky. E.g, could we be filling up the max_in_flight with this leave request that we want to allow to go before metadata, so delaying metadata in new unintended ways? Not sure, needs more thought, but I wouldn't change this without a good motivation.
1. Whether we must wait for ApiKeys.LEAVE_GROUP when metadata is expired
From my perspective, the information we get from refreshing metadata doesn’t seem to be directly related to what ApiKeys.LEAVE_GROUP needs. That said, one possibility is that the group coordinator instance mapped to a brokerId could change, and in that case leaving might end up failing anyway, especially if it coincides with a channel disconnect/reconnect scenario.
Am I missing any dependency here?
2. Risk that LEAVE_GROUP consumes in_flight_request and delays METADATA
It doesn’t seem that the information obtained from METADATA is strongly related to what KafkaConsumer.close() needs. LEAVE_GROUP appears to require only the group coordinator (and its instance identifier), so my assumption was that even if LEAVE_GROUP temporarily occupies in-flight slots, it might not have a meaningful negative impact on the subsequent flow.
That said, as you mentioned, it’s possible that this introduces an unexpected delay path, so I’d appreciate your thoughts on whether this assumption is valid.
> Then I do think we should review the best-effort approach to send the leave group. Can we do better and retry/wait while there is time? (not to address this interrupt scenario, here there's not much we can do, but to ensure we retry leaving when unsubscribing or close with time maybe). I had filed a jira for this a long time ago actually https://issues.apache.org/jira/browse/KAFKA-15954 , we can maybe resurface that separately.
At the same time, it seems there may be room to mitigate the current behavior at a higher level. Right now, after sending LEAVE_GROUP from AbstractCoordinator, we call pollNoWakeUp only once, and it looks like this can cause LEAVE_GROUP to miss its chance to actually be sent when metadata happens to be expired at that moment.
https://github.com/apache/kafka/blob/f568932e45dd055bc7b95b3f2f0e3ca09ca0ee57/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java#L1170-L1188
For example, if AbstractCoordinator had a simple flag (or a similar signal) to indicate whether LEAVE_GROUP was sent/processed, we might be able to call pollNoWakeUp a few more times to increase the chances of sending it, or make leaving more reliable when there’s still time budget available. What do you think about this approach?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21332:
URL: https://github.com/apache/kafka/pull/21332#issuecomment-3924544816
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #21332:
URL: https://github.com/apache/kafka/pull/21332#issuecomment-4084016932
> a few more times to increase the chances of sending it
It is still a best-effort policy, right? We could dig into improving this best-effort behavior in a separate Jira or KIP, since adding more retries/polls is a behavior change (we might even need a new config/flag for it).
Going back to this flaky test, instead of verifying the broker state, we could just check if the consumer intends to send the request. For example, we could verify an internal flag, or check the log output to ensure the leave group logic was triggered.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17397: Ensure ClassicKafkaConsumer sends leave request on close even if interrupted [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21332:
URL: https://github.com/apache/kafka/pull/21332#issuecomment-4095281352
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org