You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "chickenchickenlove (via GitHub)" <gi...@apache.org> on 2025/08/16 15:11:23 UTC
[PR] MINOR: Fix race conditions in KafkaStreams when topic is missing. [kafka]
chickenchickenlove opened a new pull request, #20362:
URL: https://github.com/apache/kafka/pull/20362
### Background
There is race condition in `ConsumerNetworkThread` for `Kafka Streams`.
So it make `HandlingSourceTopicDeletionIntegrationTest#shouldThrowErrorAfterSourceTopicDeleted` be flaky.
Please refer to details in attached sequence diagram.
<img width="1208" height="805" alt="image" src="https://github.com/user-attachments/assets/e62ef7df-5ca8-448d-a1c9-f48d6734f25f" />
### Result
- Fixes flaky test `HandlingSourceTopicDeletionIntegrationTest#shouldThrowErrorAfterSourceTopicDeleted`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Fix race conditions in KafkaStreams when topic is missing. [kafka]
Posted by "chickenchickenlove (via GitHub)" <gi...@apache.org>.
chickenchickenlove commented on PR #20362:
URL: https://github.com/apache/kafka/pull/20362#issuecomment-3261189175
Gently Ping, could you take a look this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Fix race conditions in KafkaStreams when topic is missing. [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #20362:
URL: https://github.com/apache/kafka/pull/20362#issuecomment-3344295186
We are somewhat swamped recently. Sorry for the delay... Don't know when I will get to it. But please keeping nagging us about it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Fix race conditions in KafkaStreams when topic is missing. [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #20362:
URL: https://github.com/apache/kafka/pull/20362#issuecomment-3432525366
> The changes and scenario that I wrote is reasonable to you?
yes, I have no concern with the fix really, I just don't fully understand the issue (want us to make sure we're not covering an issue with the assignment passed to the task revocation)
Thanks for looking into it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Fix race conditions in KafkaStreams when topic is missing. [kafka]
Posted by "chickenchickenlove (via GitHub)" <gi...@apache.org>.
chickenchickenlove commented on PR #20362:
URL: https://github.com/apache/kafka/pull/20362#issuecomment-3206361546
@mjsax sorry to mention.
Could you take a look this PR?
It is related with Kafka streams!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Fix race conditions in KafkaStreams when topic is missing. [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20362:
URL: https://github.com/apache/kafka/pull/20362#issuecomment-3938125325
This PR has been closed since it has not had any activity in 120 days. If you feel like this
was a mistake, or you would like to continue working on it, please feel free to re-open the
PR and ask for a review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Fix race conditions in KafkaStreams when topic is missing. [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20362:
URL: https://github.com/apache/kafka/pull/20362#issuecomment-3231669616
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Fix race conditions in KafkaStreams when topic is missing. [kafka]
Posted by "chickenchickenlove (via GitHub)" <gi...@apache.org>.
chickenchickenlove commented on PR #20362:
URL: https://github.com/apache/kafka/pull/20362#issuecomment-3437134660
@lianetm
It looks like a commit masked the issue I was trying to fix.
1. I tested and debugged to solve this problem based on commit hash (986322dc36)
2. Currently, this PR based on commit hash (88616295e1)
In this branch, I reverted this PR(https://github.com/apache/kafka/commit/38e3359446963f23d1d068abcaa1eb8a51fc30cc) to check `HandlingSourceTopicDeletionIntegrationTest` can be success with `new consumer protocol`.
But, it never success.
However, If I tested it based on 986322dc36, `HandlingSourceTopicDeletionIntegrationTest` is always success.
In other words, one of the PRs between those two commits made the `HandlingSourceTopicDeletionIntegrationTest#shouldThrowErrorAfterSourceTopicDeleted` test permanently fail.
That is, it succeeds under the legacy protocol but fails under the new protocol.
To put it differently, in the legacy protocol, when a topic is deleted, all Stream Threads are expected to transition to the Error state, whereas in the new protocol, when a topic is deleted, all Stream Threads remain in the Running state.
Is this the expected behavior for the new protocol in Kafka Streams?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Fix race conditions in KafkaStreams when topic is missing. [kafka]
Posted by "chickenchickenlove (via GitHub)" <gi...@apache.org>.
chickenchickenlove commented on PR #20362:
URL: https://github.com/apache/kafka/pull/20362#issuecomment-3437167014
I think this commit affects.
https://github.com/apache/kafka/commit/03190e4c22ad18eaea7c96eb6e0c15b5dd3c3578
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Fix race conditions in KafkaStreams when topic is missing. [kafka]
Posted by "chickenchickenlove (via GitHub)" <gi...@apache.org>.
chickenchickenlove commented on PR #20362:
URL: https://github.com/apache/kafka/pull/20362#issuecomment-3432293813
@lianetm , Thanks for your feedback!
I’ll dig into it and let you know If I find something suspicious. 🙇♂️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Fix race conditions in KafkaStreams when topic is missing. [kafka]
Posted by "chickenchickenlove (via GitHub)" <gi...@apache.org>.
chickenchickenlove commented on PR #20362:
URL: https://github.com/apache/kafka/pull/20362#issuecomment-3432744327
I see!
Let me check!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Fix race conditions in KafkaStreams when topic is missing. [kafka]
Posted by "chickenchickenlove (via GitHub)" <gi...@apache.org>.
chickenchickenlove commented on PR #20362:
URL: https://github.com/apache/kafka/pull/20362#issuecomment-3437511246
There was happening 😅
In summary,
1. The issue could not be reproduced on the current PR’s base branch due to the previous commit 03190e4c22ad18eaea7c96eb6e0c15b5dd3c3578
2. However, after rebasing the base branch onto the latest trunk branch, the issue reappeared. When I reverted commit 38e3359446963f23d1d068abcaa1eb8a51fc30cc and tested again with the new protocol, all tests passed successfully.
So, Let me dig into more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Fix race conditions in KafkaStreams when topic is missing. [kafka]
Posted by "chickenchickenlove (via GitHub)" <gi...@apache.org>.
chickenchickenlove commented on PR #20362:
URL: https://github.com/apache/kafka/pull/20362#issuecomment-3439545991
@lianetm
I dug into more.
I think this problem has been solved by this PR. (https://github.com/apache/kafka/pull/19400)
Because, `StreamThread` don't send a `UNSUBSCRIBE` events anymore.
<img width="1537" height="397" alt="image" src="https://github.com/user-attachments/assets/ac056358-57f9-46e0-bcc3-f6b4f26198cc" />
So, Step 5 disappear, `ConsumerNetworkThread` will not handle `UNSUBSCRIBE` event.
Therefore, there is no race condition any more.
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Fix race conditions in KafkaStreams when topic is missing. [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20362:
URL: https://github.com/apache/kafka/pull/20362#issuecomment-3217711441
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Fix race conditions in KafkaStreams when topic is missing. [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #20362: MINOR: Fix race conditions in KafkaStreams when topic is missing.
URL: https://github.com/apache/kafka/pull/20362
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Fix race conditions in KafkaStreams when topic is missing. [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20362:
URL: https://github.com/apache/kafka/pull/20362#issuecomment-3349787155
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Fix race conditions in KafkaStreams when topic is missing. [kafka]
Posted by "chickenchickenlove (via GitHub)" <gi...@apache.org>.
chickenchickenlove commented on PR #20362:
URL: https://github.com/apache/kafka/pull/20362#issuecomment-3349720406
@mjsax Thanks for your comment even if you are in busy!
If you know persons who can review this PR, could you mention him/her ?
Thanks for your help and effort always! 🙇♂️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Fix race conditions in KafkaStreams when topic is missing. [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20362:
URL: https://github.com/apache/kafka/pull/20362#issuecomment-3264447808
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Fix race conditions in KafkaStreams when topic is missing. [kafka]
Posted by "chickenchickenlove (via GitHub)" <gi...@apache.org>.
chickenchickenlove commented on PR #20362:
URL: https://github.com/apache/kafka/pull/20362#issuecomment-3224362256
I’d greatly appreciate it if someone could kindly review this PR when they have a chance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Fix race conditions in KafkaStreams when topic is missing. [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20362:
URL: https://github.com/apache/kafka/pull/20362#issuecomment-3782389251
This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.
<p>
If you are having difficulty finding a reviewer, please reach out on the
[mailing list](https://kafka.apache.org/contact).
<p>
If this PR is no longer valid or desired, please feel free to close it. If no activity
occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Fix race conditions in KafkaStreams when topic is missing. [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on PR #20362:
URL: https://github.com/apache/kafka/pull/20362#issuecomment-3429733926
Hey @chickenchickenlove , thanks for looking in to this, interesting. Help me understand this issue better (the proposed change is harmless as I see it and could fix the flakiness, but want to make sure it is not hiding something else).
High level, from the consumer POV, the 2 sides involved happen in the background (so safe there):
- updating assignment when a reconciliation completes (topic delete)
- process unsubscribe
So how exactly do we end up in step 8??? Is it that the partitions passed to the unsubscribe here (taken from the `currentAssignment`) are not in the `subscriptionState` anymore?
https://github.com/apache/kafka/blob/05999c70c14ed998e4c0bc19ade1518bdc3df748/clients/src/main/java/org/apache/kafka/clients/consumer/internals/StreamsMembershipManager.java#L921
This bit it different between streams and the consumer mgrs actually (the partitions to revoked passed in the consumer are taken directly from the subscriptionState, not from the currentAssignment like the streamsMgr does)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org