You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/02/03 02:02:50 UTC
[GitHub] [kafka] rhauch opened a new pull request #10032: KAFKA-12270: Handle race condition when Connect tasks attempt to create topics
rhauch opened a new pull request #10032:
URL: https://github.com/apache/kafka/pull/10032
_**This is option #2 for the fix, which is a bit more ideal since no additional admin client calls are made. See #10027 for option 1 that is simpler but results in additional admin client calls.**_
When a source connector is configured to create missing topics has multiple tasks that generate records for the same topic, it is possible that multiple tasks may simultaneously describe the topic, find it does not exist, and attempt to create the task. One of those create topic requests will succeed, and the other concurrent tasks will receive the response from the topic admin as having not created the task and will fail unnecessarily.
This change corrects the logic by moving the `TopicAdmin` logic to create a topic to a new `createOrFindTopics(…)` method that returns the set of created topic names and the set of existing topic names. This allows the existing `createTopics(…)` and `createTopic(…)` methods to retain the same behavior, but also allows the `WorkerSourceTask` to know from its single call to this new method whether the topic was created or was found to exist.
This adds one unit test and modifies several unit tests in `WorkerSourceTaskWithTopicCreationTest` that use mocks to verify the behavior, and modifies several existing unit tests for `TopicAdminTest` to ensure the logic of the new method is as expected.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rhauch commented on pull request #10032: KAFKA-12270: Handle race condition when Connect tasks attempt to create topics
Posted by GitBox <gi...@apache.org>.
rhauch commented on pull request #10032:
URL: https://github.com/apache/kafka/pull/10032#issuecomment-772702453
We're having infrastructure and build issues. I rebased this on the latest from `trunk`, which includes #10038 to disable Travis builds. No other changes were made since approval.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rhauch merged pull request #10032: KAFKA-12270: Handle race condition when Connect tasks attempt to create topics
Posted by GitBox <gi...@apache.org>.
rhauch merged pull request #10032:
URL: https://github.com/apache/kafka/pull/10032
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rhauch commented on pull request #10032: KAFKA-12270: Handle race condition when Connect tasks attempt to create topics
Posted by GitBox <gi...@apache.org>.
rhauch commented on pull request #10032:
URL: https://github.com/apache/kafka/pull/10032#issuecomment-772869974
JDK 11 passed, but JDK 8 and 15 each failed on spurious and seemingly unrelated tests (e.g., rebalance failure conditions, timeouts).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org