You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2020/05/04 09:16:06 UTC
[GitHub] [kafka] dajac opened a new pull request #8609: KAFKA-9946; StopReplicaRequest deletePartition changes may cause premature topic deletion handling in controller
dajac opened a new pull request #8609:
URL: https://github.com/apache/kafka/pull/8609
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #8609: KAFKA-9946; StopReplicaRequest deletePartition changes may cause premature topic deletion handling in controller
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #8609:
URL: https://github.com/apache/kafka/pull/8609#issuecomment-624186102
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hachikuji commented on pull request #8609: KAFKA-9946; StopReplicaRequest deletePartition changes may cause premature topic deletion handling in controller
Posted by GitBox <gi...@apache.org>.
hachikuji commented on pull request #8609:
URL: https://github.com/apache/kafka/pull/8609#issuecomment-624221761
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hachikuji commented on pull request #8609: KAFKA-9946; StopReplicaRequest deletePartition changes may cause premature topic deletion handling in controller
Posted by GitBox <gi...@apache.org>.
hachikuji commented on pull request #8609:
URL: https://github.com/apache/kafka/pull/8609#issuecomment-624222236
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hachikuji commented on pull request #8609: KAFKA-9946; StopReplicaRequest deletePartition changes may cause premature topic deletion handling in controller
Posted by GitBox <gi...@apache.org>.
hachikuji commented on pull request #8609:
URL: https://github.com/apache/kafka/pull/8609#issuecomment-623696783
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on a change in pull request #8609: KAFKA-9946; StopReplicaRequest deletePartition changes may cause premature topic deletion handling in controller
Posted by GitBox <gi...@apache.org>.
dajac commented on a change in pull request #8609:
URL: https://github.com/apache/kafka/pull/8609#discussion_r419946491
##########
File path: core/src/main/scala/kafka/controller/ControllerChannelManager.scala
##########
@@ -590,7 +569,26 @@ abstract class AbstractControllerBrokerRequestBatch(config: KafkaConfig,
stateChangeLog.info(s"Sending StopReplica request for ${partitionStates.size} " +
s"replicas to broker $brokerId")
- sendStopReplicaRequest(brokerId, brokerEpoch, false, stopReplicaTopicState)
+ val stopReplicaRequestBuilder = new StopReplicaRequest.Builder(
+ stopReplicaRequestVersion, controllerId, controllerEpoch, brokerEpoch,
+ false, stopReplicaTopicState.values.toBuffer.asJava)
+
+ sendRequest(brokerId, stopReplicaRequestBuilder, (r: AbstractResponse) => {
+ val stopReplicaResponse = r.asInstanceOf[StopReplicaResponse]
Review comment:
I agree as well. Passing a function is a really good idea. I should have thought about it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lbradstreet commented on pull request #8609: KAFKA-9946; StopReplicaRequest deletePartition changes may cause premature topic deletion handling in controller
Posted by GitBox <gi...@apache.org>.
lbradstreet commented on pull request #8609:
URL: https://github.com/apache/kafka/pull/8609#issuecomment-623488567
I haven't reviewed the PR, but I can confirm that I no longer see the premature deletion behaviour that lead me to report KAFKA-9946.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on pull request #8609: KAFKA-9946; StopReplicaRequest deletePartition changes may cause premature topic deletion handling in controller
Posted by GitBox <gi...@apache.org>.
dajac commented on pull request #8609:
URL: https://github.com/apache/kafka/pull/8609#issuecomment-623928516
@lbradstreet @hachikuji Thanks for your feedback. I have updated the PR accordingly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dhruvilshah3 commented on a change in pull request #8609: KAFKA-9946; StopReplicaRequest deletePartition changes may cause premature topic deletion handling in controller
Posted by GitBox <gi...@apache.org>.
dhruvilshah3 commented on a change in pull request #8609:
URL: https://github.com/apache/kafka/pull/8609#discussion_r420238891
##########
File path: core/src/main/scala/kafka/controller/ControllerChannelManager.scala
##########
@@ -550,6 +550,22 @@ abstract class AbstractControllerBrokerRequestBatch(config: KafkaConfig,
else if (config.interBrokerProtocolVersion >= KAFKA_2_2_IV0) 1
else 0
+ def responseCallback(brokerId: Int, isPartitionDeleted: TopicPartition => Boolean)
+ (response: AbstractResponse): Unit = {
+ val stopReplicaResponse = response.asInstanceOf[StopReplicaResponse]
+ val partitionErrorsForDeletingTopics = mutable.Map.empty[TopicPartition, Errors]
+ stopReplicaResponse.partitionErrors.asScala.foreach { pe =>
+ val tp = new TopicPartition(pe.topicName, pe.partitionIndex)
+ if (controllerContext.isTopicDeletionInProgress(pe.topicName) &&
+ isPartitionDeleted(tp)) {
+ partitionErrorsForDeletingTopics += tp -> Errors.forCode(pe.errorCode)
Review comment:
nit: `partitionErrorsForDeletingTopics` seems a bit ambiguous and makes it sound like it only includes partitions for which StopReplicaRequest failed. Perhaps something like `partitionToError` is better?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lbradstreet commented on pull request #8609: KAFKA-9946; StopReplicaRequest deletePartition changes may cause premature topic deletion handling in controller
Posted by GitBox <gi...@apache.org>.
lbradstreet commented on pull request #8609:
URL: https://github.com/apache/kafka/pull/8609#issuecomment-624122999
@dajac thanks! Looks good to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hachikuji commented on a change in pull request #8609: KAFKA-9946; StopReplicaRequest deletePartition changes may cause premature topic deletion handling in controller
Posted by GitBox <gi...@apache.org>.
hachikuji commented on a change in pull request #8609:
URL: https://github.com/apache/kafka/pull/8609#discussion_r419803119
##########
File path: core/src/main/scala/kafka/controller/ControllerChannelManager.scala
##########
@@ -590,7 +569,26 @@ abstract class AbstractControllerBrokerRequestBatch(config: KafkaConfig,
stateChangeLog.info(s"Sending StopReplica request for ${partitionStates.size} " +
s"replicas to broker $brokerId")
- sendStopReplicaRequest(brokerId, brokerEpoch, false, stopReplicaTopicState)
+ val stopReplicaRequestBuilder = new StopReplicaRequest.Builder(
+ stopReplicaRequestVersion, controllerId, controllerEpoch, brokerEpoch,
+ false, stopReplicaTopicState.values.toBuffer.asJava)
+
+ sendRequest(brokerId, stopReplicaRequestBuilder, (r: AbstractResponse) => {
+ val stopReplicaResponse = r.asInstanceOf[StopReplicaResponse]
Review comment:
I agree the duplication is a tad vexing. Perhaps we could pass a function `TopicPartition -> Boolean` to the callback which tells whether deletion was requested. For the old version, we would return `true` blindly.
##########
File path: core/src/main/scala/kafka/controller/ControllerChannelManager.scala
##########
@@ -590,7 +569,26 @@ abstract class AbstractControllerBrokerRequestBatch(config: KafkaConfig,
stateChangeLog.info(s"Sending StopReplica request for ${partitionStates.size} " +
s"replicas to broker $brokerId")
- sendStopReplicaRequest(brokerId, brokerEpoch, false, stopReplicaTopicState)
+ val stopReplicaRequestBuilder = new StopReplicaRequest.Builder(
+ stopReplicaRequestVersion, controllerId, controllerEpoch, brokerEpoch,
+ false, stopReplicaTopicState.values.toBuffer.asJava)
+
+ sendRequest(brokerId, stopReplicaRequestBuilder, (r: AbstractResponse) => {
+ val stopReplicaResponse = r.asInstanceOf[StopReplicaResponse]
+ val partitionErrorsForDeletingTopics = mutable.Map.empty[TopicPartition, Errors]
+ stopReplicaResponse.partitionErrors.asScala.foreach { pe =>
+ val tp = new TopicPartition(pe.topicName, pe.partitionIndex)
+ // Verify that the topic deletion is in progress and
+ // that the request deleted the replica
+ if (controllerContext.isTopicDeletionInProgress(pe.topicName) &&
+ partitionStates.get(tp).exists(_.deletePartition)) {
Review comment:
nit: could we align with the `controllerContext`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lbradstreet commented on a change in pull request #8609: KAFKA-9946; StopReplicaRequest deletePartition changes may cause premature topic deletion handling in controller
Posted by GitBox <gi...@apache.org>.
lbradstreet commented on a change in pull request #8609:
URL: https://github.com/apache/kafka/pull/8609#discussion_r419725599
##########
File path: core/src/main/scala/kafka/controller/ControllerChannelManager.scala
##########
@@ -590,7 +569,26 @@ abstract class AbstractControllerBrokerRequestBatch(config: KafkaConfig,
stateChangeLog.info(s"Sending StopReplica request for ${partitionStates.size} " +
s"replicas to broker $brokerId")
- sendStopReplicaRequest(brokerId, brokerEpoch, false, stopReplicaTopicState)
+ val stopReplicaRequestBuilder = new StopReplicaRequest.Builder(
+ stopReplicaRequestVersion, controllerId, controllerEpoch, brokerEpoch,
+ false, stopReplicaTopicState.values.toBuffer.asJava)
+
+ sendRequest(brokerId, stopReplicaRequestBuilder, (r: AbstractResponse) => {
+ val stopReplicaResponse = r.asInstanceOf[StopReplicaResponse]
Review comment:
It seems like we could keep the response callback creation method in the previous PR if we passed in partitionStates to the callback builder, or alternately create a map of the partitions that you expect to be deleted. I think this would save the code duplication in creating nearly the same callback which I believe is being done to deal with the differences in the deletePartition schemas i.e. all partitions in request vs specific partitions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hachikuji commented on a change in pull request #8609: KAFKA-9946; StopReplicaRequest deletePartition changes may cause premature topic deletion handling in controller
Posted by GitBox <gi...@apache.org>.
hachikuji commented on a change in pull request #8609:
URL: https://github.com/apache/kafka/pull/8609#discussion_r420319459
##########
File path: core/src/main/scala/kafka/controller/ControllerChannelManager.scala
##########
@@ -550,6 +550,22 @@ abstract class AbstractControllerBrokerRequestBatch(config: KafkaConfig,
else if (config.interBrokerProtocolVersion >= KAFKA_2_2_IV0) 1
else 0
+ def responseCallback(brokerId: Int, isPartitionDeleted: TopicPartition => Boolean)
+ (response: AbstractResponse): Unit = {
+ val stopReplicaResponse = response.asInstanceOf[StopReplicaResponse]
+ val partitionErrorsForDeletingTopics = mutable.Map.empty[TopicPartition, Errors]
+ stopReplicaResponse.partitionErrors.asScala.foreach { pe =>
+ val tp = new TopicPartition(pe.topicName, pe.partitionIndex)
+ if (controllerContext.isTopicDeletionInProgress(pe.topicName) &&
+ isPartitionDeleted(tp)) {
+ partitionErrorsForDeletingTopics += tp -> Errors.forCode(pe.errorCode)
Review comment:
I think the name seems ok. To me it means that the map includes the errors of all topic being deleted. It might be nice if it could reflect that this is only covering partitions which were also requested to be deleted in the StopReplica request, but that name probably becomes unwieldy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org