You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2022/04/20 12:19:28 UTC

[GitHub] [kafka] dajac opened a new pull request, #12071: KAFKA-13790; ReplicaManager should be robust to all partition updates from kraft metadata log (1)

dajac opened a new pull request, #12071:
URL: https://github.com/apache/kafka/pull/12071

   This patch does some initial cleanups in the context of KAFKA-13790. Mainly, it renames `ZkVersion` field to `PartitionEpoch` in the `LeaderAndIsrRequest`, the `LeaderAndIsr` and the `Partition`.
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] hachikuji commented on a diff in pull request #12071: MINOR: Rename `ZkVersion` to `PartitionEpoch`

Posted by GitBox <gi...@apache.org>.
hachikuji commented on code in PR #12071:
URL: https://github.com/apache/kafka/pull/12071#discussion_r855451915


##########
clients/src/main/resources/common/message/LeaderAndIsrRequest.json:
##########
@@ -78,8 +78,8 @@
         "about": "The leader epoch." },
       { "name": "Isr", "type": "[]int32", "versions": "0+", "entityType": "brokerId",
         "about": "The in-sync replica IDs." },
-      { "name": "ZkVersion", "type": "int32", "versions": "0+",
-        "about": "The ZooKeeper version." },
+      { "name": "PartitionEpoch", "type": "int32", "versions": "0+",
+        "about": "The current epoch for the partition for KRaft controllers. The current ZK version for the legacy controllers." },

Review Comment:
   Maybe something like this:
   ```
   The current epoch for the partition. The epoch is a monotonically increasing value which is incremented after every partition change. (Since the LeaderAndIsr request is only used by the legacy controller, this corresponds to the zkVersion)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] dajac commented on a diff in pull request #12071: MINOR: Rename `ZkVersion` to `PartitionEpoch`

Posted by GitBox <gi...@apache.org>.
dajac commented on code in PR #12071:
URL: https://github.com/apache/kafka/pull/12071#discussion_r855444201


##########
clients/src/main/resources/common/message/LeaderAndIsrRequest.json:
##########
@@ -78,8 +78,8 @@
         "about": "The leader epoch." },
       { "name": "Isr", "type": "[]int32", "versions": "0+", "entityType": "brokerId",
         "about": "The in-sync replica IDs." },
-      { "name": "ZkVersion", "type": "int32", "versions": "0+",
-        "about": "The ZooKeeper version." },
+      { "name": "PartitionEpoch", "type": "int32", "versions": "0+",
+        "about": "The current epoch for the partition for KRaft controllers. The current ZK version for the legacy controllers." },

Review Comment:
   We can add the same comment to `kafka.api.LeaderAndIsr`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] dajac commented on a diff in pull request #12071: MINOR: Rename `ZkVersion` to `PartitionEpoch`

Posted by GitBox <gi...@apache.org>.
dajac commented on code in PR #12071:
URL: https://github.com/apache/kafka/pull/12071#discussion_r855443278


##########
core/src/main/scala/kafka/cluster/Partition.scala:
##########
@@ -251,8 +251,9 @@ class Partition(val topicPartition: TopicPartition,
 
   // lock to prevent the follower replica log update while checking if the log dir could be replaced with future log.
   private val futureLogLock = new Object()
-  private var zkVersion: Int = LeaderAndIsr.initialZKVersion
-  @volatile private var leaderEpoch: Int = LeaderAndIsr.initialLeaderEpoch - 1
+  // The current epoch for the partition for KRaft controllers. The current ZK version for the legacy controllers.
+  private var partitionEpoch: Int = LeaderAndIsr.InitialPartitionEpoch

Review Comment:
   It seems that `partitionEpoch` is always accessed within the `leaderIsrUpdateLock` lock whereas `leaderEpoch` is also accessed by the `ReplicaManager`. Do you think that we should make it volatile to be consistent?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] dajac commented on pull request #12071: MINOR: Rename `ZkVersion` to `PartitionEpoch`

Posted by GitBox <gi...@apache.org>.
dajac commented on PR #12071:
URL: https://github.com/apache/kafka/pull/12071#issuecomment-1105624346

   Thanks @hachikuji. I have addressed your comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] hachikuji commented on a diff in pull request #12071: MINOR: Rename `ZkVersion` to `PartitionEpoch`

Posted by GitBox <gi...@apache.org>.
hachikuji commented on code in PR #12071:
URL: https://github.com/apache/kafka/pull/12071#discussion_r855451915


##########
clients/src/main/resources/common/message/LeaderAndIsrRequest.json:
##########
@@ -78,8 +78,8 @@
         "about": "The leader epoch." },
       { "name": "Isr", "type": "[]int32", "versions": "0+", "entityType": "brokerId",
         "about": "The in-sync replica IDs." },
-      { "name": "ZkVersion", "type": "int32", "versions": "0+",
-        "about": "The ZooKeeper version." },
+      { "name": "PartitionEpoch", "type": "int32", "versions": "0+",
+        "about": "The current epoch for the partition for KRaft controllers. The current ZK version for the legacy controllers." },

Review Comment:
   Maybe something like this:
   
   > The current epoch for the partition. The epoch is a monotonically increasing value which is incremented after every partition change. (Since the LeaderAndIsr request is only used by the legacy controller, this corresponds to the zkVersion)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] dengziming commented on a diff in pull request #12071: KAFKA-13790; ReplicaManager should be robust to all partition updates from kraft metadata log (1)

Posted by GitBox <gi...@apache.org>.
dengziming commented on code in PR #12071:
URL: https://github.com/apache/kafka/pull/12071#discussion_r854082813


##########
clients/src/main/resources/common/message/LeaderAndIsrRequest.json:
##########
@@ -78,8 +78,8 @@
         "about": "The leader epoch." },
       { "name": "Isr", "type": "[]int32", "versions": "0+", "entityType": "brokerId",
         "about": "The in-sync replica IDs." },
-      { "name": "ZkVersion", "type": "int32", "versions": "0+",
-        "about": "The ZooKeeper version." },
+      { "name": "PartitionEpoch", "type": "int32", "versions": "0+",
+        "about": "The current epoch for the partition for KRaft controllers. The current ZK version for the legacy controllers." },

Review Comment:
   It seems LeaderAndIsrRequest is only used in ZkController and will be removed after KIP-500, will we use it in KRaft mode?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] hachikuji commented on a diff in pull request #12071: MINOR: Rename `ZkVersion` to `PartitionEpoch`

Posted by GitBox <gi...@apache.org>.
hachikuji commented on code in PR #12071:
URL: https://github.com/apache/kafka/pull/12071#discussion_r855450737


##########
core/src/main/scala/kafka/cluster/Partition.scala:
##########
@@ -251,8 +251,9 @@ class Partition(val topicPartition: TopicPartition,
 
   // lock to prevent the follower replica log update while checking if the log dir could be replaced with future log.
   private val futureLogLock = new Object()
-  private var zkVersion: Int = LeaderAndIsr.initialZKVersion
-  @volatile private var leaderEpoch: Int = LeaderAndIsr.initialLeaderEpoch - 1
+  // The current epoch for the partition for KRaft controllers. The current ZK version for the legacy controllers.
+  private var partitionEpoch: Int = LeaderAndIsr.InitialPartitionEpoch

Review Comment:
   Either that or we should try to document the expectation that we have the lock. One thing that makes me a little uncomfortable in regard to these volatile fields is any expectation of consistency between them. Maybe a potential refactor for a subsequent patch could be to pull these fields into `PartitionState` so that the state can be updated atomically. What do you think?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] dajac commented on a diff in pull request #12071: KAFKA-13790; ReplicaManager should be robust to all partition updates from kraft metadata log (1)

Posted by GitBox <gi...@apache.org>.
dajac commented on code in PR #12071:
URL: https://github.com/apache/kafka/pull/12071#discussion_r854191977


##########
clients/src/main/resources/common/message/LeaderAndIsrRequest.json:
##########
@@ -78,8 +78,8 @@
         "about": "The leader epoch." },
       { "name": "Isr", "type": "[]int32", "versions": "0+", "entityType": "brokerId",
         "about": "The in-sync replica IDs." },
-      { "name": "ZkVersion", "type": "int32", "versions": "0+",
-        "about": "The ZooKeeper version." },
+      { "name": "PartitionEpoch", "type": "int32", "versions": "0+",
+        "about": "The current epoch for the partition for KRaft controllers. The current ZK version for the legacy controllers." },

Review Comment:
   That's right. However, we use `LeaderAndIsrPartitionState` in many places internally in the ReplicaManager and in the Partition so it makes sense to unify the naming at this level, I think.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] hachikuji commented on pull request #12071: MINOR: Rename `ZkVersion` to `PartitionEpoch`

Posted by GitBox <gi...@apache.org>.
hachikuji commented on PR #12071:
URL: https://github.com/apache/kafka/pull/12071#issuecomment-1105463549

   cc @jsancio 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] dajac commented on a diff in pull request #12071: MINOR: Rename `ZkVersion` to `PartitionEpoch`

Posted by GitBox <gi...@apache.org>.
dajac commented on code in PR #12071:
URL: https://github.com/apache/kafka/pull/12071#discussion_r855453938


##########
core/src/main/scala/kafka/cluster/Partition.scala:
##########
@@ -251,8 +251,9 @@ class Partition(val topicPartition: TopicPartition,
 
   // lock to prevent the follower replica log update while checking if the log dir could be replaced with future log.
   private val futureLogLock = new Object()
-  private var zkVersion: Int = LeaderAndIsr.initialZKVersion
-  @volatile private var leaderEpoch: Int = LeaderAndIsr.initialLeaderEpoch - 1
+  // The current epoch for the partition for KRaft controllers. The current ZK version for the legacy controllers.
+  private var partitionEpoch: Int = LeaderAndIsr.InitialPartitionEpoch

Review Comment:
   Yeah, that makes sense. I will make it volatile for now. I have filled https://issues.apache.org/jira/browse/KAFKA-13844 for the refactor. I will take care of this as a follow up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] dajac commented on pull request #12071: MINOR: Rename `ZkVersion` to `PartitionEpoch`

Posted by GitBox <gi...@apache.org>.
dajac commented on PR #12071:
URL: https://github.com/apache/kafka/pull/12071#issuecomment-1106066340

   Thanks @hachikuji. I have addressed your comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] hachikuji commented on a diff in pull request #12071: MINOR: Rename `ZkVersion` to `PartitionEpoch`

Posted by GitBox <gi...@apache.org>.
hachikuji commented on code in PR #12071:
URL: https://github.com/apache/kafka/pull/12071#discussion_r855689378


##########
core/src/test/scala/unit/kafka/zk/KafkaZkClientTest.scala:
##########
@@ -922,10 +922,10 @@ class KafkaZkClientTest extends QuorumTestHarness {
   private def leaderIsrAndControllerEpochs(state: Int, zkVersion: Int): Map[TopicPartition, LeaderIsrAndControllerEpoch] =

Review Comment:
   nit: `zkVersion` -> `partitionEpoch`?



##########
core/src/test/scala/unit/kafka/utils/TestUtils.scala:
##########
@@ -1277,7 +1277,7 @@ object TestUtils extends Logging {
     def completeIsrUpdate(newZkVersion: Int): Unit = {

Review Comment:
   Should we change `newZkVersion` to `newPartitionEpoch`?



##########
core/src/test/scala/unit/kafka/controller/ReplicaStateMachineTest.scala:
##########
@@ -213,7 +213,7 @@ class ReplicaStateMachineTest {
 
     val stat = new Stat(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)
     val adjustedLeaderAndIsr = leaderAndIsr.newLeaderAndIsr(LeaderAndIsr.NoLeader, List(otherBrokerId))
-    val updatedLeaderAndIsr = adjustedLeaderAndIsr.withZkVersion(adjustedLeaderAndIsr .zkVersion + 1)
+    val updatedLeaderAndIsr = adjustedLeaderAndIsr.withPartitionEpoch(adjustedLeaderAndIsr .partitionEpoch + 1)

Review Comment:
   nit: looks like an extra space before the `.partitionEpoch`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] hachikuji commented on a diff in pull request #12071: MINOR: Rename `ZkVersion` to `PartitionEpoch`

Posted by GitBox <gi...@apache.org>.
hachikuji commented on code in PR #12071:
URL: https://github.com/apache/kafka/pull/12071#discussion_r855411055


##########
core/src/main/scala/kafka/cluster/Partition.scala:
##########
@@ -251,8 +251,9 @@ class Partition(val topicPartition: TopicPartition,
 
   // lock to prevent the follower replica log update while checking if the log dir could be replaced with future log.
   private val futureLogLock = new Object()
-  private var zkVersion: Int = LeaderAndIsr.initialZKVersion
-  @volatile private var leaderEpoch: Int = LeaderAndIsr.initialLeaderEpoch - 1
+  // The current epoch for the partition for KRaft controllers. The current ZK version for the legacy controllers.
+  private var partitionEpoch: Int = LeaderAndIsr.InitialPartitionEpoch

Review Comment:
   Do you know why this field is not volatile like `leaderEpoch`. 



##########
clients/src/main/resources/common/message/LeaderAndIsrRequest.json:
##########
@@ -78,8 +78,8 @@
         "about": "The leader epoch." },
       { "name": "Isr", "type": "[]int32", "versions": "0+", "entityType": "brokerId",
         "about": "The in-sync replica IDs." },
-      { "name": "ZkVersion", "type": "int32", "versions": "0+",
-        "about": "The ZooKeeper version." },
+      { "name": "PartitionEpoch", "type": "int32", "versions": "0+",
+        "about": "The current epoch for the partition for KRaft controllers. The current ZK version for the legacy controllers." },

Review Comment:
   nit: It does seem a little misleading to mention KRaft here since `LeaderAndIsr` is not used. Maybe we could somehow call that out?
   
   More generally, do you know if there is a good place we can document assumptions about the partition epoch? Mainly looking for a good place we can document that it is a monotonically increasing value which is incremented after every partition change. Maybe `kafka.api.LeaderAndIsr`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] hachikuji commented on a diff in pull request #12071: MINOR: Rename `ZkVersion` to `PartitionEpoch`

Posted by GitBox <gi...@apache.org>.
hachikuji commented on code in PR #12071:
URL: https://github.com/apache/kafka/pull/12071#discussion_r855395294


##########
clients/src/main/resources/common/message/LeaderAndIsrRequest.json:
##########
@@ -78,8 +78,8 @@
         "about": "The leader epoch." },
       { "name": "Isr", "type": "[]int32", "versions": "0+", "entityType": "brokerId",
         "about": "The in-sync replica IDs." },
-      { "name": "ZkVersion", "type": "int32", "versions": "0+",
-        "about": "The ZooKeeper version." },
+      { "name": "PartitionEpoch", "type": "int32", "versions": "0+",
+        "about": "The current epoch for the partition for KRaft controllers. The current ZK version for the legacy controllers." },

Review Comment:
   nit: It does seem a little misleading to mention KRaft here since the `LeaderAndIsr` API is not used. Maybe we could somehow call that out?
   
   More generally, do you know if there is a good place we can document assumptions about the partition epoch? Mainly looking for a good place we can document that it is a monotonically increasing value which is incremented after every partition change. Maybe `kafka.api.LeaderAndIsr`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] dajac commented on a diff in pull request #12071: MINOR: Rename `ZkVersion` to `PartitionEpoch`

Posted by GitBox <gi...@apache.org>.
dajac commented on code in PR #12071:
URL: https://github.com/apache/kafka/pull/12071#discussion_r855440057


##########
clients/src/main/resources/common/message/LeaderAndIsrRequest.json:
##########
@@ -78,8 +78,8 @@
         "about": "The leader epoch." },
       { "name": "Isr", "type": "[]int32", "versions": "0+", "entityType": "brokerId",
         "about": "The in-sync replica IDs." },
-      { "name": "ZkVersion", "type": "int32", "versions": "0+",
-        "about": "The ZooKeeper version." },
+      { "name": "PartitionEpoch", "type": "int32", "versions": "0+",
+        "about": "The current epoch for the partition for KRaft controllers. The current ZK version for the legacy controllers." },

Review Comment:
   I do agree that mentioning KRaft is misleading. How about the following `The current epoch for the partition. The epoch is a monotonically increasing value which is incremented after every partition change.` for the about field?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] dajac merged pull request #12071: MINOR: Rename `ZkVersion` to `PartitionEpoch`

Posted by GitBox <gi...@apache.org>.
dajac merged PR #12071:
URL: https://github.com/apache/kafka/pull/12071


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org