You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2022/10/13 08:02:52 UTC
[GitHub] [kafka] dajac opened a new pull request, #12741: KAFKA-14296; Partition leaders are not demoted during kraft controlled shutdown
dajac opened a new pull request, #12741:
URL: https://github.com/apache/kafka/pull/12741
When the `BrokerServer` starts its shutting down process, it transitions to `SHUTTING_DOWN` and sets `isShuttingDown` to `true`. With this state change, the follower state changes are short-cutted. This means that a broker which was serving as leader would remain acting as a leader until controlled shutdown completes. Instead, we want the leader and ISR state to be updated so that requests will return NOT_LEADER and the client can find the new leader.
We missed this case while implementing https://github.com/apache/kafka/pull/12187.
This patch fixes the issue and updates an existing test to ensure that `isShuttingDown` has not effect. We should consider adding integration tests for this as well. We can do this separately.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on a diff in pull request #12741: KAFKA-14296; Partition leaders are not demoted during kraft controlled shutdown
Posted by GitBox <gi...@apache.org>.
ijuma commented on code in PR #12741:
URL: https://github.com/apache/kafka/pull/12741#discussion_r994648305
##########
core/src/main/scala/kafka/server/ReplicaManager.scala:
##########
@@ -2151,28 +2150,24 @@ class ReplicaManager(val config: KafkaConfig,
try {
followerTopicSet.add(tp.topic)
- if (shuttingDown) {
- stateChangeLogger.trace(s"Unable to start fetching $tp with topic " +
- s"ID ${info.topicId} because the replica manager is shutting down.")
- } else {
- // We always update the follower state.
- // - This ensure that a replica with no leader can step down;
- // - This also ensures that the local replica is created even if the leader
- // is unavailable. This is required to ensure that we include the partition's
- // high watermark in the checkpoint file (see KAFKA-1647).
- val state = info.partition.toLeaderAndIsrPartitionState(tp, isNew)
- val isNewLeaderEpoch = partition.makeFollower(state, offsetCheckpoints, Some(info.topicId))
-
- if (isInControlledShutdown && (info.partition.leader == NO_LEADER ||
Review Comment:
Similarly, why do we need this special field versus relying on the broker state as I mentioned above? This kind of approach is extremely brittle in my opinion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on a diff in pull request #12741: KAFKA-14296; Partition leaders are not demoted during kraft controlled shutdown
Posted by GitBox <gi...@apache.org>.
dajac commented on code in PR #12741:
URL: https://github.com/apache/kafka/pull/12741#discussion_r994698379
##########
core/src/main/scala/kafka/server/ReplicaManager.scala:
##########
@@ -2151,28 +2150,24 @@ class ReplicaManager(val config: KafkaConfig,
try {
followerTopicSet.add(tp.topic)
- if (shuttingDown) {
- stateChangeLogger.trace(s"Unable to start fetching $tp with topic " +
- s"ID ${info.topicId} because the replica manager is shutting down.")
- } else {
- // We always update the follower state.
- // - This ensure that a replica with no leader can step down;
- // - This also ensures that the local replica is created even if the leader
- // is unavailable. This is required to ensure that we include the partition's
- // high watermark in the checkpoint file (see KAFKA-1647).
- val state = info.partition.toLeaderAndIsrPartitionState(tp, isNew)
- val isNewLeaderEpoch = partition.makeFollower(state, offsetCheckpoints, Some(info.topicId))
-
- if (isInControlledShutdown && (info.partition.leader == NO_LEADER ||
Review Comment:
We want to do this only when we are in controlled shutdown and it could be disabled. If you look at the caller side, we aligned this on how we do it for the `lifecycleManager` as both go together. We could revise this though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hachikuji commented on a diff in pull request #12741: KAFKA-14296; Partition leaders are not demoted during kraft controlled shutdown
Posted by GitBox <gi...@apache.org>.
hachikuji commented on code in PR #12741:
URL: https://github.com/apache/kafka/pull/12741#discussion_r994891612
##########
core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala:
##########
@@ -2210,7 +2210,8 @@ class ReplicaManagerTest {
aliveBrokerIds: Seq[Int] = Seq(0, 1),
propsModifier: Properties => Unit = _ => {},
mockReplicaFetcherManager: Option[ReplicaFetcherManager] = None,
- mockReplicaAlterLogDirsManager: Option[ReplicaAlterLogDirsManager] = None
+ mockReplicaAlterLogDirsManager: Option[ReplicaAlterLogDirsManager] = None,
+ isShuttingDown: AtomicBoolean = new AtomicBoolean(false)
Review Comment:
I found the use of the field a bit confusing. We don't necessarily have to fix it here, but what do you think about letting `ReplicaManager` own its own shutdown? Basically create a private field which serves the same purpose and then expose a method to explicitly toggle it. It is not obvious when looking at `BrokerServer` that altering the local field has this remote effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on a diff in pull request #12741: KAFKA-14296; Partition leaders are not demoted during kraft controlled shutdown
Posted by GitBox <gi...@apache.org>.
ijuma commented on code in PR #12741:
URL: https://github.com/apache/kafka/pull/12741#discussion_r994646630
##########
core/src/main/scala/kafka/server/ReplicaManager.scala:
##########
@@ -2142,7 +2142,6 @@ class ReplicaManager(val config: KafkaConfig,
): Unit = {
stateChangeLogger.info(s"Transitioning ${localFollowers.size} partition(s) to " +
"local followers.")
- val shuttingDown = isShuttingDown.get()
Review Comment:
This code is a bit brittle, can we have something like the following as the field instead?
> private Supplier<BrokerState> brokerState = null;
Then we don't have to make sure to update `isShuttingDown` when the logic changes in `KafkaServer`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on pull request #12741: KAFKA-14296; Partition leaders are not demoted during kraft controlled shutdown
Posted by GitBox <gi...@apache.org>.
dajac commented on PR #12741:
URL: https://github.com/apache/kafka/pull/12741#issuecomment-1277982773
Will merge it tomorrow to trunk and 3.3. I am waiting on getting a clean build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on a diff in pull request #12741: KAFKA-14296; Partition leaders are not demoted during kraft controlled shutdown
Posted by GitBox <gi...@apache.org>.
dajac commented on code in PR #12741:
URL: https://github.com/apache/kafka/pull/12741#discussion_r994688472
##########
core/src/main/scala/kafka/server/ReplicaManager.scala:
##########
@@ -2142,7 +2142,6 @@ class ReplicaManager(val config: KafkaConfig,
): Unit = {
stateChangeLogger.info(s"Transitioning ${localFollowers.size} partition(s) to " +
"local followers.")
- val shuttingDown = isShuttingDown.get()
Review Comment:
I suppose that you are saying that relying on `isShuttingDown` is brittle, right? If that is the case, I do agree that relying on the broker state is better. I think that we can refactor the remaining usages as a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hachikuji merged pull request #12741: KAFKA-14296; Partition leaders are not demoted during kraft controlled shutdown
Posted by GitBox <gi...@apache.org>.
hachikuji merged PR #12741:
URL: https://github.com/apache/kafka/pull/12741
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on a diff in pull request #12741: KAFKA-14296; Partition leaders are not demoted during kraft controlled shutdown
Posted by GitBox <gi...@apache.org>.
ijuma commented on code in PR #12741:
URL: https://github.com/apache/kafka/pull/12741#discussion_r994648305
##########
core/src/main/scala/kafka/server/ReplicaManager.scala:
##########
@@ -2151,28 +2150,24 @@ class ReplicaManager(val config: KafkaConfig,
try {
followerTopicSet.add(tp.topic)
- if (shuttingDown) {
- stateChangeLogger.trace(s"Unable to start fetching $tp with topic " +
- s"ID ${info.topicId} because the replica manager is shutting down.")
- } else {
- // We always update the follower state.
- // - This ensure that a replica with no leader can step down;
- // - This also ensures that the local replica is created even if the leader
- // is unavailable. This is required to ensure that we include the partition's
- // high watermark in the checkpoint file (see KAFKA-1647).
- val state = info.partition.toLeaderAndIsrPartitionState(tp, isNew)
- val isNewLeaderEpoch = partition.makeFollower(state, offsetCheckpoints, Some(info.topicId))
-
- if (isInControlledShutdown && (info.partition.leader == NO_LEADER ||
Review Comment:
Similarly, why do we need this special `isInControlledShutdown` field versus relying on the broker state as I mentioned above? This kind of approach is extremely brittle in my opinion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org