You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Luke Chen (Jira)" <ji...@apache.org> on 2024/10/24 10:32:00 UTC
[jira] [Updated] (KAFKA-17866) kraft.version/dirID is not updated after upgrading to MV 3.9
[ https://issues.apache.org/jira/browse/KAFKA-17866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luke Chen updated KAFKA-17866:
------------------------------
Description:
When upgrading old controllers to v3.9.0, we didn't update the kraft.version to 1 and the directory ID, which causes it blocks the following controller addition/removal.
Steps to reproduce:
1. format 1 controller node in MV 3.8:
{code:java}
bin/kafka-storage.sh format -t r0NrSlvGTqmZCtFtZ8tBIg -c config/kraft/controller1.properties --release-version 3.8
{code}
2. start up the controller
3. update the metadata version to 3.9:
{code:java}
bin/kafka-features.sh --bootstrap-controller localhost:9093 upgrade --metadata 3.9
or
bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --feature kraft.version=1
{code}
4. format one additional controller to be expanded
{code:java}
bin/kafka-storage.sh format --cluster-id r0NrSlvGTqmZCtFtZ8tBIg --config config/kraft/controller2.properties --no-initial-controllers
{code}
5. Try to add this node
{code:java}
bin/kafka-metadata-quorum.sh --command-config config/kraft/controller2.properties --bootstrap-controller localhost:9093 add-controller
{code}
It'll fail with:
{code:java}
org.apache.kafka.common.errors.UnsupportedVersionException: Cluster doesn't support adding voter because the kraft.version feature is 0
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnsupportedVersionException: Cluster doesn't support adding voter because the kraft.version feature is 0
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
at org.apache.kafka.tools.MetadataQuorumCommand.handleAddController(MetadataQuorumCommand.java:431)
at org.apache.kafka.tools.MetadataQuorumCommand.execute(MetadataQuorumCommand.java:147)
at org.apache.kafka.tools.MetadataQuorumCommand.mainNoExit(MetadataQuorumCommand.java:81)
at org.apache.kafka.tools.MetadataQuorumCommand.main(MetadataQuorumCommand.java:76)
Caused by: org.apache.kafka.common.errors.UnsupportedVersionException: Cluster doesn't support adding voter because the kraft.version feature is 0
{code}
Under this situation, I can't find any workaround to update kraft.version even if rebooting the node.
Besides, we also don't generate directory id for the upgraded nodes. Without the directory id, even if we have correct kraft.version, the controller removal still cannot succeed.
{code:java}
./bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --replication
NodeId DirectoryId LogEndOffset Lag LastFetchTimestamp LastCaughtUpTimestamp Status
1 AAAAAAAAAAAAAAAAAAAAAA 588 0 1729752652123 1729752652123 Leader {code}
was:
When upgrading old controllers to v3.9.0, we didn't update the kraft.version to 1, which causes it blocks the following controller addition/removal.
Steps to reproduce:
1. format 1 controller node in MV 3.8:
{code:java}
bin/kafka-storage.sh format -t r0NrSlvGTqmZCtFtZ8tBIg -c config/kraft/controller1.properties --release-version 3.8
{code}
2. start up the controller
3. update the metadata version to 3.9:
{code:java}
bin/kafka-features.sh --bootstrap-controller localhost:9093 upgrade --metadata 3.9
or
bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --feature kraft.version=1
{code}
4. format one additional controller to be expanded
{code:java}
bin/kafka-storage.sh format --cluster-id r0NrSlvGTqmZCtFtZ8tBIg --config config/kraft/controller2.properties --no-initial-controllers
{code}
5. Try to add this node
{code:java}
bin/kafka-metadata-quorum.sh --command-config config/kraft/controller2.properties --bootstrap-controller localhost:9093 add-controller
{code}
It'll fail with:
{code:java}
org.apache.kafka.common.errors.UnsupportedVersionException: Cluster doesn't support adding voter because the kraft.version feature is 0
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnsupportedVersionException: Cluster doesn't support adding voter because the kraft.version feature is 0
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
at org.apache.kafka.tools.MetadataQuorumCommand.handleAddController(MetadataQuorumCommand.java:431)
at org.apache.kafka.tools.MetadataQuorumCommand.execute(MetadataQuorumCommand.java:147)
at org.apache.kafka.tools.MetadataQuorumCommand.mainNoExit(MetadataQuorumCommand.java:81)
at org.apache.kafka.tools.MetadataQuorumCommand.main(MetadataQuorumCommand.java:76)
Caused by: org.apache.kafka.common.errors.UnsupportedVersionException: Cluster doesn't support adding voter because the kraft.version feature is 0
{code}
Under this situation, I can't find any workaround to update kraft.version even if rebooting the node.
Besides, we also don't generate directory id for the upgraded nodes. Without the directory id, even if we have correct kraft.version, the controller removal still cannot succeed.
{code:java}
./bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --replication
NodeId DirectoryId LogEndOffset Lag LastFetchTimestamp LastCaughtUpTimestamp Status
1 AAAAAAAAAAAAAAAAAAAAAA 588 0 1729752652123 1729752652123 Leader {code}
> kraft.version/dirID is not updated after upgrading to MV 3.9
> ------------------------------------------------------------
>
> Key: KAFKA-17866
> URL: https://issues.apache.org/jira/browse/KAFKA-17866
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 3.9.0
> Reporter: Luke Chen
> Priority: Blocker
> Fix For: 3.9.0
>
>
> When upgrading old controllers to v3.9.0, we didn't update the kraft.version to 1 and the directory ID, which causes it blocks the following controller addition/removal.
>
> Steps to reproduce:
> 1. format 1 controller node in MV 3.8:
> {code:java}
> bin/kafka-storage.sh format -t r0NrSlvGTqmZCtFtZ8tBIg -c config/kraft/controller1.properties --release-version 3.8
> {code}
> 2. start up the controller
> 3. update the metadata version to 3.9:
> {code:java}
> bin/kafka-features.sh --bootstrap-controller localhost:9093 upgrade --metadata 3.9
> or
> bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --feature kraft.version=1
> {code}
> 4. format one additional controller to be expanded
> {code:java}
> bin/kafka-storage.sh format --cluster-id r0NrSlvGTqmZCtFtZ8tBIg --config config/kraft/controller2.properties --no-initial-controllers
> {code}
> 5. Try to add this node
> {code:java}
> bin/kafka-metadata-quorum.sh --command-config config/kraft/controller2.properties --bootstrap-controller localhost:9093 add-controller
> {code}
> It'll fail with:
> {code:java}
> org.apache.kafka.common.errors.UnsupportedVersionException: Cluster doesn't support adding voter because the kraft.version feature is 0
> java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnsupportedVersionException: Cluster doesn't support adding voter because the kraft.version feature is 0
> at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
> at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
> at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
> at org.apache.kafka.tools.MetadataQuorumCommand.handleAddController(MetadataQuorumCommand.java:431)
> at org.apache.kafka.tools.MetadataQuorumCommand.execute(MetadataQuorumCommand.java:147)
> at org.apache.kafka.tools.MetadataQuorumCommand.mainNoExit(MetadataQuorumCommand.java:81)
> at org.apache.kafka.tools.MetadataQuorumCommand.main(MetadataQuorumCommand.java:76)
> Caused by: org.apache.kafka.common.errors.UnsupportedVersionException: Cluster doesn't support adding voter because the kraft.version feature is 0
> {code}
>
> Under this situation, I can't find any workaround to update kraft.version even if rebooting the node.
> Besides, we also don't generate directory id for the upgraded nodes. Without the directory id, even if we have correct kraft.version, the controller removal still cannot succeed.
> {code:java}
> ./bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --replication
> NodeId DirectoryId LogEndOffset Lag LastFetchTimestamp LastCaughtUpTimestamp Status
> 1 AAAAAAAAAAAAAAAAAAAAAA 588 0 1729752652123 1729752652123 Leader {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)