You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kafka.apache.org by sh...@apache.org on 2022/06/02 13:30:30 UTC
[kafka] branch trunk updated: MINOR: Update the kafka-reassign-partitions script command in documentation (#12237)
This is an automated email from the ASF dual-hosted git repository.
showuon pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/kafka.git
The following commit(s) were added to refs/heads/trunk by this push:
new d8d92f0f80 MINOR: Update the kafka-reassign-partitions script command in documentation (#12237)
d8d92f0f80 is described below
commit d8d92f0f8014e5686e956a0c82bb2afdd41592d5
Author: RivenSun <91...@users.noreply.github.com>
AuthorDate: Thu Jun 2 21:30:22 2022 +0800
MINOR: Update the kafka-reassign-partitions script command in documentation (#12237)
Reviewers: Luke Chen <sh...@gmail.com>
---
.../kafka/admin/ReassignPartitionsCommand.scala | 2 +-
.../kafka/admin/ReassignPartitionsUnitTest.scala | 2 +-
docs/ops.html | 93 ++++++++++------------
3 files changed, 43 insertions(+), 54 deletions(-)
diff --git a/core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala b/core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala
index ac6304b449..ac454584e9 100755
--- a/core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala
+++ b/core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala
@@ -331,7 +331,7 @@ object ReassignPartitionsCommand extends Logging {
val state = states(topicPartition)
if (state.done) {
if (state.currentReplicas.equals(state.targetReplicas)) {
- bld.append("Reassignment of partition %s is complete.".
+ bld.append("Reassignment of partition %s is completed.".
format(topicPartition.toString))
} else {
bld.append(s"There is no active reassignment of partition ${topicPartition}, " +
diff --git a/core/src/test/scala/unit/kafka/admin/ReassignPartitionsUnitTest.scala b/core/src/test/scala/unit/kafka/admin/ReassignPartitionsUnitTest.scala
index cbbebe7c82..3b7c1ab719 100644
--- a/core/src/test/scala/unit/kafka/admin/ReassignPartitionsUnitTest.scala
+++ b/core/src/test/scala/unit/kafka/admin/ReassignPartitionsUnitTest.scala
@@ -66,7 +66,7 @@ class ReassignPartitionsUnitTest {
assertEquals(Seq(
"Status of partition reassignment:",
"Reassignment of partition bar-0 is still in progress.",
- "Reassignment of partition foo-0 is complete.",
+ "Reassignment of partition foo-0 is completed.",
"Reassignment of partition foo-1 is still in progress.").
mkString(System.lineSeparator()),
partitionReassignmentStatesToString(Map(
diff --git a/docs/ops.html b/docs/ops.html
index 183d8acb49..843c5cc59e 100644
--- a/docs/ops.html
+++ b/docs/ops.html
@@ -258,23 +258,23 @@
Current partition replica assignment
{"version":1,
- "partitions":[{"topic":"foo1","partition":2,"replicas":[1,2]},
- {"topic":"foo1","partition":0,"replicas":[3,4]},
- {"topic":"foo2","partition":2,"replicas":[1,2]},
- {"topic":"foo2","partition":0,"replicas":[3,4]},
- {"topic":"foo1","partition":1,"replicas":[2,3]},
- {"topic":"foo2","partition":1,"replicas":[2,3]}]
+ "partitions":[{"topic":"foo1","partition":0,"replicas":[2,1]},
+ {"topic":"foo1","partition":1,"replicas":[1,3]},
+ {"topic":"foo1","partition":2,"replicas":[3,4]},
+ {"topic":"foo2","partition":0,"replicas":[4,2]},
+ {"topic":"foo2","partition":1,"replicas":[2,1]},
+ {"topic":"foo2","partition":2,"replicas":[1,3]}]
}
Proposed partition reassignment configuration
{"version":1,
- "partitions":[{"topic":"foo1","partition":2,"replicas":[5,6]},
- {"topic":"foo1","partition":0,"replicas":[5,6]},
- {"topic":"foo2","partition":2,"replicas":[5,6]},
- {"topic":"foo2","partition":0,"replicas":[5,6]},
+ "partitions":[{"topic":"foo1","partition":0,"replicas":[6,5]},
{"topic":"foo1","partition":1,"replicas":[5,6]},
- {"topic":"foo2","partition":1,"replicas":[5,6]}]
+ {"topic":"foo1","partition":2,"replicas":[6,5]},
+ {"topic":"foo2","partition":0,"replicas":[5,6]},
+ {"topic":"foo2","partition":1,"replicas":[6,5]},
+ {"topic":"foo2","partition":2,"replicas":[5,6]}]
}</code></pre>
<p>
The tool generates a candidate assignment that will move all partitions from topics foo1,foo2 to brokers 5,6. Note, however, that at this point, the partition movement has not started, it merely tells you the current assignment and the proposed new assignment. The current assignment should be saved in case you want to rollback to it. The new assignment should be saved in a json file (e.g. expand-cluster-reassignment.json) to be input to the tool with the --execute option as follows:
@@ -282,34 +282,27 @@
Current partition replica assignment
{"version":1,
- "partitions":[{"topic":"foo1","partition":2,"replicas":[1,2]},
- {"topic":"foo1","partition":0,"replicas":[3,4]},
- {"topic":"foo2","partition":2,"replicas":[1,2]},
- {"topic":"foo2","partition":0,"replicas":[3,4]},
- {"topic":"foo1","partition":1,"replicas":[2,3]},
- {"topic":"foo2","partition":1,"replicas":[2,3]}]
+ "partitions":[{"topic":"foo1","partition":0,"replicas":[2,1]},
+ {"topic":"foo1","partition":1,"replicas":[1,3]},
+ {"topic":"foo1","partition":2,"replicas":[3,4]},
+ {"topic":"foo2","partition":0,"replicas":[4,2]},
+ {"topic":"foo2","partition":1,"replicas":[2,1]},
+ {"topic":"foo2","partition":2,"replicas":[1,3]}]
}
Save this to use as the --reassignment-json-file option during rollback
- Successfully started reassignment of partitions
- {"version":1,
- "partitions":[{"topic":"foo1","partition":2,"replicas":[5,6]},
- {"topic":"foo1","partition":0,"replicas":[5,6]},
- {"topic":"foo2","partition":2,"replicas":[5,6]},
- {"topic":"foo2","partition":0,"replicas":[5,6]},
- {"topic":"foo1","partition":1,"replicas":[5,6]},
- {"topic":"foo2","partition":1,"replicas":[5,6]}]
- }</code></pre>
+ Successfully started partition reassignments for foo1-0,foo1-1,foo1-2,foo2-0,foo2-1,foo2-2
+ </code></pre>
<p>
Finally, the --verify option can be used with the tool to check the status of the partition reassignment. Note that the same expand-cluster-reassignment.json (used with the --execute option) should be used with the --verify option:
<pre class="line-numbers"><code class="language-bash"> > bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file expand-cluster-reassignment.json --verify
Status of partition reassignment:
- Reassignment of partition [foo1,0] completed successfully
- Reassignment of partition [foo1,1] is in progress
- Reassignment of partition [foo1,2] is in progress
- Reassignment of partition [foo2,0] completed successfully
- Reassignment of partition [foo2,1] completed successfully
- Reassignment of partition [foo2,2] completed successfully</code></pre>
+ Reassignment of partition [foo1,0] is completed
+ Reassignment of partition [foo1,1] is still in progress
+ Reassignment of partition [foo1,2] is still in progress
+ Reassignment of partition [foo2,0] is completed
+ Reassignment of partition [foo2,1] is completed
+ Reassignment of partition [foo2,2] is completed</code></pre>
<h5 class="anchor-heading"><a id="basic_ops_partitionassignment" class="anchor-link"></a><a href="#basic_ops_partitionassignment">Custom partition assignment and migration</a></h5>
The partition reassignment tool can also be used to selectively move replicas of a partition to a specific set of brokers. When used in this manner, it is assumed that the user knows the reassignment plan and does not require the tool to generate a candidate reassignment, effectively skipping the --generate step and moving straight to the --execute step
@@ -329,17 +322,14 @@
}
Save this to use as the --reassignment-json-file option during rollback
- Successfully started reassignment of partitions
- {"version":1,
- "partitions":[{"topic":"foo1","partition":0,"replicas":[5,6]},
- {"topic":"foo2","partition":1,"replicas":[2,3]}]
- }</code></pre>
+ Successfully started partition reassignments for foo1-0,foo2-1
+ </code></pre>
<p>
The --verify option can be used with the tool to check the status of the partition reassignment. Note that the same custom-reassignment.json (used with the --execute option) should be used with the --verify option:
<pre class="line-numbers"><code class="language-bash"> > bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file custom-reassignment.json --verify
Status of partition reassignment:
- Reassignment of partition [foo1,0] completed successfully
- Reassignment of partition [foo2,1] completed successfully</code></pre>
+ Reassignment of partition [foo1,0] is completed
+ Reassignment of partition [foo2,1] is completed</code></pre>
<h4 class="anchor-heading"><a id="basic_ops_decommissioning_brokers" class="anchor-link"></a><a href="#basic_ops_decommissioning_brokers">Decommissioning brokers</a></h4>
The partition reassignment tool does not have the ability to automatically generate a reassignment plan for decommissioning brokers yet. As such, the admin has to come up with a reassignment plan to move the replica for all partitions hosted on the broker to be decommissioned, to the rest of the brokers. This can be relatively tedious as the reassignment needs to ensure that all the replicas are not moved from the decommissioned broker to only one other broker. To make this process eff [...]
@@ -361,14 +351,12 @@
"partitions":[{"topic":"foo","partition":0,"replicas":[5]}]}
Save this to use as the --reassignment-json-file option during rollback
- Successfully started reassignment of partitions
- {"version":1,
- "partitions":[{"topic":"foo","partition":0,"replicas":[5,6,7]}]}</code></pre>
+ Successfully started partition reassignment for foo-0</code></pre>
<p>
The --verify option can be used with the tool to check the status of the partition reassignment. Note that the same increase-replication-factor.json (used with the --execute option) should be used with the --verify option:
<pre class="line-numbers"><code class="language-bash"> > bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file increase-replication-factor.json --verify
Status of partition reassignment:
- Reassignment of partition [foo,0] completed successfully</code></pre>
+ Reassignment of partition [foo,0] is completed</code></pre>
You can also verify the increase in replication factor with the kafka-topics tool:
<pre class="line-numbers"><code class="language-bash"> > bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic foo --describe
Topic:foo PartitionCount:1 ReplicationFactor:3 Configs:
@@ -382,12 +370,11 @@
So for example, if you were to execute a rebalance, with the below command, it would move partitions at no more than 50MB/s.
<pre class="language-bash">$ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --execute --reassignment-json-file bigger-cluster.json --throttle 50000000</code></pre>
When you execute this script you will see the throttle engage:
- <pre class="line-numbers"><code class="language-bash"> The throttle limit was set to 50000000 B/s
- Successfully started reassignment of partitions.</code></pre>
- <p>Should you wish to alter the throttle, during a rebalance, say to increase the throughput so it completes quicker, you can do this by re-running the execute command passing the same reassignment-json-file:</p>
- <pre class="language-bash">$ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --execute --reassignment-json-file bigger-cluster.json --throttle 700000000
- There is an existing assignment running.
- The throttle limit was set to 700000000 B/s</code></pre>
+ <pre class="line-numbers"><code class="language-bash"> The inter-broker throttle limit was set to 50000000 B/s
+ Successfully started partition reassignment for foo1-0</code></pre>
+ <p>Should you wish to alter the throttle, during a rebalance, say to increase the throughput so it completes quicker, you can do this by re-running the execute command with the --additional option passing the same reassignment-json-file:</p>
+ <pre class="language-bash">$ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --additional --execute --reassignment-json-file bigger-cluster.json --throttle 700000000
+ The inter-broker throttle limit was set to 700000000 B/s</code></pre>
<p>Once the rebalance completes the administrator can check the status of the rebalance using the --verify option.
If the rebalance has completed, the throttle will be removed via the --verify command. It is important that
@@ -397,9 +384,11 @@
<pre class="line-numbers"><code class="language-bash"> > bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --verify --reassignment-json-file bigger-cluster.json
Status of partition reassignment:
- Reassignment of partition [my-topic,1] completed successfully
- Reassignment of partition [mytopic,0] completed successfully
- Throttle was removed.</code></pre>
+ Reassignment of partition [my-topic,1] is completed
+ Reassignment of partition [my-topic,0] is completed
+
+ Clearing broker-level throttles on brokers 1,2,3
+ Clearing topic-level throttles on topic my-topic</code></pre>
<p>The administrator can also validate the assigned configs using the kafka-configs.sh. There are two pairs of throttle
configuration used to manage the throttling process. First pair refers to the throttle value itself. This is configured, at a broker