You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kafka.apache.org by da...@apache.org on 2023/02/06 22:14:48 UTC
[kafka-site] branch zk-migration-docs-34 created (now 3cab444f)
This is an automated email from the ASF dual-hosted git repository.
davidarthur pushed a change to branch zk-migration-docs-34
in repository https://gitbox.apache.org/repos/asf/kafka-site.git
at 3cab444f Start on ZK to KRaft migration docs
This branch includes the following new commits:
new 3cab444f Start on ZK to KRaft migration docs
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[kafka-site] 01/01: Start on ZK to KRaft migration docs
Posted by da...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
davidarthur pushed a commit to branch zk-migration-docs-34
in repository https://gitbox.apache.org/repos/asf/kafka-site.git
commit 3cab444f7e5a02f313c24e9be4dfcb7ee8e02749
Author: David Arthur <mu...@gmail.com>
AuthorDate: Mon Feb 6 17:14:33 2023 -0500
Start on ZK to KRaft migration docs
---
34/ops.html | 121 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 120 insertions(+), 1 deletion(-)
diff --git a/34/ops.html b/34/ops.html
index c3ac673f..4ad72f12 100644
--- a/34/ops.html
+++ b/34/ops.html
@@ -3553,6 +3553,7 @@ foo
<li>Kafka server's <code>process.role</code> should be set to either <code>broker</code> or <code>controller</code> but not both. Combined mode can be used in development enviroment but it should be avoided in critical deployment evironments.</li>
<li>For redundancy, a Kafka cluster should use 3 controllers. More than 3 servers is not recommended in critical environments. In the rare case of a partial network failure it is possible for the cluster metadata quorum to become unavailable. This limitation will be addresses in a future release of Kafka.</li>
<li>The Kafka controllers store all of the metadata for the cluster in memory and on disk. We believe that for a typical Kafka cluster 5GB of main memory and 5GB of disk space on the metadata log director is sufficient.</li>
+ </ul>
<h4 class="anchor-heading"><a id="kraft_missing" class="anchor-link"></a><a href="#kraft_missing">Missing Features</a></h4>
@@ -3563,9 +3564,127 @@ foo
<li>Supporting JBOD configurations with multiple storage directories</li>
<li>Modifying certain dynamic configurations on the standalone KRaft controller</li>
<li>Delegation tokens</li>
- <li>Upgrade from ZooKeeper mode</li>
</ul>
+ <h4 class="anchor-heading"><a id="kraft_zk_migration" class="anchor-link"></a><a href="#kraft_zk_migration">ZooKeeper to KRaft Migration</a></h4>
+
+ <p>
+ <b>The ZooKeeper to KRaft migration feature is considered Early Access in 3.4.0. It is not recommended for production clusters.</b>
+ In this documentation, we use the term "migration" to refer to the process to changing a Kafka cluster's metadata
+ system from ZooKeeper to KRaft. An "upgrade" refers to installing a newer version of Kafka. It is not recommended to
+ upgrade the software at the same time as performing a migration
+ </p>
+
+ <p>The following features are not yet supported for ZK to KRaft migrations:</p>
+
+ <ul>
+ <li>Downgrading to ZooKeeper mode during or after the migration</li>
+ <li>Migration of ACLs</li>
+ <li>Other features <a href="#kraft_missing">not yet supported in KRaft</a></li>
+ </ul>
+
+ <h3>Preparing for migration</h3>
+
+ <p>
+ Before beginning the migration, the Kafka brokers must be upgraded to software version 3.4.0 and have the
+ "inter.broker.protocol.version" configuration set to "3.4". See <a href="#upgrade_3_4_0">Upgrading to 3.4.0</a> for
+ upgrade instructions.
+ </p>
+
+ <p>
+ It is recommended to enable TRACE level logging for the migration components while the migration is active. This can
+ be done by adding the following log4j configuration to each KRaft controller's "log4j.properties" file.
+ </p>
+
+ <pre>
+ log4j.logger.org.apache.kafka.metadata.migration=TRACE
+ </pre>
+
+ <p>
+ It may generally useful to enable DEBUG logging on the KRaft controllers and the ZK brokers during the migration.
+ </p>
+
+ <h3>Provisioning the KRaft controller quorum</h3>
+
+ <p>
+ Two things are needed before the migration can begin. The brokers must be configured to support the migration and
+ a KRaft controller quorum must be deployed. For the KRaft deployment, please refer to <a href="#kraft_config">the above documentation</a>.
+ The KRaft controllers should be provisioned with the latest <code>metadata.version</code> of "3.4".
+ In addition to the standard KRaft configuration, the KRaft controllers will need to enable support for the migration
+ as well as provide ZooKeeper connection configuration.
+ </p>
+
+ <pre>
+ # Sample KRaft cluster controller.properties listening on 9093
+ process.roles=controller
+ node.id=3000
+ controller.quorum.voters=1@localhost:9093
+ controller.listener.names=CONTROLLER
+ listeners=CONTROLLER://:9093
+
+ # Enable the migration
+ zookeeper.metadata.migration.enable=true
+
+ # ZooKeeper client configuration
+ zookeeper.connect=localhost:2181
+
+ # Other configs ...
+ </pre>
+
+ <p><em>Note: The KRaft cluster <code>node.id</code> values must be different from any existing ZK broker <code>broker.id</code>.
+ In KRaft-mode, the brokers and controllers share the same Node ID namespace.</em></p>
+
+ <h3>Enabling the migration on the brokers</h3>
+
+ <p>
+ Once the KRaft controller quorum has been started, the brokers will need to be reconfigured and restarted. Brokers
+ may be restarted in a rolling fashion to avoid impacting cluster availability. Each broker will need to add the
+ following configurations to allow it to communicate with the KRaft controllers and to enable the migration.
+ </p>
+
+ <ul>
+ <li><a href="#brokerconfigs_controller.quorum.voters">controller.quorum.voters</a></li>
+ <li><a href="#brokerconfigs_controller.listener.names">controller.listener.names</a></li>
+ <li>The controller.listener.name should also be added to <a href="#brokerconfigs_listener.security.protocol.map">listener.security.property.map</a></li>
+ <li><a href="#brokerconfigs_zookeeper.metadata.migration.enable">zookeeper.metadata.migration.enable</a></li>
+ </ul>
+
+ <p>Here is a sample config for a broker that is ready for migration:</p>
+
+ <pre>
+ # Sample ZK broker server.properties listening on 9092
+ broker.id=0
+ listeners=PLAINTEXT://:9092
+ advertised.listeners=PLAINTEXT://localhost:9092
+ listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
+
+ # Set the IBP
+ inter.broker.protocol.version=3.4
+
+ # Enable the migration
+ zookeeper.metadata.migration.enable=true
+
+ # ZooKeeper client configuration
+ zookeeper.connect=localhost:2181
+
+ # KRaft controller quorum configuration
+ controller.quorum.voters=3000@localhost:9093
+ controller.listener.names=CONTROLLER
+ </pre>
+
+ <p>
+ <em>Note: Once the final ZK broker has been restarted with the necessary configuration, the migration will automatically begin.</em>
+ When the migration is complete, a INFO level log can be observed on the active controller.
+ </p>
+
+ <pre>Completed migration of metadata from Zookeeper to KRaft</pre>
+
+ <h3>Migrating brokers to KRaft</h3>
+
+ <p>
+ Once the KRaft controller completes the metadata migration. The ZK brokers can be restarted as KRaft brokers.
+ </p>
+
</script>
<div class="p-ops"></div>