You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kafka.apache.org by jl...@apache.org on 2024/10/08 08:38:51 UTC
(kafka-site) branch MINOR-fix-upgrade-instructions created (now 4b737d824)
This is an automated email from the ASF dual-hosted git repository.
jlprat pushed a change to branch MINOR-fix-upgrade-instructions
in repository https://gitbox.apache.org/repos/asf/kafka-site.git
at 4b737d824 MINOR Add upgrade instructiosn for 38 and 39
This branch includes the following new commits:
new 4b737d824 MINOR Add upgrade instructiosn for 38 and 39
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(kafka-site) 01/01: MINOR Add upgrade instructiosn for 38 and 39
Posted by jl...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jlprat pushed a commit to branch MINOR-fix-upgrade-instructions
in repository https://gitbox.apache.org/repos/asf/kafka-site.git
commit 4b737d82485872c86ac3ca47ccbbb6399feb004b
Author: Josep Prat <jo...@aiven.io>
AuthorDate: Tue Oct 8 10:38:27 2024 +0200
MINOR Add upgrade instructiosn for 38 and 39
Signed-off-by: Josep Prat <jo...@aiven.io>
---
38/upgrade.html | 61 ++++++++++++++++++++++++++++
39/upgrade.html | 124 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 185 insertions(+)
diff --git a/38/upgrade.html b/38/upgrade.html
index be12f0441..2dcdba7e8 100644
--- a/38/upgrade.html
+++ b/38/upgrade.html
@@ -21,6 +21,67 @@
<h4><a id="upgrade_3_8_0" href="#upgrade_3_8_0">Upgrading to 3.8.0 from any version 0.8.x through 3.7.x</a></h4>
+ <h5><a id="upgrade_380_zk" href="#upgrade_380_zk">Upgrading ZooKeeper-based clusters</a></h5>
+ <p><b>If you are upgrading from a version prior to 2.1.x, please see the note in step 5 below about the change to the schema used to store consumer offsets.
+ Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.</b></p>
+
+ <p><b>For a rolling upgrade:</b></p>
+
+ <ol>
+ <li>Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
+ are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
+ overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
+ to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
+ <ul>
+ <li>inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. <code>3.7</code>, <code>3.6</code>, etc.)</li>
+ <li>log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See <a href="#upgrade_10_performance_impact">potential performance impact
+ following the upgrade</a> for the details on what this configuration does.)</li>
+ </ul>
+ If you are upgrading from version 0.11.0.x or above, and you have not overridden the message format, then you only need to override
+ the inter-broker protocol version.
+ <ul>
+ <li>inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. <code>3.7</code>, <code>3.6</code>, etc.)</li>
+ </ul>
+ </li>
+ <li>Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the
+ brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations.
+ It is still possible to downgrade at this point if there are any problems.
+ </li>
+ <li>Once the cluster's behavior and performance has been verified, bump the protocol version by editing
+ <code>inter.broker.protocol.version</code> and setting it to <code>3.8</code>.
+ </li>
+ <li>Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest
+ protocol version, it will no longer be possible to downgrade the cluster to an older version.
+ </li>
+ <li>If you have overridden the message format version as instructed above, then you need to do one more rolling restart to
+ upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later,
+ change log.message.format.version to 3.8 on each broker and restart them one by one. Note that the older Scala clients,
+ which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs
+ (or to take advantage of <a href="#upgrade_11_exactly_once_semantics">exactly once semantics</a>),
+ the newer Java clients must be used.
+ </li>
+ </ol>
+
+ <h5><a id="upgrade_380_kraft" href="#upgrade_380_kraft">Upgrading KRaft-based clusters</a></h5>
+ <p><b>If you are upgrading from a version prior to 3.3.0, please see the note in step 3 below. Once you have changed the metadata.version to the latest version, it will not be possible to downgrade to a version prior to 3.3-IV0.</b></p>
+
+ <p><b>For a rolling upgrade:</b></p>
+
+ <ol>
+ <li>Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the
+ brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations.
+ </li>
+ <li>Once the cluster's behavior and performance has been verified, bump the metadata.version by running
+ <code>
+ bin/kafka-features.sh upgrade --metadata 3.8
+ </code>
+ </li>
+ <li>Note that cluster metadata downgrade is not supported in this version since it has metadata changes.
+ Every <a href="https://github.com/apache/kafka/blob/trunk/server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java">MetadataVersion</a>
+ after 3.2.x has a boolean parameter that indicates if there are metadata changes (i.e. <code>IBP_3_3_IV3(7, "3.3", "IV3", true)</code> means this version has metadata changes).
+ Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.</li>
+ </ol>
+
<h5><a id="upgrade_380_notable" href="#upgrade_380_notable">Notable changes in 3.8.0</a></h5>
<ul>
<li>MirrorMaker 2 can now emit checkpoints for offsets mirrored before the start of the Checkpoint task for improved offset translation.
diff --git a/39/upgrade.html b/39/upgrade.html
index faa486215..d769cc70d 100644
--- a/39/upgrade.html
+++ b/39/upgrade.html
@@ -20,6 +20,69 @@
<script id="upgrade-template" type="text/x-handlebars-template">
<h4><a id="upgrade_3_9_0" href="#upgrade_3_9_0">Upgrading to 3.9.0 from any version 0.8.x through 3.8.x</a></h4>
+
+ <h5><a id="upgrade_390_zk" href="#upgrade_390_zk">Upgrading ZooKeeper-based clusters</a></h5>
+ <p><b>If you are upgrading from a version prior to 2.1.x, please see the note in step 5 below about the change to the schema used to store consumer offsets.
+ Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.</b></p>
+
+ <p><b>For a rolling upgrade:</b></p>
+
+ <ol>
+ <li>Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
+ are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
+ overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
+ to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
+ <ul>
+ <li>inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. <code>3.8</code>, <code>3.7</code>, etc.)</li>
+ <li>log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See <a href="#upgrade_10_performance_impact">potential performance impact
+ following the upgrade</a> for the details on what this configuration does.)</li>
+ </ul>
+ If you are upgrading from version 0.11.0.x or above, and you have not overridden the message format, then you only need to override
+ the inter-broker protocol version.
+ <ul>
+ <li>inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. <code>3.8</code>, <code>3.7</code>, etc.)</li>
+ </ul>
+ </li>
+ <li>Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the
+ brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations.
+ It is still possible to downgrade at this point if there are any problems.
+ </li>
+ <li>Once the cluster's behavior and performance has been verified, bump the protocol version by editing
+ <code>inter.broker.protocol.version</code> and setting it to <code>3.9</code>.
+ </li>
+ <li>Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest
+ protocol version, it will no longer be possible to downgrade the cluster to an older version.
+ </li>
+ <li>If you have overridden the message format version as instructed above, then you need to do one more rolling restart to
+ upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later,
+ change log.message.format.version to 3.8 on each broker and restart them one by one. Note that the older Scala clients,
+ which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs
+ (or to take advantage of <a href="#upgrade_11_exactly_once_semantics">exactly once semantics</a>),
+ the newer Java clients must be used.
+ </li>
+ </ol>
+
+ <h5><a id="upgrade_390_kraft" href="#upgrade_390_kraft">Upgrading KRaft-based clusters</a></h5>
+ <p><b>If you are upgrading from a version prior to 3.3.0, please see the note in step 3 below. Once you have changed the metadata.version to the latest version, it will not be possible to downgrade to a version prior to 3.3-IV0.</b></p>
+
+ <p><b>For a rolling upgrade:</b></p>
+
+ <ol>
+ <li>Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the
+ brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations.
+ </li>
+ <li>Once the cluster's behavior and performance has been verified, bump the metadata.version by running
+ <code>
+ bin/kafka-features.sh upgrade --metadata 3.9
+ </code>
+ </li>
+ <li>Note that cluster metadata downgrade is not supported in this version since it has metadata changes.
+ Every <a href="https://github.com/apache/kafka/blob/trunk/server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java">MetadataVersion</a>
+ after 3.2.x has a boolean parameter that indicates if there are metadata changes (i.e. <code>IBP_3_3_IV3(7, "3.3", "IV3", true)</code> means this version has metadata changes).
+ Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.</li>
+ </ol>
+
+
<h5><a id="upgrade_390_notable" href="#upgrade_390_notable">Notable changes in 3.9.0</a></h5>
<ul>
<li>In case you run your Kafka clusters with no execution permission for the <code>/tmp</code> partition, Kafka will not work properly. It might either refuse to start or fail
@@ -45,6 +108,67 @@
<h4><a id="upgrade_3_8_0" href="#upgrade_3_8_0">Upgrading to 3.8.0 from any version 0.8.x through 3.7.x</a></h4>
+ <h5><a id="upgrade_380_zk" href="#upgrade_380_zk">Upgrading ZooKeeper-based clusters</a></h5>
+ <p><b>If you are upgrading from a version prior to 2.1.x, please see the note in step 5 below about the change to the schema used to store consumer offsets.
+ Once you have changed the inter.broker.protocol.version to the latest version, it will not be possible to downgrade to a version prior to 2.1.</b></p>
+
+ <p><b>For a rolling upgrade:</b></p>
+
+ <ol>
+ <li>Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
+ are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
+ overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
+ to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
+ <ul>
+ <li>inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. <code>3.7</code>, <code>3.6</code>, etc.)</li>
+ <li>log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See <a href="#upgrade_10_performance_impact">potential performance impact
+ following the upgrade</a> for the details on what this configuration does.)</li>
+ </ul>
+ If you are upgrading from version 0.11.0.x or above, and you have not overridden the message format, then you only need to override
+ the inter-broker protocol version.
+ <ul>
+ <li>inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. <code>3.7</code>, <code>3.6</code>, etc.)</li>
+ </ul>
+ </li>
+ <li>Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the
+ brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations.
+ It is still possible to downgrade at this point if there are any problems.
+ </li>
+ <li>Once the cluster's behavior and performance has been verified, bump the protocol version by editing
+ <code>inter.broker.protocol.version</code> and setting it to <code>3.8</code>.
+ </li>
+ <li>Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest
+ protocol version, it will no longer be possible to downgrade the cluster to an older version.
+ </li>
+ <li>If you have overridden the message format version as instructed above, then you need to do one more rolling restart to
+ upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later,
+ change log.message.format.version to 3.8 on each broker and restart them one by one. Note that the older Scala clients,
+ which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs
+ (or to take advantage of <a href="#upgrade_11_exactly_once_semantics">exactly once semantics</a>),
+ the newer Java clients must be used.
+ </li>
+ </ol>
+
+ <h5><a id="upgrade_380_kraft" href="#upgrade_380_kraft">Upgrading KRaft-based clusters</a></h5>
+ <p><b>If you are upgrading from a version prior to 3.3.0, please see the note in step 3 below. Once you have changed the metadata.version to the latest version, it will not be possible to downgrade to a version prior to 3.3-IV0.</b></p>
+
+ <p><b>For a rolling upgrade:</b></p>
+
+ <ol>
+ <li>Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the
+ brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations.
+ </li>
+ <li>Once the cluster's behavior and performance has been verified, bump the metadata.version by running
+ <code>
+ bin/kafka-features.sh upgrade --metadata 3.8
+ </code>
+ </li>
+ <li>Note that cluster metadata downgrade is not supported in this version since it has metadata changes.
+ Every <a href="https://github.com/apache/kafka/blob/trunk/server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java">MetadataVersion</a>
+ after 3.2.x has a boolean parameter that indicates if there are metadata changes (i.e. <code>IBP_3_3_IV3(7, "3.3", "IV3", true)</code> means this version has metadata changes).
+ Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.</li>
+ </ol>
+
<h5><a id="upgrade_380_notable" href="#upgrade_380_notable">Notable changes in 3.8.0</a></h5>
<ul>
<li>MirrorMaker 2 can now emit checkpoints for offsets mirrored before the start of the Checkpoint task for improved offset translation.