You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kafka.apache.org by ij...@apache.org on 2016/07/29 13:07:40 UTC
kafka-site git commit: Documentation updates from kafka's 0.10.0
branch [Forced Update!]
Repository: kafka-site
Updated Branches:
refs/heads/asf-site d8eda9f4d -> aaad0d2e2 (forced update)
Documentation updates from kafka's 0.10.0 branch
Javadoc updates excluded until 0.10.0.1 is released.
Project: http://git-wip-us.apache.org/repos/asf/kafka-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/kafka-site/commit/aaad0d2e
Tree: http://git-wip-us.apache.org/repos/asf/kafka-site/tree/aaad0d2e
Diff: http://git-wip-us.apache.org/repos/asf/kafka-site/diff/aaad0d2e
Branch: refs/heads/asf-site
Commit: aaad0d2e2519c8147f9fd9185030f303ada79e7d
Parents: dc14da4
Author: Ismael Juma <is...@juma.me.uk>
Authored: Fri Jul 29 14:06:36 2016 +0100
Committer: Ismael Juma <is...@juma.me.uk>
Committed: Fri Jul 29 14:07:21 2016 +0100
----------------------------------------------------------------------
0100/generated/producer_config.html | 2 +-
0100/ops.html | 2 +-
0100/upgrade.html | 29 ++++++++++++++++-------------
3 files changed, 18 insertions(+), 15 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kafka-site/blob/aaad0d2e/0100/generated/producer_config.html
----------------------------------------------------------------------
diff --git a/0100/generated/producer_config.html b/0100/generated/producer_config.html
index d8b4e63..e5f96e2 100644
--- a/0100/generated/producer_config.html
+++ b/0100/generated/producer_config.html
@@ -20,7 +20,7 @@
<tr>
<td>compression.type</td><td>The compression type for all data generated by the producer. The default is none (i.e. no compression). Valid values are <code>none</code>, <code>gzip</code>, <code>snappy</code>, or <code>lz4</code>. Compression is of full batches of data, so the efficacy of batching will also impact the compression ratio (more batching means better compression).</td><td>string</td><td>none</td><td></td><td>high</td></tr>
<tr>
-<td>retries</td><td>Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries will potentially change the ordering of records because if two records are sent to a single partition, and the first fails and is retried but the second succeeds, then the second record may appear first.</td><td>int</td><td>0</td><td>[0,...,2147483647]</td><td>high</td></tr>
+<td>retries</td><td>Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting <code>max.in.flight.requests.per.connection</code> to 1 will potentially change the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch may appear first.</td><td>int</td><td>0</td><td>[0,...,2147483647]</td><td>high</td></tr>
<tr>
<td>ssl.key.password</td><td>The password of the private key in the key store file. This is optional for client.</td><td>password</td><td>null</td><td></td><td>high</td></tr>
<tr>
http://git-wip-us.apache.org/repos/asf/kafka-site/blob/aaad0d2e/0100/ops.html
----------------------------------------------------------------------
diff --git a/0100/ops.html b/0100/ops.html
index 5c76305..d7b87e1 100644
--- a/0100/ops.html
+++ b/0100/ops.html
@@ -388,7 +388,7 @@ The most important consumer configuration is the fetch size.
All configurations are documented in the <a href="#configuration">configuration</a> section.
<p>
<h4><a id="prodconfig" href="#prodconfig">A Production Server Config</a></h4>
-Here is our server production server configuration:
+Here is our production server configuration:
<pre>
# Replication configurations
num.replica.fetchers=4
http://git-wip-us.apache.org/repos/asf/kafka-site/blob/aaad0d2e/0100/upgrade.html
----------------------------------------------------------------------
diff --git a/0100/upgrade.html b/0100/upgrade.html
index a9a1443..34adce1 100644
--- a/0100/upgrade.html
+++ b/0100/upgrade.html
@@ -18,9 +18,9 @@
<h3><a id="upgrade" href="#upgrade">1.5 Upgrading From Previous Versions</a></h3>
<h4><a id="upgrade_10" href="#upgrade_10">Upgrading from 0.8.x or 0.9.x to 0.10.0.0</a></h4>
-0.10.0.0 has <a href="#upgrade_10_breaking">potential breaking changes</a> (please review before upgrading) and
-there may be a <a href="#upgrade_10_performance_impact">performance impact during the upgrade</a>. Because new protocols
-are introduced, it is important to upgrade your Kafka clusters before upgrading your clients.
+0.10.0.0 has <a href="#upgrade_10_breaking">potential breaking changes</a> (please review before upgrading) and possible <a href="#upgrade_10_performance_impact"> performance impact following the upgrade</a>. By following the recommended rolling upgrade plan below, you guarantee no downtime and no performance impact during and following the upgrade.
+<br>
+Note: Because new protocols are introduced, it is important to upgrade your Kafka clusters before upgrading your clients.
<p/>
<b>Notes to clients with version 0.9.0.0: </b>Due to a bug introduced in 0.9.0.0,
clients that depend on ZooKeeper (old Scala high-level Consumer and MirrorMaker if used with the old consumer) will not
@@ -30,15 +30,16 @@ work with 0.10.0.x brokers. Therefore, 0.9.0.0 clients should be upgraded to 0.9
<p><b>For a rolling upgrade:</b></p>
<ol>
- <li> Update server.properties file on all brokers and add the following property: inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2 or 0.9.0.0).
- We recommend that users set log.message.format.version=CURRENT_KAFKA_VERSION as well to ensure that performance of 0.8 and 0.9 consumers is not affected
- during the upgrade. See <a href="#upgrade_10_performance_impact">potential performance impact during upgrade</a> for the details.
+ <li> Update server.properties file on all brokers and add the following properties:
+ <ul>
+ <li>inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2 or 0.9.0.0).</li>
+ <li>log.message.format.version=CURRENT_KAFKA_VERSION (See <a href="#upgrade_10_performance_impact">potential performance impact following the upgrade</a> for the details on what this configuration does.)
+ </ul>
</li>
<li> Upgrade the brokers. This can be done a broker at a time by simply bringing it down, updating the code, and restarting it. </li>
- <li> Once the entire cluster is upgraded, bump the protocol version by editing inter.broker.protocol.version and setting it to 0.10.0.0. </li>
+ <li> Once the entire cluster is upgraded, bump the protocol version by editing inter.broker.protocol.version and setting it to 0.10.0.0. NOTE: You shouldn't touch log.message.format.version yet - this parameter should only change once all consumers have been upgraded to 0.10.0.0 </li>
<li> Restart the brokers one by one for the new protocol version to take effect. </li>
- <li> Once most consumers have been upgraded to 0.10.0 and if you followed the recommendation to set log.message.format.version=CURRENT_KAFKA_VERSION, change
- log.message.format.version to 0.10.0 on each broker and restart them one by one.
+ <li> Once all consumers have been upgraded to 0.10.0, change log.message.format.version to 0.10.0 on each broker and restart them one by one.
</li>
</ol>
@@ -46,7 +47,7 @@ work with 0.10.0.x brokers. Therefore, 0.9.0.0 clients should be upgraded to 0.9
<p><b>Note:</b> Bumping the protocol version and restarting can be done any time after the brokers were upgraded. It does not have to be immediately after.
-<h5><a id="upgrade_10_performance_impact" href="#upgrade_10_performance_impact">Potential performance impact during upgrade to 0.10.0.0</a></h5>
+<h5><a id="upgrade_10_performance_impact" href="#upgrade_10_performance_impact">Potential performance impact following upgrade to 0.10.0.0</a></h5>
<p>
The message format in 0.10.0 includes a new timestamp field and uses relative offsets for compressed messages.
The on disk message format can be configured through log.message.format.version in the server.properties file.
@@ -54,9 +55,11 @@ work with 0.10.0.x brokers. Therefore, 0.9.0.0 clients should be upgraded to 0.9
message formats before 0.10.0. In this case, the broker is able to convert messages from the 0.10.0 format to an earlier format
before sending the response to the consumer on an older version. However, the broker can't use zero-copy transfer in this case.
- To avoid such message conversion before consumers are upgraded to 0.10.0.0, one can set the message format to
- e.g. 0.9.0 when upgrading the broker to 0.10.0.0. This way, the broker can still use zero-copy transfer to send the
- data to the old consumers. Once most consumers are upgraded, one can change the message format to 0.10.0 on the broker.
+ Reports from the Kafka community on the performance impact have shown CPU utilization going from 20% before to 100% after an upgrade, which forced an immediate upgrade of all clients to bring performance back to normal.
+
+ To avoid such message conversion before consumers are upgraded to 0.10.0.0, one can set log.message.format.version to 0.8.2 or 0.9.0 when upgrading the broker to 0.10.0.0. This way, the broker can still use zero-copy transfer to send the data to the old consumers. Once consumers are upgraded, one can change the message format to 0.10.0 on the broker and enjoy the new message format that includes new timestamp and improved compression.
+
+ The conversion is supported to ensure compatibility and can be useful to support a few apps that have not updated to newer clients yet, but is impractical to support all consumer traffic on even an overprovisioned cluster. Therefore it is critical to avoid the message conversion as much as possible when brokers have been upgraded but the majority of clients have not.
</p>
<p>
For clients that are upgraded to 0.10.0.0, there is no performance impact.