You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kafka.apache.org by jg...@apache.org on 2018/03/06 06:21:56 UTC
[kafka] branch trunk updated: KAFKA-3806: Increase offsets
retention default to 7 days (KIP-186) (#4648)
This is an automated email from the ASF dual-hosted git repository.
jgus pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/kafka.git
The following commit(s) were added to refs/heads/trunk by this push:
new d13cbd0 KAFKA-3806: Increase offsets retention default to 7 days (KIP-186) (#4648)
d13cbd0 is described below
commit d13cbd0cae26d13c36c25d04a489ee309187ebeb
Author: Ewen Cheslack-Postava <me...@ewencp.org>
AuthorDate: Mon Mar 5 22:21:53 2018 -0800
KAFKA-3806: Increase offsets retention default to 7 days (KIP-186) (#4648)
Reviewers: Ismael Juma <is...@juma.me.uk>, Jason Gustafson <ja...@confluent.io>
---
core/src/main/scala/kafka/server/KafkaConfig.scala | 2 +-
.../scala/unit/kafka/server/OffsetCommitTest.scala | 4 +-
docs/upgrade.html | 61 ++++++++++++++++++++++
3 files changed, 64 insertions(+), 3 deletions(-)
diff --git a/core/src/main/scala/kafka/server/KafkaConfig.scala b/core/src/main/scala/kafka/server/KafkaConfig.scala
index 8b2fb10..cf22305 100755
--- a/core/src/main/scala/kafka/server/KafkaConfig.scala
+++ b/core/src/main/scala/kafka/server/KafkaConfig.scala
@@ -158,7 +158,7 @@ object Defaults {
val OffsetsTopicPartitions: Int = OffsetConfig.DefaultOffsetsTopicNumPartitions
val OffsetsTopicSegmentBytes: Int = OffsetConfig.DefaultOffsetsTopicSegmentBytes
val OffsetsTopicCompressionCodec: Int = OffsetConfig.DefaultOffsetsTopicCompressionCodec.codec
- val OffsetsRetentionMinutes: Int = 24 * 60
+ val OffsetsRetentionMinutes: Int = 7 * 24 * 60
val OffsetsRetentionCheckIntervalMs: Long = OffsetConfig.DefaultOffsetsRetentionCheckIntervalMs
val OffsetCommitTimeoutMs = OffsetConfig.DefaultOffsetCommitTimeoutMs
val OffsetCommitRequiredAcks = OffsetConfig.DefaultOffsetCommitRequiredAcks
diff --git a/core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala b/core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala
index 0793fa3..3e8a535 100755
--- a/core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala
+++ b/core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala
@@ -249,11 +249,11 @@ class OffsetCommitTest extends ZooKeeperTestHarness {
Thread.sleep(retentionCheckInterval * 2)
assertEquals(2L, simpleConsumer.fetchOffsets(fetchRequest).requestInfo.get(topicPartition).get.offset)
- // v1 version commit request with commit timestamp set to now - two days
+ // v1 version commit request with commit timestamp set to now - seven + a bit days
// committed offset should expire
val commitRequest2 = OffsetCommitRequest(
groupId = group,
- requestInfo = immutable.Map(topicPartition -> OffsetAndMetadata(3L, "metadata", Time.SYSTEM.milliseconds - 2*24*60*60*1000L)),
+ requestInfo = immutable.Map(topicPartition -> OffsetAndMetadata(3L, "metadata", Time.SYSTEM.milliseconds - (Defaults.OffsetsRetentionMinutes + 1) * 60 * 1000L)),
versionId = 1
)
assertEquals(Errors.NONE, simpleConsumer.commitOffsets(commitRequest2).commitStatus.get(topicPartition).get)
diff --git a/docs/upgrade.html b/docs/upgrade.html
index 3ac293d..324f8df 100644
--- a/docs/upgrade.html
+++ b/docs/upgrade.html
@@ -19,6 +19,67 @@
<script id="upgrade-template" type="text/x-handlebars-template">
+
+<h4><a id="upgrade_1_2_0" href="#upgrade_1_2_0">Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x, 0.10.2.x, 0.11.0.x, 1.0.x, or 1.1.x to 1.2.x</a></h4>
+<p>Kafka 1.2.0 introduces wire protocol changes. By following the recommended rolling upgrade plan below,
+ you guarantee no downtime during the upgrade. However, please review the <a href="#upgrade_120_notable">notable changes in 1.2.0</a> before upgrading.
+</p>
+
+<p><b>For a rolling upgrade:</b></p>
+
+<ol>
+ <li> Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you
+ are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously
+ overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior
+ to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
+ <ul>
+ <li>inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2, 0.9.0, 0.10.0, 0.10.1, 0.10.2, 0.11.0, 1.0, 1.1).</li>
+ <li>log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION (See <a href="#upgrade_10_performance_impact">potential performance impact
+ following the upgrade</a> for the details on what this configuration does.)</li>
+ </ul>
+ If you are upgrading from 0.11.0.x, 1.0.x, or 1.1.x and you have not overridden the message format, then you only need to override
+ the inter-broker protocol format.
+ <ul>
+ <li>inter.broker.protocol.version=CURRENT_KAFKA_VERSION (0.11.0, 1.0, 1.1).</li>
+ </ul>
+ </li>
+ <li> Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. </li>
+ <li> Once the entire cluster is upgraded, bump the protocol version by editing <code>inter.broker.protocol.version</code> and setting it to 1.1.
+ <li> Restart the brokers one by one for the new protocol version to take effect.</li>
+ <li> If you have overridden the message format version as instructed above, then you need to do one more rolling restart to
+ upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later,
+ change log.message.format.version to 1.2 on each broker and restart them one by one. Note that the older Scala consumer
+ does not support the new message format introduced in 0.11, so to avoid the performance cost of down-conversion (or to
+ take advantage of <a href="#upgrade_11_exactly_once_semantics">exactly once semantics</a>), the newer Java consumer must be used.</li>
+</ol>
+
+<p><b>Additional Upgrade Notes:</b></p>
+
+<ol>
+ <li>If you are willing to accept downtime, you can simply take all the brokers down, update the code and start them back up. They will start
+ with the new protocol by default.</li>
+ <li>Bumping the protocol version and restarting can be done any time after the brokers are upgraded. It does not have to be immediately after.
+ Similarly for the message format version.</li>
+ <li>If you are using Java8 method references in your Kafka Streams code you might need to update your code to resolve method ambiguties.
+ Hot-swaping the jar-file only might not work.</li>
+</ol>
+
+<h5><a id="upgrade_120_notable" href="#upgrade_120_notable">Notable changes in 1.2.0</a></h5>
+<ul>
+ <li><a href="https://cwiki.apache.org/confluence/x/oYtjB">KIP-186</a> increases the default offset retention time from 1 day to 7 days. This makes it less likely to "lose" offsets in an application that commits infrequently. It also increases the active set of offsets and therefore can increase memory usage on the broker. Note that the console consumer currently enables offset commit by default and can be the source of a large number of offsets which this change will now preserve for [...]
+</ul>
+
+<h5><a id="upgrade_120_new_protocols" href="#upgrade_120_new_protocols">New Protocol Versions</a></h5>
+<ul></ul>
+
+<h5><a id="upgrade_120_streams" href="#upgrade_120_streams">Upgrading a 1.2.0 Kafka Streams Application</a></h5>
+<ul>
+ <li> Upgrading your Streams application from 1.1.0 to 1.2.0 does not require a broker upgrade.
+ A Kafka Streams 1.2.0 application can connect to 1.2, 1.1, 1.0, 0.11.0, 0.10.2 and 0.10.1 brokers (it is not possible to connect to 0.10.0 brokers though). </li>
+ <li> See <a href="/{{version}}/documentation/streams/upgrade-guide#streams_api_changes_120">Streams API changes in 1.2.0</a> for more details. </li>
+</ul>
+
+
<h4><a id="upgrade_1_1_0" href="#upgrade_1_1_0">Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x, 0.10.2.x, 0.11.0.x or 1.0.x to 1.1.x</a></h4>
<p>Kafka 1.1.0 introduces wire protocol changes. By following the recommended rolling upgrade plan below,
you guarantee no downtime during the upgrade. However, please review the <a href="#upgrade_110_notable">notable changes in 1.1.0</a> before upgrading.
--
To stop receiving notification emails like this one, please contact
jgus@apache.org.