You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kafka.apache.org by gu...@apache.org on 2019/05/27 21:25:58 UTC
[kafka] branch trunk updated: KAFKA-8219: add doc changes for
static membership release (#6790)
This is an automated email from the ASF dual-hosted git repository.
guozhang pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/kafka.git
The following commit(s) were added to refs/heads/trunk by this push:
new 27943e3 KAFKA-8219: add doc changes for static membership release (#6790)
27943e3 is described below
commit 27943e39876efad87fce7b558af3419f1a7a025c
Author: Boyang Chen <bo...@confluent.io>
AuthorDate: Mon May 27 14:25:47 2019 -0700
KAFKA-8219: add doc changes for static membership release (#6790)
* add doc changes for static membership release
* address comments
* address comments for a separate page
* address Matthias' comment
---
docs/design.html | 22 ++++++++++++++++++++++
docs/streams/upgrade-guide.html | 7 +++++++
docs/upgrade.html | 8 ++++++++
3 files changed, 37 insertions(+)
diff --git a/docs/design.html b/docs/design.html
index eadf29d..4c4e796 100644
--- a/docs/design.html
+++ b/docs/design.html
@@ -216,6 +216,28 @@
In the case of Hadoop we parallelize the data load by splitting the load over individual map tasks, one for each node/topic/partition combination, allowing full parallelism in the loading. Hadoop provides the task
management, and tasks which fail can restart without danger of duplicate data—they simply restart from their original position.
+ <h4><a id="static_membership" href="#static_membership">Static Membership</a></h4>
+ Static membership aims to improve the availability of stream applications, consumer groups and other applications built on top of the group rebalance protocol.
+ The rebalance protocol relies on the group coordinator to allocate entity ids to group members. These generated ids are ephemeral and will change when members restart and rejoin.
+ For consumer based apps, this "dynamic membership" can cause a large percentage of tasks re-assigned to different instances during administrative operations
+ such as code deploys, configuration updates and periodic restarts. For large state applications, shuffled tasks need a long time to recover their local states before processing
+ and cause applications to be partially or entirely unavailable. Motivated by this observation, Kafka’s group management protocol allows group members to provide persistent entity ids.
+ Group membership remains unchanged based on those ids, thus no rebalance will be triggered.
+ <p>
+ If you want to use static membership,
+ <ul>
+ <li>Upgrade both broker cluster and client apps to 2.3 or beyond, and also make sure the upgraded brokers are using <code>inter.broker.protocol.version</code>
+ of 2.3 or beyond as well.</li>
+ <li>Set the config <code>ConsumerConfig#GROUP_INSTANCE_ID_CONFIG</code> to a unique value for each consumer instance under one group.</li>
+ <li>For Kafka Streams applications, it is sufficient to set a unique <code>ConsumerConfig#GROUP_INSTANCE_ID_CONFIG</code> per KafkaStreams instance,
+ independent of the number of used threads for an instance.</li>
+ </ul>
+ If your broker is on an older version than 2.3, but you choose to set <code>ConsumerConfig#GROUP_INSTANCE_ID_CONFIG</code> on the client side, the application will detect
+ the broker version and then throws an UnsupportedException. If you accidentally configure duplicate ids for different instances,
+ a fencing mechanism on broker side will inform your duplicate client to shutdown immediately by triggering a <code>org.apache.kafka.common.errors.FencedInstanceIdException</code>.
+ For more details, see
+ <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances">KIP-345</a>
+
<h3><a id="semantics" href="#semantics">4.6 Message Delivery Semantics</a></h3>
<p>
Now that we understand a little about how producers and consumers work, let's discuss the semantic guarantees Kafka provides between producer and consumer. Clearly there are multiple possible message delivery
diff --git a/docs/streams/upgrade-guide.html b/docs/streams/upgrade-guide.html
index f4c5d48..4b3dac2 100644
--- a/docs/streams/upgrade-guide.html
+++ b/docs/streams/upgrade-guide.html
@@ -241,6 +241,13 @@
details, see <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-312%3A+Add+Overloaded+StreamsBuilder+Build+Method+to+Accept+java.util.Properties">KIP-312</a>
</p>
+ <p>
+ We are introducing static membership towards Kafka Streams user. This feature reduces unnecessary rebalances during normal application upgrades or rolling bounces.
+ For more details on how to use it, checkout <a href="/{{version}}/design/#static_membership">static membership design</a>.
+ Note, Kafka Streams uses the same <code>ConsumerConfig#GROUP_INSTANCE_ID_CONFIG</code>, and you only need to make sure it is uniquely defined across
+ different stream instances in one application.
+ </p>
+
<h3><a id="streams_api_changes_200" href="#streams_api_changes_200">Streams API changes in 2.0.0</a></h3>
<p>
In 2.0.0 we have added a few new APIs on the <code>ReadOnlyWindowStore</code> interface (for details please read <a href="#streams_api_changes_200">Streams API changes</a> below).
diff --git a/docs/upgrade.html b/docs/upgrade.html
index be9886c..d9232fa 100644
--- a/docs/upgrade.html
+++ b/docs/upgrade.html
@@ -71,6 +71,14 @@
</li>
</ol>
+<h5><a id="upgrade_230_notable" href="#upgrade_230_notable">Notable changes in 2.3.0</a></h5>
+<ul>
+ <li>
+ We are introducing static membership towards consumer user. This feature reduces unnecessary rebalances during normal application upgrades or rolling bounces.
+ For more details on how to use it, checkout <a href="/{{version}}/design/#static_membership">static membership design</a>.
+ </li>
+</ul>
+
<h5><a id="upgrade_220_notable" href="#upgrade_220_notable">Notable changes in 2.2.0</a></h5>
<ul>
<li>The default consumer group id has been changed from the empty string (<code>""</code>) to <code>null</code>. Consumers who use the new default group id will not be able to subscribe to topics,