You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kafka.apache.org by rh...@apache.org on 2019/06/18 04:53:34 UTC
[kafka] branch trunk updated: MINOR: Update docs for KIP-415 (#6958)
This is an automated email from the ASF dual-hosted git repository.
rhauch pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/kafka.git
The following commit(s) were added to refs/heads/trunk by this push:
new c6ddd31 MINOR: Update docs for KIP-415 (#6958)
c6ddd31 is described below
commit c6ddd31887c3008da97d8a96dc53a408f420fcad
Author: Konstantine Karantasis <ko...@confluent.io>
AuthorDate: Mon Jun 17 21:53:20 2019 -0700
MINOR: Update docs for KIP-415 (#6958)
Update docs with KIP-415 details for incremental cooperative rebalancing
Author: Konstantine Karantasis <ko...@confluent.io>
Reviewer: Randall Hauch <rh...@gmail.com>
---
docs/connect.html | 35 ++++++++++++++++++++++++++++++++++-
docs/upgrade.html | 35 +++++++++++++++++++----------------
2 files changed, 53 insertions(+), 17 deletions(-)
diff --git a/docs/connect.html b/docs/connect.html
index baefc1b..a92bb04 100644
--- a/docs/connect.html
+++ b/docs/connect.html
@@ -508,7 +508,39 @@
</p>
<p>
- When a connector is first submitted to the cluster, the workers rebalance the full set of connectors in the cluster and their tasks so that each worker has approximately the same amount of work. This same rebalancing procedure is also used when connectors increase or decrease the number of tasks they require, or when a connector's configuration is changed. You can use the REST API to view the current status of a connector and its tasks, including the id of the worker to which each wa [...]
+ When a connector is first submitted to the cluster, a rebalance is triggered between the Connect workers in order to distribute the load that consists of the tasks of the new connector.
+ This same rebalancing procedure is also used when connectors increase or decrease the number of tasks they require, when a connector's configuration is changed, or when a
+ worker is added or removed from the group as part of an intentional upgrade of the Connect cluster or due to a failure.
+ </p>
+
+ <p>
+ In versions prior to 2.3.0, the Connect workers would rebalance the full set of connectors and their tasks in the cluster as a simple way to make sure that each worker has approximately the same amount of work.
+ This behavior can be still enabled by setting <code>connect.protocol=eager</code>.
+ </p>
+
+ <p>
+ Starting with 2.3.0, Kafka Connect is using by default a protocol that performs
+ <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka+Connect">incremental cooperative rebalancing</a>
+ that incrementally balances the connectors and tasks across the Connect workers, affecting only tasks that are new, to be removed, or need to move from one worker to another.
+ Other tasks are not stopped and restarted during the rebalance, as they would have been with the old protocol.
+ </p>
+
+ <p>
+ If a Connect worker leaves the group, intentionally or due to a failure, Connect waits for <code>scheduled.rebalance.max.delay.ms</code> before triggering a rebalance.
+ This delay defaults to five minutes (<code>300000ms</code>) to tolerate failures or upgrades of workers without immediately redistributing the load of a departing worker.
+ If this worker returns within the configured delay, it gets its previously assigned tasks in full.
+ However, this means that the tasks will remain unassigned until the time specified by <code>scheduled.rebalance.max.delay.ms</code> elapses.
+ If a worker does not return within that time limit, Connect will reassign those tasks among the remaining workers in the Connect cluster.
+ </p>
+
+ <p>
+ The new Connect protocol is enabled when all the workers that form the Connect cluster are configured with <code>connect.protocol=compatible</code>, which is also the default value when this property is missing.
+ Therefore, upgrading to the new Connect protocol happens automatically when all the workers upgrade to 2.3.0.
+ A rolling upgrade of the Connect cluster will activate incremental cooperative rebalancing when the last worker joins on version 2.3.0.
+ </p>
+
+ <p>
+ You can use the REST API to view the current status of a connector and its tasks, including the ID of the worker to which each was assigned. For example, the <code>GET /connectors/file-source/status</code> request shows the status of a connector named <code>file-source</code>:
</p>
<pre class="brush: json;">
@@ -537,6 +569,7 @@
<li><b>RUNNING:</b> The connector/task is running.</li>
<li><b>PAUSED:</b> The connector/task has been administratively paused.</li>
<li><b>FAILED:</b> The connector/task has failed (usually by raising an exception, which is reported in the status output).</li>
+ <li><b>DESTROYED:</b> The connector/task has been administratively removed and will stop appearing in the Connect cluster.</li>
</ul>
<p>
diff --git a/docs/upgrade.html b/docs/upgrade.html
index 4f483e6..4668e63 100644
--- a/docs/upgrade.html
+++ b/docs/upgrade.html
@@ -19,14 +19,32 @@
<script id="upgrade-template" type="text/x-handlebars-template">
+<h5><a id="upgrade_240_notable" href="#upgrade_240_notable">Notable changes in 2.4.0</a></h5>
+<ul>
+ <li>The <code>bin/kafka-preferred-replica-election.sh</code> command line tool has been deprecated. It has been replaced by <code>bin/kafka-leader-election.sh</code>.</li>
+ <li>The methods <code>electPreferredLeaders</code> in the Java <code>AdminClient</code> class have been deprecated in favor of the methods <code>electLeaders</code>.</li>
+ <li>Scala code leveraging the <code>NewTopic(String, int, short)</code> constructor with literal values will need to explicitly call <code>toShort</code> on the second literal.</li>
+</ul>
+
<h4><a id="upgrade_2_3_0" href="#upgrade_2_3_0">Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x, 0.10.2.x, 0.11.0.x, 1.0.x, 1.1.x, 2.0.x or 2.1.x or 2.2.x to 2.3.0</a></h4>
<!-- TODO core-team -->
<h5><a id="upgrade_230_notable" href="#upgrade_230_notable">Notable changes in 2.3.0</a></h5>
<ul>
+ <li> We are introducing a new rebalancing protocol for Kafka Connect based on
+ <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka+Connect">incremental cooperative rebalancing</a>.
+ The new protocol does not require stopping all the tasks during a rebalancing phase between Connect workers. Instead, only the tasks that need to be exchanged
+ between workers are stopped and they are started in a follow up rebalance. The new Connect protocol is enabled by default beginning with 2.3.0.
+ For more details on how it works and how to enable the old behavior of eager rebalancing, checkout
+ <a href="/{{version}}/documentation/#connect_administration">incremental cooperative rebalancing design</a>.
+ </li>
+ <li> We are introducing static membership towards consumer user. This feature reduces unnecessary rebalances during normal application upgrades or rolling bounces.
+ For more details on how to use it, checkout <a href="/{{version}}/design/#static_membership">static membership design</a>.
+ </li>
<li> Kafka Streams DSL switches its used store types. While this change is mainly transparent to users, there are some corner cases that may require code changes.
- See the <a href="/{{version}}/documentation/streams/upgrade-guide#streams_api_changes_230">Kafka Streams upgrade section</a> for more details.</li>
+ See the <a href="/{{version}}/documentation/streams/upgrade-guide#streams_api_changes_230">Kafka Streams upgrade section</a> for more details.
+ </li>
</ul>
<h4><a id="upgrade_2_2_0" href="#upgrade_2_2_0">Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x, 0.10.2.x, 0.11.0.x, 1.0.x, 1.1.x, 2.0.x or 2.1.x to 2.2.0</a></h4>
@@ -71,21 +89,6 @@
</li>
</ol>
-<h5><a id="upgrade_240_notable" href="#upgrade_240_notable">Notable changes in 2.4.0</a></h5>
-<ul>
- <li>The <code>bin/kafka-preferred-replica-election.sh</code> command line tool has been deprecated. It has been replaced by <code>bin/kafka-leader-election.sh</code>.</li>
- <li>The methods <code>electPreferredLeaders</code> in the Java <code>AdminClient</code> class have been deprecated in favor of the methods <code>electLeaders</code>.</li>
- <li>Scala code leveraging the <code>NewTopic(String, int, short)</code> constructor with literal values will need to explicitly call <code>toShort</code> on the second literal.</li>
-</ul>
-
-<h5><a id="upgrade_230_notable" href="#upgrade_230_notable">Notable changes in 2.3.0</a></h5>
-<ul>
- <li>
- We are introducing static membership towards consumer user. This feature reduces unnecessary rebalances during normal application upgrades or rolling bounces.
- For more details on how to use it, checkout <a href="/{{version}}/design/#static_membership">static membership design</a>.
- </li>
-</ul>
-
<h5><a id="upgrade_220_notable" href="#upgrade_220_notable">Notable changes in 2.2.0</a></h5>
<ul>
<li>The default consumer group id has been changed from the empty string (<code>""</code>) to <code>null</code>. Consumers who use the new default group id will not be able to subscribe to topics,