You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kafka.apache.org by ma...@apache.org on 2019/12/14 11:57:46 UTC
[kafka] branch 2.4 updated (c37fe3c -> 82d06d4)
This is an automated email from the ASF dual-hosted git repository.
manikumar pushed a change to branch 2.4
in repository https://gitbox.apache.org/repos/asf/kafka.git.
from c37fe3c Bump version to 2.4.1-SNAPSHOT
new 7f0f794 MINOR: clarify node grouping of input topics using pattern subscription (#7793)
new b79308f fix typo in processor-api developer guide docs (#7689)
new f35c795 port paragrpah from CP docs (#7808)
new 82d06d4 MINOR: Add missing parentheses in docs/streams/tutorial.html (#7696)
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/streams/architecture.html | 8 ++++++++
docs/streams/developer-guide/dsl-api.html | 2 +-
docs/streams/developer-guide/processor-api.html | 2 +-
docs/streams/tutorial.html | 2 +-
.../src/main/java/org/apache/kafka/streams/StreamsBuilder.java | 8 ++++++--
5 files changed, 17 insertions(+), 5 deletions(-)
[kafka] 02/04: fix typo in processor-api developer guide docs
(#7689)
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
manikumar pushed a commit to branch 2.4
in repository https://gitbox.apache.org/repos/asf/kafka.git
commit b79308fddf2b273069d2b2ca187351d4ed81c850
Author: Michael Gyarmathy <mg...@gmail.com>
AuthorDate: Thu Nov 14 09:32:35 2019 -0600
fix typo in processor-api developer guide docs (#7689)
Fixed a small typo on the Processor API page of the Kafka Streams developer guide docs. ("buildeer" changed to "builder")
Reviewers: Bill Bejeck <bb...@gmail.com>
---
docs/streams/developer-guide/processor-api.html | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/streams/developer-guide/processor-api.html b/docs/streams/developer-guide/processor-api.html
index ae33ebb..3295116 100644
--- a/docs/streams/developer-guide/processor-api.html
+++ b/docs/streams/developer-guide/processor-api.html
@@ -195,7 +195,7 @@
<a class="reference internal" href="#streams-developer-guide-state-store-custom"><span class="std std-ref">implement your own custom store type</span></a>.
It’s common practice to leverage an existing store type via the <code class="docutils literal"><span class="pre">Stores</span></code> factory.</p>
<p>Note that, when using Kafka Streams, you normally don’t create or instantiate state stores directly in your code.
- Rather, you define state stores indirectly by creating a so-called <code class="docutils literal"><span class="pre">StoreBuilder</span></code>. This buildeer is used by
+ Rather, you define state stores indirectly by creating a so-called <code class="docutils literal"><span class="pre">StoreBuilder</span></code>. This builder is used by
Kafka Streams as a factory to instantiate the actual state stores locally in application instances when and where
needed.</p>
<p>The following store types are available out of the box.</p>
[kafka] 03/04: port paragrpah from CP docs (#7808)
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
manikumar pushed a commit to branch 2.4
in repository https://gitbox.apache.org/repos/asf/kafka.git
commit f35c7958cf35dda9f4380f87d7c4a36883062bf9
Author: A. Sophie Blee-Goldman <so...@confluent.io>
AuthorDate: Mon Dec 9 13:35:17 2019 -0800
port paragrpah from CP docs (#7808)
The AK Streams architecture docs should explain how the maximum parallelism is determined
Reviewers: Bill Bejeck <bb...@gmail.com>
---
docs/streams/architecture.html | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/docs/streams/architecture.html b/docs/streams/architecture.html
index 8bc3156..7efd7ea 100644
--- a/docs/streams/architecture.html
+++ b/docs/streams/architecture.html
@@ -66,6 +66,14 @@
</p>
<p>
+ Slightly simplified, the maximum parallelism at which your application may run is bounded by the maximum number of stream tasks, which itself is determined by
+ maximum number of partitions of the input topic(s) the application is reading from. For example, if your input topic has 5 partitions, then you can run up to 5
+ applications instances. These instances will collaboratively process the topic’s data. If you run a larger number of app instances than partitions of the input
+ topic, the “excess” app instances will launch but remain idle; however, if one of the busy instances goes down, one of the idle instances will resume the former’s
+ work.
+ </p>
+
+ <p>
It is important to understand that Kafka Streams is not a resource manager, but a library that "runs" anywhere its stream processing application runs.
Multiple instances of the application are executed either on the same machine, or spread across multiple machines and tasks can be distributed automatically
by the library to those running application instances. The assignment of partitions to tasks never changes; if an application instance fails, all its assigned
[kafka] 04/04: MINOR: Add missing parentheses in
docs/streams/tutorial.html (#7696)
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
manikumar pushed a commit to branch 2.4
in repository https://gitbox.apache.org/repos/asf/kafka.git
commit 82d06d4553ad2f6f8447e3432ca2dc42efead930
Author: Xin Wang <wa...@gmail.com>
AuthorDate: Tue Nov 19 00:41:40 2019 +0800
MINOR: Add missing parentheses in docs/streams/tutorial.html (#7696)
Reviewers: Jason Gustafson <ja...@confluent.io>
---
docs/streams/tutorial.html | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/streams/tutorial.html b/docs/streams/tutorial.html
index 4b333bd..eeb90d6 100644
--- a/docs/streams/tutorial.html
+++ b/docs/streams/tutorial.html
@@ -608,7 +608,7 @@
.groupBy((key, value) -> value)
.count(Materialized.<String, Long, KeyValueStore<Bytes, byte[]>>as("counts-store"))
.toStream()
- .to("streams-wordcount-output", Produced.with(Serdes.String(), Serdes.Long());
+ .to("streams-wordcount-output", Produced.with(Serdes.String(), Serdes.Long()));
final Topology topology = builder.build();
final KafkaStreams streams = new KafkaStreams(topology, props);
[kafka] 01/04: MINOR: clarify node grouping of input topics using
pattern subscription (#7793)
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
manikumar pushed a commit to branch 2.4
in repository https://gitbox.apache.org/repos/asf/kafka.git
commit 7f0f794a9d0a8d89e80b01646704a962133d5c58
Author: A. Sophie Blee-Goldman <so...@confluent.io>
AuthorDate: Fri Dec 6 14:03:42 2019 -0800
MINOR: clarify node grouping of input topics using pattern subscription (#7793)
Updates the HTML docs and the javadoc.
Reviewers: John Roesler <vv...@apache.org>
---
docs/streams/developer-guide/dsl-api.html | 2 +-
.../src/main/java/org/apache/kafka/streams/StreamsBuilder.java | 8 ++++++--
2 files changed, 7 insertions(+), 3 deletions(-)
diff --git a/docs/streams/developer-guide/dsl-api.html b/docs/streams/developer-guide/dsl-api.html
index e54d449..02f2441 100644
--- a/docs/streams/developer-guide/dsl-api.html
+++ b/docs/streams/developer-guide/dsl-api.html
@@ -261,7 +261,7 @@
<p>You <strong>must specify SerDes explicitly</strong> if the key or value types of the records in the Kafka input
topics do not match the configured default SerDes. For information about configuring default SerDes, available
SerDes, and implementing your own custom SerDes see <a class="reference internal" href="datatypes.html#streams-developer-guide-serdes"><span class="std std-ref">Data Types and Serialization</span></a>.</p>
- <p class="last">Several variants of <code class="docutils literal"><span class="pre">stream</span></code> exist, for example to specify a regex pattern for input topics to read from).</p>
+ <p class="last">Several variants of <code class="docutils literal"><span class="pre">stream</span></code> exist. For example, you can specify a regex pattern for input topics to read from (note that all matching topics will be part of the same input topic group, and the work will not be parallelized for different topics if subscribed to in this way).</p>
</td>
</tr>
<tr class="row-odd"><td><p class="first"><strong>Table</strong></p>
diff --git a/streams/src/main/java/org/apache/kafka/streams/StreamsBuilder.java b/streams/src/main/java/org/apache/kafka/streams/StreamsBuilder.java
index 609bce5..8d650d2 100644
--- a/streams/src/main/java/org/apache/kafka/streams/StreamsBuilder.java
+++ b/streams/src/main/java/org/apache/kafka/streams/StreamsBuilder.java
@@ -148,7 +148,9 @@ public class StreamsBuilder {
* deserializers as specified in the {@link StreamsConfig config} are used.
* <p>
* If multiple topics are matched by the specified pattern, the created {@link KStream} will read data from all of
- * them and there is no ordering guarantee between records from different topics.
+ * them and there is no ordering guarantee between records from different topics. This also means that the work
+ * will not be parallelized for multiple topics, and the number of tasks will scale with the maximum partition
+ * count of any matching topic rather than the total number of partitions across all topics.
* <p>
* Note that the specified input topics must be partitioned by key.
* If this is not the case it is the user's responsibility to repartition the data before any key based operation
@@ -167,7 +169,9 @@ public class StreamsBuilder {
* are defined by the options in {@link Consumed} are used.
* <p>
* If multiple topics are matched by the specified pattern, the created {@link KStream} will read data from all of
- * them and there is no ordering guarantee between records from different topics.
+ * them and there is no ordering guarantee between records from different topics. This also means that the work
+ * will not be parallelized for multiple topics, and the number of tasks will scale with the maximum partition
+ * count of any matching topic rather than the total number of partitions across all topics.
* <p>
* Note that the specified input topics must be partitioned by key.
* If this is not the case it is the user's responsibility to repartition the data before any key based operation