You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Poul Costinsky <po...@adbrain.com> on 2016/07/13 21:59:44 UTC
Kafka Streams question
Hi! I am prototyping some code using Kafka Streams, and have a question. I need to map a stream into another (with different partition key) and join it with a table. How do I control number of partitions of the mapped stream?
Thanks!
Poul Costinsky
Chief Architect
<http://www.adbrain.com/>
(360) 207-1753 <tel:%28360%29%20207-1753>
Re: Kafka Streams question
Posted by Michael Noll <mi...@confluent.io>.
Also, in the next version of Kafka / Kafka Streams such "intermediate"
topics will automatically be created for you when you do joins or
aggregations:
https://issues.apache.org/jira/browse/KAFKA-3561
So my previous message explained your options today when using the current
release of Kafka Streams (v0.10.0.0).
-Michael
On Thu, Jul 14, 2016 at 10:32 AM, Michael Noll <mi...@confluent.io> wrote:
> Poul,
>
> to add to what Matthias said: If you are wondering how to manually create
> a topic, you have basically two options.
>
> A. Use Kafka's CLI tools to create the topic "from the outside".
>
> # Example
> $ kafka-topics.sh --create --topic my-custom-toipc --zookeeper
> localhost:2181 --partitions 1 --replication-factor 1
>
> B. Use Kafka's API to programmatically create the topic. See [1] for an
> example.
>
> Question for you to learn how we could perhaps improve the status quo:
> How would you have expected this to work in the current Kafka Streams
> API? For example, would you have expected that, say, the `through()`
> method would accept parameters to specify the number of partitions?
>
>
> Hope this helps,
> Michael
>
> [1]
> https://github.com/confluentinc/examples/blob/kafka-0.10.0.0-cp-3.0.0/kafka-streams/src/test/java/io/confluent/examples/streams/kafka/KafkaEmbedded.java#L133-L160
>
>
>
>
> On Thu, Jul 14, 2016 at 10:08 AM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> Hi,
>>
>> you can manually create a topic with the number of partitions you want
>> to have and use this topic via through()
>>
>> KStream input = ...
>>
>> input.map().through("manually-created-topic").join(...)
>>
>> However, both KStream and KTable need to have the same number of
>> partitions for perform the join. Thus, you might need to create a topic
>> (with the same number of partitions) for the table, too.
>>
>> See
>>
>> http://docs.confluent.io/3.0.0/streams/developer-guide.html#joining-streams
>>
>>
>> -Matthias
>>
>> On 07/13/2016 11:59 PM, Poul Costinsky wrote:
>> > Hi! I am prototyping some code using Kafka Streams, and have a
>> question. I need to map a stream into another (with different partition
>> key) and join it with a table. How do I control number of partitions of the
>> mapped stream?
>> >
>> > Thanks!
>> >
>> > Poul Costinsky
>> > Chief Architect
>> >
>> > <http://www.adbrain.com/>
>> > (360) 207-1753 <tel:%28360%29%20207-1753>
>> >
>> >
>> >
>> >
>> >
>>
>>
>
>
> --
>
> *Michael G. Noll | Product Manager | Confluent | +1 650.453.5860
> <%2B1%20650.453.5860>Download Apache Kafka and Confluent Platform:
> www.confluent.io/download <http://www.confluent.io/download>*
>
--
*Michael G. Noll | Product Manager | Confluent | +1 650.453.5860Download
Apache Kafka and Confluent Platform: www.confluent.io/download
<http://www.confluent.io/download>*
Re: Kafka Streams question
Posted by Michael Noll <mi...@confluent.io>.
Poul,
to add to what Matthias said: If you are wondering how to manually create
a topic, you have basically two options.
A. Use Kafka's CLI tools to create the topic "from the outside".
# Example
$ kafka-topics.sh --create --topic my-custom-toipc --zookeeper
localhost:2181 --partitions 1 --replication-factor 1
B. Use Kafka's API to programmatically create the topic. See [1] for an
example.
Question for you to learn how we could perhaps improve the status quo: How
would you have expected this to work in the current Kafka Streams API? For
example, would you have expected that, say, the `through()` method would
accept parameters to specify the number of partitions?
Hope this helps,
Michael
[1]
https://github.com/confluentinc/examples/blob/kafka-0.10.0.0-cp-3.0.0/kafka-streams/src/test/java/io/confluent/examples/streams/kafka/KafkaEmbedded.java#L133-L160
On Thu, Jul 14, 2016 at 10:08 AM, Matthias J. Sax <ma...@confluent.io>
wrote:
> Hi,
>
> you can manually create a topic with the number of partitions you want
> to have and use this topic via through()
>
> KStream input = ...
>
> input.map().through("manually-created-topic").join(...)
>
> However, both KStream and KTable need to have the same number of
> partitions for perform the join. Thus, you might need to create a topic
> (with the same number of partitions) for the table, too.
>
> See
> http://docs.confluent.io/3.0.0/streams/developer-guide.html#joining-streams
>
>
> -Matthias
>
> On 07/13/2016 11:59 PM, Poul Costinsky wrote:
> > Hi! I am prototyping some code using Kafka Streams, and have a question.
> I need to map a stream into another (with different partition key) and join
> it with a table. How do I control number of partitions of the mapped stream?
> >
> > Thanks!
> >
> > Poul Costinsky
> > Chief Architect
> >
> > <http://www.adbrain.com/>
> > (360) 207-1753 <tel:%28360%29%20207-1753>
> >
> >
> >
> >
> >
>
>
--
*Michael G. Noll | Product Manager | Confluent | +1 650.453.5860Download
Apache Kafka and Confluent Platform: www.confluent.io/download
<http://www.confluent.io/download>*
Re: Kafka Streams question
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Hi,
you can manually create a topic with the number of partitions you want
to have and use this topic via through()
KStream input = ...
input.map().through("manually-created-topic").join(...)
However, both KStream and KTable need to have the same number of
partitions for perform the join. Thus, you might need to create a topic
(with the same number of partitions) for the table, too.
See
http://docs.confluent.io/3.0.0/streams/developer-guide.html#joining-streams
-Matthias
On 07/13/2016 11:59 PM, Poul Costinsky wrote:
> Hi! I am prototyping some code using Kafka Streams, and have a question. I need to map a stream into another (with different partition key) and join it with a table. How do I control number of partitions of the mapped stream?
>
> Thanks!
>
> Poul Costinsky
> Chief Architect
>
> <http://www.adbrain.com/>
> (360) 207-1753 <tel:%28360%29%20207-1753>
>
>
>
>
>