You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Matthias J. Sax" <ma...@confluent.io> on 2018/04/08 21:04:24 UTC
Re: join 2 topic streams --> to another topic
Check out this blog post that explain how the different joins work:
https://www.confluent.io/blog/crossing-streams-joins-apache-kafka/
It's hard to give a general answer -- it depends on the context of your
application. Are keys unique? Do you want to get exactly one result or
should a single stock join with multiple dividends? Do you want Stock
and Dividend join depending the their timestamps?
-Matthias
On 4/8/18 1:34 PM, adrien ruffie wrote:
> Hello all,
>
> I have 2 topics streamed by KStream<String, Stock> and one KStream<String, Dividend>
>
> I want to merge both object's informations (Stock & Dividend) and send to another topic
>
> with for example <String, DividendStockJoin>
>
>
> The <String> key of 2 two topic is the same. I need to use, leftJoin, merge, KTable, ...
>
> what is the best solution ? What do you think ?
>
>
> Moreover, how can I make sure the timestamp are pretty much the same? (for merging same key at the same time produced) Example:
>
> KStream<String, Stock> --> key: "YHOO", and one timestamp of record 2018-04-08 19:56:30
> KStream<String, Dividend> --> key: "YHOO", and one timestamp of record 2018-04-08 19:55:27
>
>
> I need a Windowing ?
>
>
> Great thank & best regards,
>
>
> Adrien
>
Re: join 2 topic streams --> to another topic
Posted by Ted Yu <yu...@gmail.com>.
The blog is an interesting reading material.
There was a minor typo:
bq. an event arrives for either the left of right input stream
'of' above should be 'or'.
Cheers
On Sun, Apr 8, 2018 at 2:04 PM, Matthias J. Sax <ma...@confluent.io>
wrote:
> Check out this blog post that explain how the different joins work:
> https://www.confluent.io/blog/crossing-streams-joins-apache-kafka/
>
> It's hard to give a general answer -- it depends on the context of your
> application. Are keys unique? Do you want to get exactly one result or
> should a single stock join with multiple dividends? Do you want Stock
> and Dividend join depending the their timestamps?
>
>
> -Matthias
>
> On 4/8/18 1:34 PM, adrien ruffie wrote:
> > Hello all,
> >
> > I have 2 topics streamed by KStream<String, Stock> and one
> KStream<String, Dividend>
> >
> > I want to merge both object's informations (Stock & Dividend) and send
> to another topic
> >
> > with for example <String, DividendStockJoin>
> >
> >
> > The <String> key of 2 two topic is the same. I need to use, leftJoin,
> merge, KTable, ...
> >
> > what is the best solution ? What do you think ?
> >
> >
> > Moreover, how can I make sure the timestamp are pretty much the same?
> (for merging same key at the same time produced) Example:
> >
> > KStream<String, Stock> --> key: "YHOO", and one timestamp of record
> 2018-04-08 19:56:30
> > KStream<String, Dividend> --> key: "YHOO", and one timestamp of record
> 2018-04-08 19:55:27
> >
> >
> > I need a Windowing ?
> >
> >
> > Great thank & best regards,
> >
> >
> > Adrien
> >
>
>
RE: join 2 topic streams --> to another topic
Posted by adrien ruffie <ad...@hotmail.fr>.
You're right Matthias, I will take your suggestion of KTable-KTable join 😊
I suppose so that meant I don't need windowing ?
-Adrien
________________________________
De : Matthias J. Sax <ma...@confluent.io>
Envoyé : lundi 9 avril 2018 20:15:17
À : users@kafka.apache.org
Objet : Re: join 2 topic streams --> to another topic
From what you describe I infer (but it might be a wild guess), that you
are actually trying to do KTable-KTable join?
It's all about the semantics of your input data... Note, that a Kafka
topic does not have semantics per-se; you apply semantics when you read
a topic either as stream or table though.
-Matthias
On 4/9/18 12:16 AM, adrien ruffie wrote:
> Hello Matthias,
>
> thank for your response. I will try to read the blog post today.
>
>
> For the keys, not really, In fact, keys are always the same "symbol" of a STOCK example "YHOO", "INTC" and they never change.
>
> Do you with I need to use another key ? Or add a timestamp to the key ?
>
> For single stock join with multiple dividends, I didn't think about it before ... is it possible ?
>
>
> For join depending of timestamps why not, is it possible with windowing ?
>
>
> Thank Matthias
>
> Adrien
>
> ________________________________
> De : Matthias J. Sax <ma...@confluent.io>
> Envoyé : dimanche 8 avril 2018 23:04:24
> À : users@kafka.apache.org
> Objet : Re: join 2 topic streams --> to another topic
>
> Check out this blog post that explain how the different joins work:
> https://www.confluent.io/blog/crossing-streams-joins-apache-kafka/
>
> It's hard to give a general answer -- it depends on the context of your
> application. Are keys unique? Do you want to get exactly one result or
> should a single stock join with multiple dividends? Do you want Stock
> and Dividend join depending the their timestamps?
>
>
> -Matthias
>
> On 4/8/18 1:34 PM, adrien ruffie wrote:
>> Hello all,
>>
>> I have 2 topics streamed by KStream<String, Stock> and one KStream<String, Dividend>
>>
>> I want to merge both object's informations (Stock & Dividend) and send to another topic
>>
>> with for example <String, DividendStockJoin>
>>
>>
>> The <String> key of 2 two topic is the same. I need to use, leftJoin, merge, KTable, ...
>>
>> what is the best solution ? What do you think ?
>>
>>
>> Moreover, how can I make sure the timestamp are pretty much the same? (for merging same key at the same time produced) Example:
>>
>> KStream<String, Stock> --> key: "YHOO", and one timestamp of record 2018-04-08 19:56:30
>> KStream<String, Dividend> --> key: "YHOO", and one timestamp of record 2018-04-08 19:55:27
>>
>>
>> I need a Windowing ?
>>
>>
>> Great thank & best regards,
>>
>>
>> Adrien
>>
>
>
Re: join 2 topic streams --> to another topic
Posted by "Matthias J. Sax" <ma...@confluent.io>.
From what you describe I infer (but it might be a wild guess), that you
are actually trying to do KTable-KTable join?
It's all about the semantics of your input data... Note, that a Kafka
topic does not have semantics per-se; you apply semantics when you read
a topic either as stream or table though.
-Matthias
On 4/9/18 12:16 AM, adrien ruffie wrote:
> Hello Matthias,
>
> thank for your response. I will try to read the blog post today.
>
>
> For the keys, not really, In fact, keys are always the same "symbol" of a STOCK example "YHOO", "INTC" and they never change.
>
> Do you with I need to use another key ? Or add a timestamp to the key ?
>
> For single stock join with multiple dividends, I didn't think about it before ... is it possible ?
>
>
> For join depending of timestamps why not, is it possible with windowing ?
>
>
> Thank Matthias
>
> Adrien
>
> ________________________________
> De : Matthias J. Sax <ma...@confluent.io>
> Envoyé : dimanche 8 avril 2018 23:04:24
> À : users@kafka.apache.org
> Objet : Re: join 2 topic streams --> to another topic
>
> Check out this blog post that explain how the different joins work:
> https://www.confluent.io/blog/crossing-streams-joins-apache-kafka/
>
> It's hard to give a general answer -- it depends on the context of your
> application. Are keys unique? Do you want to get exactly one result or
> should a single stock join with multiple dividends? Do you want Stock
> and Dividend join depending the their timestamps?
>
>
> -Matthias
>
> On 4/8/18 1:34 PM, adrien ruffie wrote:
>> Hello all,
>>
>> I have 2 topics streamed by KStream<String, Stock> and one KStream<String, Dividend>
>>
>> I want to merge both object's informations (Stock & Dividend) and send to another topic
>>
>> with for example <String, DividendStockJoin>
>>
>>
>> The <String> key of 2 two topic is the same. I need to use, leftJoin, merge, KTable, ...
>>
>> what is the best solution ? What do you think ?
>>
>>
>> Moreover, how can I make sure the timestamp are pretty much the same? (for merging same key at the same time produced) Example:
>>
>> KStream<String, Stock> --> key: "YHOO", and one timestamp of record 2018-04-08 19:56:30
>> KStream<String, Dividend> --> key: "YHOO", and one timestamp of record 2018-04-08 19:55:27
>>
>>
>> I need a Windowing ?
>>
>>
>> Great thank & best regards,
>>
>>
>> Adrien
>>
>
>
RE: join 2 topic streams --> to another topic
Posted by adrien ruffie <ad...@hotmail.fr>.
Hello Matthias,
thank for your response. I will try to read the blog post today.
For the keys, not really, In fact, keys are always the same "symbol" of a STOCK example "YHOO", "INTC" and they never change.
Do you with I need to use another key ? Or add a timestamp to the key ?
For single stock join with multiple dividends, I didn't think about it before ... is it possible ?
For join depending of timestamps why not, is it possible with windowing ?
Thank Matthias
Adrien
________________________________
De : Matthias J. Sax <ma...@confluent.io>
Envoyé : dimanche 8 avril 2018 23:04:24
À : users@kafka.apache.org
Objet : Re: join 2 topic streams --> to another topic
Check out this blog post that explain how the different joins work:
https://www.confluent.io/blog/crossing-streams-joins-apache-kafka/
It's hard to give a general answer -- it depends on the context of your
application. Are keys unique? Do you want to get exactly one result or
should a single stock join with multiple dividends? Do you want Stock
and Dividend join depending the their timestamps?
-Matthias
On 4/8/18 1:34 PM, adrien ruffie wrote:
> Hello all,
>
> I have 2 topics streamed by KStream<String, Stock> and one KStream<String, Dividend>
>
> I want to merge both object's informations (Stock & Dividend) and send to another topic
>
> with for example <String, DividendStockJoin>
>
>
> The <String> key of 2 two topic is the same. I need to use, leftJoin, merge, KTable, ...
>
> what is the best solution ? What do you think ?
>
>
> Moreover, how can I make sure the timestamp are pretty much the same? (for merging same key at the same time produced) Example:
>
> KStream<String, Stock> --> key: "YHOO", and one timestamp of record 2018-04-08 19:56:30
> KStream<String, Dividend> --> key: "YHOO", and one timestamp of record 2018-04-08 19:55:27
>
>
> I need a Windowing ?
>
>
> Great thank & best regards,
>
>
> Adrien
>