You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Nick DeCoursin <n....@foodpanda.com> on 2017/01/24 09:31:48 UTC
Table a KStream
Hello,
How can I simply table a Kafka Stream? I have a Kafka Stream, and I want to
create a table from it backed by a state store. The key of the stream could
be the same as the table.
I've tried following examples, but it seems all examples use `groupBy` or
`count` to convert `KStream`s into `KTable`s. Is there any other way?
Thank you very much,
Nick DeCoursin
--
Nick DeCoursin
Software Engineer
foodpanda
Tel | +1 920 450 5434
Mail | n.decoursin@foodpanda.com
Skype | nick.foodpanda
Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
USt-ID-Nr | DE 283789080
Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
CONFIDENTIALITY NOTICE: This message (including any attachments) is
confidential and may be privileged. It may be read, copied and used only by
the intended recipient. If you have received it in error please contact the
sender (by return e-mail) immediately and delete this message. Any
unauthorized use or dissemination of this message in whole or in parts is
strictly prohibited.
Re: Table a KStream
Posted by Elliot Crosby-McCullough <el...@gmail.com>.
Thanks!
On 11 February 2017 at 18:39, Eno Thereska <en...@gmail.com> wrote:
> Yes you are correct Elliot. If you look at it just from the point of view
> of what's in the topic, and how the topic is configured (e.g., with
> "compact") there is no difference.
>
>
> Thanks
> Eno
>
> > On 11 Feb 2017, at 17:56, Elliot Crosby-McCullough <el...@gmail.com>
> wrote:
> >
> > For my own clarity, is there any actual distinction between
> > `stream.to('topic')`
> > where `topic` is set to compact and the upcoming
> `stream.toTable('topic')`
> > if you're not going to immediately use the table in this topology, i.e.
> if
> > you want to use it as a table in some other processor application? Am I
> > right in my understanding that the table aspect is related to behaviour
> > within this specific processor (GlobalKTables aside)?
> >
> > On 10 February 2017 at 21:59, Matthias J. Sax <ma...@confluent.io>
> wrote:
> >
> >> I agree that the API can be improved and we are working on that.
> >>
> >> Btw: KStream#toTable() was already suggested in KIP-114 discussion:
> >>
> >> http://search-hadoop.com/m/Kafka/uyzND19QaLMqiR2e1?subj=
> >> Re+DISCUSS+KIP+114+KTable+materialization+and+improved+semantics
> >>
> >>
> >> However for now, you can only choose from the two options as described.
> >>
> >>
> >> -Matthias
> >>
> >>
> >>
> >> On 2/10/17 1:49 PM, Nick DeCoursin wrote:
> >>> To be honest, I don't think either of these options are very good.
> >>>
> >>>
> >>> stream.to("some-other-topic");
> >>> builder.table("some-other-topic");
> >>>
> >>>
> >>> As explained here
> >>> <http://mail-archives.apache.org/mod_mbox/kafka-users/
> >> 201702.mbox/browser>,
> >>> if the underlying topic doesn't have cleanup.policy=compact, the state
> >>> store may invisibly lose data!
> >>>
> >>> As for the second option, it's obviously verbose, and it doesn't let me
> >>> choose the topic name. As a result, I end with some long weird
> changelog
> >>> topic name; and, some of the Kafka Connectors map topics to the table
> >> name,
> >>> eeeh.
> >>>
> >>> To improve this, just add a topic parameter to both of the overloaded
> >>> KGroupedStream.reduce methods:
> >>>
> >>> reduce(Reducer<V> reducer, String, topicName, String storeName)
> >>>
> >>>
> >>> Furthermore, there should be a short cut to the second option, like
> this:
> >>>
> >>> // Creates a KTable from the KStream where
> >>> // the key of the KStream is the key of the KTable.
> >>> // Any latter key overwrites the former.
> >>> someStream.table(
> >>> Serde<K>,
> >>> Serde<V>,
> >>> topicName,
> >>> tableName
> >>> );
> >>>
> >>>
> >>> or maybe the Serdes can be inferred? Either way, this would be a nice
> >> clean
> >>> approach to a (maybe?) common use case.
> >>>
> >>> Thank you,
> >>> Nick
> >>>
> >>> On 25 January 2017 at 07:46, Nick DeCoursin <n.decoursin@foodpanda.com
> >
> >>> wrote:
> >>>
> >>>> Thank you very much, both suggestions are wonderful, and I will try
> >> them.
> >>>> Have a great day!
> >>>>
> >>>> Kind regards,
> >>>> Nick
> >>>>
> >>>> On 24 January 2017 at 19:46, Matthias J. Sax <ma...@confluent.io>
> >>>> wrote:
> >>>>
> >>>>> If your data is already partitioned by key, you can save writing to a
> >>>>> topic by doing a dummy reduce instead:
> >>>>>
> >>>>> stream
> >>>>> .groupByKey()
> >>>>> .reduce(new Reducer() {
> >>>>> V apply(V value1, V value2) {
> >>>>> return value2;
> >>>>> }
> >>>>> },
> >>>>> "yourStoreName");
> >>>>>
> >>>>> (replace V with your actuall value type)
> >>>>>
> >>>>>
> >>>>> -Matthias
> >>>>>
> >>>>> On 1/24/17 8:53 AM, Damian Guy wrote:
> >>>>>> Hi Nick,
> >>>>>>
> >>>>>> I guess there is some reason why you can't just build it as a table
> to
> >>>>>> begin with?
> >>>>>> There isn't a convenient method for doing this right now, but you
> >> could
> >>>>> do
> >>>>>> something like:
> >>>>>>
> >>>>>> stream.to("some-other-topic");
> >>>>>> builder.table("some-other-topic");
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Damian
> >>>>>>
> >>>>>> On Tue, 24 Jan 2017 at 16:32 Nick DeCoursin <
> >> n.decoursin@foodpanda.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>> How can I simply table a Kafka Stream? I have a Kafka Stream, and I
> >>>>> want to
> >>>>>>> create a table from it backed by a state store. The key of the
> stream
> >>>>> could
> >>>>>>> be the same as the table.
> >>>>>>>
> >>>>>>> I've tried following examples, but it seems all examples use
> >> `groupBy`
> >>>>> or
> >>>>>>> `count` to convert `KStream`s into `KTable`s. Is there any other
> way?
> >>>>>>>
> >>>>>>> Thank you very much,
> >>>>>>> Nick DeCoursin
> >>>>>>>
> >>>>>>> --
> >>>>>>>
> >>>>>>> Nick DeCoursin
> >>>>>>> Software Engineer
> >>>>>>> foodpanda
> >>>>>>>
> >>>>>>> Tel | +1 920 450 5434 <(920)%20450-5434>
> >>>>>>>
> >>>>>>> Mail | n.decoursin@foodpanda.com
> >>>>>>>
> >>>>>>> Skype | nick.foodpanda
> >>>>>>>
> >>>>>>> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
> >>>>>>> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
> >>>>>>> USt-ID-Nr | DE 283789080
> >>>>>>> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
> >>>>>>>
> >>>>>>> CONFIDENTIALITY NOTICE: This message (including any attachments) is
> >>>>>>> confidential and may be privileged. It may be read, copied and used
> >>>>> only by
> >>>>>>> the intended recipient. If you have received it in error please
> >>>>> contact the
> >>>>>>> sender (by return e-mail) immediately and delete this message. Any
> >>>>>>> unauthorized use or dissemination of this message in whole or in
> >> parts
> >>>>> is
> >>>>>>> strictly prohibited.
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>>
> >>>> Nick DeCoursin
> >>>> Software Engineer
> >>>> foodpanda
> >>>>
> >>>> Tel | +1 920 450 5434 <(920)%20450-5434>
> >>>>
> >>>> Mail | n.decoursin@foodpanda.com
> >>>>
> >>>> Skype | nick.foodpanda
> >>>>
> >>>> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
> >>>> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
> >>>> USt-ID-Nr | DE 283789080
> >>>> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
> >>>>
> >>>> CONFIDENTIALITY NOTICE: This message (including any attachments) is
> >>>> confidential and may be privileged. It may be read, copied and used
> >> only by
> >>>> the intended recipient. If you have received it in error please
> contact
> >> the
> >>>> sender (by return e-mail) immediately and delete this message. Any
> >>>> unauthorized use or dissemination of this message in whole or in parts
> >> is
> >>>> strictly prohibited.
> >>>>
> >>>
> >>>
> >>>
> >>
> >>
>
>
Re: Table a KStream
Posted by Eno Thereska <en...@gmail.com>.
Yes you are correct Elliot. If you look at it just from the point of view of what's in the topic, and how the topic is configured (e.g., with "compact") there is no difference.
Thanks
Eno
> On 11 Feb 2017, at 17:56, Elliot Crosby-McCullough <el...@gmail.com> wrote:
>
> For my own clarity, is there any actual distinction between
> `stream.to('topic')`
> where `topic` is set to compact and the upcoming `stream.toTable('topic')`
> if you're not going to immediately use the table in this topology, i.e. if
> you want to use it as a table in some other processor application? Am I
> right in my understanding that the table aspect is related to behaviour
> within this specific processor (GlobalKTables aside)?
>
> On 10 February 2017 at 21:59, Matthias J. Sax <ma...@confluent.io> wrote:
>
>> I agree that the API can be improved and we are working on that.
>>
>> Btw: KStream#toTable() was already suggested in KIP-114 discussion:
>>
>> http://search-hadoop.com/m/Kafka/uyzND19QaLMqiR2e1?subj=
>> Re+DISCUSS+KIP+114+KTable+materialization+and+improved+semantics
>>
>>
>> However for now, you can only choose from the two options as described.
>>
>>
>> -Matthias
>>
>>
>>
>> On 2/10/17 1:49 PM, Nick DeCoursin wrote:
>>> To be honest, I don't think either of these options are very good.
>>>
>>>
>>> stream.to("some-other-topic");
>>> builder.table("some-other-topic");
>>>
>>>
>>> As explained here
>>> <http://mail-archives.apache.org/mod_mbox/kafka-users/
>> 201702.mbox/browser>,
>>> if the underlying topic doesn't have cleanup.policy=compact, the state
>>> store may invisibly lose data!
>>>
>>> As for the second option, it's obviously verbose, and it doesn't let me
>>> choose the topic name. As a result, I end with some long weird changelog
>>> topic name; and, some of the Kafka Connectors map topics to the table
>> name,
>>> eeeh.
>>>
>>> To improve this, just add a topic parameter to both of the overloaded
>>> KGroupedStream.reduce methods:
>>>
>>> reduce(Reducer<V> reducer, String, topicName, String storeName)
>>>
>>>
>>> Furthermore, there should be a short cut to the second option, like this:
>>>
>>> // Creates a KTable from the KStream where
>>> // the key of the KStream is the key of the KTable.
>>> // Any latter key overwrites the former.
>>> someStream.table(
>>> Serde<K>,
>>> Serde<V>,
>>> topicName,
>>> tableName
>>> );
>>>
>>>
>>> or maybe the Serdes can be inferred? Either way, this would be a nice
>> clean
>>> approach to a (maybe?) common use case.
>>>
>>> Thank you,
>>> Nick
>>>
>>> On 25 January 2017 at 07:46, Nick DeCoursin <n....@foodpanda.com>
>>> wrote:
>>>
>>>> Thank you very much, both suggestions are wonderful, and I will try
>> them.
>>>> Have a great day!
>>>>
>>>> Kind regards,
>>>> Nick
>>>>
>>>> On 24 January 2017 at 19:46, Matthias J. Sax <ma...@confluent.io>
>>>> wrote:
>>>>
>>>>> If your data is already partitioned by key, you can save writing to a
>>>>> topic by doing a dummy reduce instead:
>>>>>
>>>>> stream
>>>>> .groupByKey()
>>>>> .reduce(new Reducer() {
>>>>> V apply(V value1, V value2) {
>>>>> return value2;
>>>>> }
>>>>> },
>>>>> "yourStoreName");
>>>>>
>>>>> (replace V with your actuall value type)
>>>>>
>>>>>
>>>>> -Matthias
>>>>>
>>>>> On 1/24/17 8:53 AM, Damian Guy wrote:
>>>>>> Hi Nick,
>>>>>>
>>>>>> I guess there is some reason why you can't just build it as a table to
>>>>>> begin with?
>>>>>> There isn't a convenient method for doing this right now, but you
>> could
>>>>> do
>>>>>> something like:
>>>>>>
>>>>>> stream.to("some-other-topic");
>>>>>> builder.table("some-other-topic");
>>>>>>
>>>>>> Thanks,
>>>>>> Damian
>>>>>>
>>>>>> On Tue, 24 Jan 2017 at 16:32 Nick DeCoursin <
>> n.decoursin@foodpanda.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> How can I simply table a Kafka Stream? I have a Kafka Stream, and I
>>>>> want to
>>>>>>> create a table from it backed by a state store. The key of the stream
>>>>> could
>>>>>>> be the same as the table.
>>>>>>>
>>>>>>> I've tried following examples, but it seems all examples use
>> `groupBy`
>>>>> or
>>>>>>> `count` to convert `KStream`s into `KTable`s. Is there any other way?
>>>>>>>
>>>>>>> Thank you very much,
>>>>>>> Nick DeCoursin
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Nick DeCoursin
>>>>>>> Software Engineer
>>>>>>> foodpanda
>>>>>>>
>>>>>>> Tel | +1 920 450 5434 <(920)%20450-5434>
>>>>>>>
>>>>>>> Mail | n.decoursin@foodpanda.com
>>>>>>>
>>>>>>> Skype | nick.foodpanda
>>>>>>>
>>>>>>> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
>>>>>>> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
>>>>>>> USt-ID-Nr | DE 283789080
>>>>>>> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
>>>>>>>
>>>>>>> CONFIDENTIALITY NOTICE: This message (including any attachments) is
>>>>>>> confidential and may be privileged. It may be read, copied and used
>>>>> only by
>>>>>>> the intended recipient. If you have received it in error please
>>>>> contact the
>>>>>>> sender (by return e-mail) immediately and delete this message. Any
>>>>>>> unauthorized use or dissemination of this message in whole or in
>> parts
>>>>> is
>>>>>>> strictly prohibited.
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Nick DeCoursin
>>>> Software Engineer
>>>> foodpanda
>>>>
>>>> Tel | +1 920 450 5434 <(920)%20450-5434>
>>>>
>>>> Mail | n.decoursin@foodpanda.com
>>>>
>>>> Skype | nick.foodpanda
>>>>
>>>> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
>>>> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
>>>> USt-ID-Nr | DE 283789080
>>>> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
>>>>
>>>> CONFIDENTIALITY NOTICE: This message (including any attachments) is
>>>> confidential and may be privileged. It may be read, copied and used
>> only by
>>>> the intended recipient. If you have received it in error please contact
>> the
>>>> sender (by return e-mail) immediately and delete this message. Any
>>>> unauthorized use or dissemination of this message in whole or in parts
>> is
>>>> strictly prohibited.
>>>>
>>>
>>>
>>>
>>
>>
Re: Table a KStream
Posted by Elliot Crosby-McCullough <el...@gmail.com>.
For my own clarity, is there any actual distinction between
`stream.to('topic')`
where `topic` is set to compact and the upcoming `stream.toTable('topic')`
if you're not going to immediately use the table in this topology, i.e. if
you want to use it as a table in some other processor application? Am I
right in my understanding that the table aspect is related to behaviour
within this specific processor (GlobalKTables aside)?
On 10 February 2017 at 21:59, Matthias J. Sax <ma...@confluent.io> wrote:
> I agree that the API can be improved and we are working on that.
>
> Btw: KStream#toTable() was already suggested in KIP-114 discussion:
>
> http://search-hadoop.com/m/Kafka/uyzND19QaLMqiR2e1?subj=
> Re+DISCUSS+KIP+114+KTable+materialization+and+improved+semantics
>
>
> However for now, you can only choose from the two options as described.
>
>
> -Matthias
>
>
>
> On 2/10/17 1:49 PM, Nick DeCoursin wrote:
> > To be honest, I don't think either of these options are very good.
> >
> >
> > stream.to("some-other-topic");
> > builder.table("some-other-topic");
> >
> >
> > As explained here
> > <http://mail-archives.apache.org/mod_mbox/kafka-users/
> 201702.mbox/browser>,
> > if the underlying topic doesn't have cleanup.policy=compact, the state
> > store may invisibly lose data!
> >
> > As for the second option, it's obviously verbose, and it doesn't let me
> > choose the topic name. As a result, I end with some long weird changelog
> > topic name; and, some of the Kafka Connectors map topics to the table
> name,
> > eeeh.
> >
> > To improve this, just add a topic parameter to both of the overloaded
> > KGroupedStream.reduce methods:
> >
> > reduce(Reducer<V> reducer, String, topicName, String storeName)
> >
> >
> > Furthermore, there should be a short cut to the second option, like this:
> >
> > // Creates a KTable from the KStream where
> > // the key of the KStream is the key of the KTable.
> > // Any latter key overwrites the former.
> > someStream.table(
> > Serde<K>,
> > Serde<V>,
> > topicName,
> > tableName
> > );
> >
> >
> > or maybe the Serdes can be inferred? Either way, this would be a nice
> clean
> > approach to a (maybe?) common use case.
> >
> > Thank you,
> > Nick
> >
> > On 25 January 2017 at 07:46, Nick DeCoursin <n....@foodpanda.com>
> > wrote:
> >
> >> Thank you very much, both suggestions are wonderful, and I will try
> them.
> >> Have a great day!
> >>
> >> Kind regards,
> >> Nick
> >>
> >> On 24 January 2017 at 19:46, Matthias J. Sax <ma...@confluent.io>
> >> wrote:
> >>
> >>> If your data is already partitioned by key, you can save writing to a
> >>> topic by doing a dummy reduce instead:
> >>>
> >>> stream
> >>> .groupByKey()
> >>> .reduce(new Reducer() {
> >>> V apply(V value1, V value2) {
> >>> return value2;
> >>> }
> >>> },
> >>> "yourStoreName");
> >>>
> >>> (replace V with your actuall value type)
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 1/24/17 8:53 AM, Damian Guy wrote:
> >>>> Hi Nick,
> >>>>
> >>>> I guess there is some reason why you can't just build it as a table to
> >>>> begin with?
> >>>> There isn't a convenient method for doing this right now, but you
> could
> >>> do
> >>>> something like:
> >>>>
> >>>> stream.to("some-other-topic");
> >>>> builder.table("some-other-topic");
> >>>>
> >>>> Thanks,
> >>>> Damian
> >>>>
> >>>> On Tue, 24 Jan 2017 at 16:32 Nick DeCoursin <
> n.decoursin@foodpanda.com>
> >>>> wrote:
> >>>>
> >>>>> Hello,
> >>>>>
> >>>>> How can I simply table a Kafka Stream? I have a Kafka Stream, and I
> >>> want to
> >>>>> create a table from it backed by a state store. The key of the stream
> >>> could
> >>>>> be the same as the table.
> >>>>>
> >>>>> I've tried following examples, but it seems all examples use
> `groupBy`
> >>> or
> >>>>> `count` to convert `KStream`s into `KTable`s. Is there any other way?
> >>>>>
> >>>>> Thank you very much,
> >>>>> Nick DeCoursin
> >>>>>
> >>>>> --
> >>>>>
> >>>>> Nick DeCoursin
> >>>>> Software Engineer
> >>>>> foodpanda
> >>>>>
> >>>>> Tel | +1 920 450 5434 <(920)%20450-5434>
> >>>>>
> >>>>> Mail | n.decoursin@foodpanda.com
> >>>>>
> >>>>> Skype | nick.foodpanda
> >>>>>
> >>>>> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
> >>>>> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
> >>>>> USt-ID-Nr | DE 283789080
> >>>>> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
> >>>>>
> >>>>> CONFIDENTIALITY NOTICE: This message (including any attachments) is
> >>>>> confidential and may be privileged. It may be read, copied and used
> >>> only by
> >>>>> the intended recipient. If you have received it in error please
> >>> contact the
> >>>>> sender (by return e-mail) immediately and delete this message. Any
> >>>>> unauthorized use or dissemination of this message in whole or in
> parts
> >>> is
> >>>>> strictly prohibited.
> >>>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >> --
> >>
> >> Nick DeCoursin
> >> Software Engineer
> >> foodpanda
> >>
> >> Tel | +1 920 450 5434 <(920)%20450-5434>
> >>
> >> Mail | n.decoursin@foodpanda.com
> >>
> >> Skype | nick.foodpanda
> >>
> >> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
> >> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
> >> USt-ID-Nr | DE 283789080
> >> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
> >>
> >> CONFIDENTIALITY NOTICE: This message (including any attachments) is
> >> confidential and may be privileged. It may be read, copied and used
> only by
> >> the intended recipient. If you have received it in error please contact
> the
> >> sender (by return e-mail) immediately and delete this message. Any
> >> unauthorized use or dissemination of this message in whole or in parts
> is
> >> strictly prohibited.
> >>
> >
> >
> >
>
>
Re: Table a KStream
Posted by Nick DeCoursin <n....@foodpanda.com>.
Sounds good, thank you!
Kind regards,
Nick
On 10 February 2017 at 22:59, Matthias J. Sax <ma...@confluent.io> wrote:
> I agree that the API can be improved and we are working on that.
>
> Btw: KStream#toTable() was already suggested in KIP-114 discussion:
>
> http://search-hadoop.com/m/Kafka/uyzND19QaLMqiR2e1?subj=
> Re+DISCUSS+KIP+114+KTable+materialization+and+improved+semantics
>
>
> However for now, you can only choose from the two options as described.
>
>
> -Matthias
>
>
>
> On 2/10/17 1:49 PM, Nick DeCoursin wrote:
> > To be honest, I don't think either of these options are very good.
> >
> >
> > stream.to("some-other-topic");
> > builder.table("some-other-topic");
> >
> >
> > As explained here
> > <http://mail-archives.apache.org/mod_mbox/kafka-users/
> 201702.mbox/browser>,
> > if the underlying topic doesn't have cleanup.policy=compact, the state
> > store may invisibly lose data!
> >
> > As for the second option, it's obviously verbose, and it doesn't let me
> > choose the topic name. As a result, I end with some long weird changelog
> > topic name; and, some of the Kafka Connectors map topics to the table
> name,
> > eeeh.
> >
> > To improve this, just add a topic parameter to both of the overloaded
> > KGroupedStream.reduce methods:
> >
> > reduce(Reducer<V> reducer, String, topicName, String storeName)
> >
> >
> > Furthermore, there should be a short cut to the second option, like this:
> >
> > // Creates a KTable from the KStream where
> > // the key of the KStream is the key of the KTable.
> > // Any latter key overwrites the former.
> > someStream.table(
> > Serde<K>,
> > Serde<V>,
> > topicName,
> > tableName
> > );
> >
> >
> > or maybe the Serdes can be inferred? Either way, this would be a nice
> clean
> > approach to a (maybe?) common use case.
> >
> > Thank you,
> > Nick
> >
> > On 25 January 2017 at 07:46, Nick DeCoursin <n....@foodpanda.com>
> > wrote:
> >
> >> Thank you very much, both suggestions are wonderful, and I will try
> them.
> >> Have a great day!
> >>
> >> Kind regards,
> >> Nick
> >>
> >> On 24 January 2017 at 19:46, Matthias J. Sax <ma...@confluent.io>
> >> wrote:
> >>
> >>> If your data is already partitioned by key, you can save writing to a
> >>> topic by doing a dummy reduce instead:
> >>>
> >>> stream
> >>> .groupByKey()
> >>> .reduce(new Reducer() {
> >>> V apply(V value1, V value2) {
> >>> return value2;
> >>> }
> >>> },
> >>> "yourStoreName");
> >>>
> >>> (replace V with your actuall value type)
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 1/24/17 8:53 AM, Damian Guy wrote:
> >>>> Hi Nick,
> >>>>
> >>>> I guess there is some reason why you can't just build it as a table to
> >>>> begin with?
> >>>> There isn't a convenient method for doing this right now, but you
> could
> >>> do
> >>>> something like:
> >>>>
> >>>> stream.to("some-other-topic");
> >>>> builder.table("some-other-topic");
> >>>>
> >>>> Thanks,
> >>>> Damian
> >>>>
> >>>> On Tue, 24 Jan 2017 at 16:32 Nick DeCoursin <
> n.decoursin@foodpanda.com>
> >>>> wrote:
> >>>>
> >>>>> Hello,
> >>>>>
> >>>>> How can I simply table a Kafka Stream? I have a Kafka Stream, and I
> >>> want to
> >>>>> create a table from it backed by a state store. The key of the stream
> >>> could
> >>>>> be the same as the table.
> >>>>>
> >>>>> I've tried following examples, but it seems all examples use
> `groupBy`
> >>> or
> >>>>> `count` to convert `KStream`s into `KTable`s. Is there any other way?
> >>>>>
> >>>>> Thank you very much,
> >>>>> Nick DeCoursin
> >>>>>
> >>>>> --
> >>>>>
> >>>>> Nick DeCoursin
> >>>>> Software Engineer
> >>>>> foodpanda
> >>>>>
> >>>>> Tel | +1 920 450 5434 <(920)%20450-5434>
> >>>>>
> >>>>> Mail | n.decoursin@foodpanda.com
> >>>>>
> >>>>> Skype | nick.foodpanda
> >>>>>
> >>>>> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
> >>>>> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
> >>>>> USt-ID-Nr | DE 283789080
> >>>>> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
> >>>>>
> >>>>> CONFIDENTIALITY NOTICE: This message (including any attachments) is
> >>>>> confidential and may be privileged. It may be read, copied and used
> >>> only by
> >>>>> the intended recipient. If you have received it in error please
> >>> contact the
> >>>>> sender (by return e-mail) immediately and delete this message. Any
> >>>>> unauthorized use or dissemination of this message in whole or in
> parts
> >>> is
> >>>>> strictly prohibited.
> >>>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >> --
> >>
> >> Nick DeCoursin
> >> Software Engineer
> >> foodpanda
> >>
> >> Tel | +1 920 450 5434 <(920)%20450-5434>
> >>
> >> Mail | n.decoursin@foodpanda.com
> >>
> >> Skype | nick.foodpanda
> >>
> >> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
> >> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
> >> USt-ID-Nr | DE 283789080
> >> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
> >>
> >> CONFIDENTIALITY NOTICE: This message (including any attachments) is
> >> confidential and may be privileged. It may be read, copied and used
> only by
> >> the intended recipient. If you have received it in error please contact
> the
> >> sender (by return e-mail) immediately and delete this message. Any
> >> unauthorized use or dissemination of this message in whole or in parts
> is
> >> strictly prohibited.
> >>
> >
> >
> >
>
>
--
Nick DeCoursin
Software Engineer
foodpanda
Tel | +1 920 450 5434
Mail | n.decoursin@foodpanda.com
Skype | nick.foodpanda
Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
USt-ID-Nr | DE 283789080
Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
CONFIDENTIALITY NOTICE: This message (including any attachments) is
confidential and may be privileged. It may be read, copied and used only by
the intended recipient. If you have received it in error please contact the
sender (by return e-mail) immediately and delete this message. Any
unauthorized use or dissemination of this message in whole or in parts is
strictly prohibited.
Re: Table a KStream
Posted by "Matthias J. Sax" <ma...@confluent.io>.
I agree that the API can be improved and we are working on that.
Btw: KStream#toTable() was already suggested in KIP-114 discussion:
http://search-hadoop.com/m/Kafka/uyzND19QaLMqiR2e1?subj=Re+DISCUSS+KIP+114+KTable+materialization+and+improved+semantics
However for now, you can only choose from the two options as described.
-Matthias
On 2/10/17 1:49 PM, Nick DeCoursin wrote:
> To be honest, I don't think either of these options are very good.
>
>
> stream.to("some-other-topic");
> builder.table("some-other-topic");
>
>
> As explained here
> <http://mail-archives.apache.org/mod_mbox/kafka-users/201702.mbox/browser>,
> if the underlying topic doesn't have cleanup.policy=compact, the state
> store may invisibly lose data!
>
> As for the second option, it's obviously verbose, and it doesn't let me
> choose the topic name. As a result, I end with some long weird changelog
> topic name; and, some of the Kafka Connectors map topics to the table name,
> eeeh.
>
> To improve this, just add a topic parameter to both of the overloaded
> KGroupedStream.reduce methods:
>
> reduce(Reducer<V> reducer, String, topicName, String storeName)
>
>
> Furthermore, there should be a short cut to the second option, like this:
>
> // Creates a KTable from the KStream where
> // the key of the KStream is the key of the KTable.
> // Any latter key overwrites the former.
> someStream.table(
> Serde<K>,
> Serde<V>,
> topicName,
> tableName
> );
>
>
> or maybe the Serdes can be inferred? Either way, this would be a nice clean
> approach to a (maybe?) common use case.
>
> Thank you,
> Nick
>
> On 25 January 2017 at 07:46, Nick DeCoursin <n....@foodpanda.com>
> wrote:
>
>> Thank you very much, both suggestions are wonderful, and I will try them.
>> Have a great day!
>>
>> Kind regards,
>> Nick
>>
>> On 24 January 2017 at 19:46, Matthias J. Sax <ma...@confluent.io>
>> wrote:
>>
>>> If your data is already partitioned by key, you can save writing to a
>>> topic by doing a dummy reduce instead:
>>>
>>> stream
>>> .groupByKey()
>>> .reduce(new Reducer() {
>>> V apply(V value1, V value2) {
>>> return value2;
>>> }
>>> },
>>> "yourStoreName");
>>>
>>> (replace V with your actuall value type)
>>>
>>>
>>> -Matthias
>>>
>>> On 1/24/17 8:53 AM, Damian Guy wrote:
>>>> Hi Nick,
>>>>
>>>> I guess there is some reason why you can't just build it as a table to
>>>> begin with?
>>>> There isn't a convenient method for doing this right now, but you could
>>> do
>>>> something like:
>>>>
>>>> stream.to("some-other-topic");
>>>> builder.table("some-other-topic");
>>>>
>>>> Thanks,
>>>> Damian
>>>>
>>>> On Tue, 24 Jan 2017 at 16:32 Nick DeCoursin <n....@foodpanda.com>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> How can I simply table a Kafka Stream? I have a Kafka Stream, and I
>>> want to
>>>>> create a table from it backed by a state store. The key of the stream
>>> could
>>>>> be the same as the table.
>>>>>
>>>>> I've tried following examples, but it seems all examples use `groupBy`
>>> or
>>>>> `count` to convert `KStream`s into `KTable`s. Is there any other way?
>>>>>
>>>>> Thank you very much,
>>>>> Nick DeCoursin
>>>>>
>>>>> --
>>>>>
>>>>> Nick DeCoursin
>>>>> Software Engineer
>>>>> foodpanda
>>>>>
>>>>> Tel | +1 920 450 5434 <(920)%20450-5434>
>>>>>
>>>>> Mail | n.decoursin@foodpanda.com
>>>>>
>>>>> Skype | nick.foodpanda
>>>>>
>>>>> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
>>>>> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
>>>>> USt-ID-Nr | DE 283789080
>>>>> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
>>>>>
>>>>> CONFIDENTIALITY NOTICE: This message (including any attachments) is
>>>>> confidential and may be privileged. It may be read, copied and used
>>> only by
>>>>> the intended recipient. If you have received it in error please
>>> contact the
>>>>> sender (by return e-mail) immediately and delete this message. Any
>>>>> unauthorized use or dissemination of this message in whole or in parts
>>> is
>>>>> strictly prohibited.
>>>>>
>>>>
>>>
>>>
>>
>>
>> --
>>
>> Nick DeCoursin
>> Software Engineer
>> foodpanda
>>
>> Tel | +1 920 450 5434 <(920)%20450-5434>
>>
>> Mail | n.decoursin@foodpanda.com
>>
>> Skype | nick.foodpanda
>>
>> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
>> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
>> USt-ID-Nr | DE 283789080
>> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
>>
>> CONFIDENTIALITY NOTICE: This message (including any attachments) is
>> confidential and may be privileged. It may be read, copied and used only by
>> the intended recipient. If you have received it in error please contact the
>> sender (by return e-mail) immediately and delete this message. Any
>> unauthorized use or dissemination of this message in whole or in parts is
>> strictly prohibited.
>>
>
>
>
Re: Table a KStream
Posted by Nick DeCoursin <n....@foodpanda.com>.
To be honest, I don't think either of these options are very good.
stream.to("some-other-topic");
builder.table("some-other-topic");
As explained here
<http://mail-archives.apache.org/mod_mbox/kafka-users/201702.mbox/browser>,
if the underlying topic doesn't have cleanup.policy=compact, the state
store may invisibly lose data!
As for the second option, it's obviously verbose, and it doesn't let me
choose the topic name. As a result, I end with some long weird changelog
topic name; and, some of the Kafka Connectors map topics to the table name,
eeeh.
To improve this, just add a topic parameter to both of the overloaded
KGroupedStream.reduce methods:
reduce(Reducer<V> reducer, String, topicName, String storeName)
Furthermore, there should be a short cut to the second option, like this:
// Creates a KTable from the KStream where
// the key of the KStream is the key of the KTable.
// Any latter key overwrites the former.
someStream.table(
Serde<K>,
Serde<V>,
topicName,
tableName
);
or maybe the Serdes can be inferred? Either way, this would be a nice clean
approach to a (maybe?) common use case.
Thank you,
Nick
On 25 January 2017 at 07:46, Nick DeCoursin <n....@foodpanda.com>
wrote:
> Thank you very much, both suggestions are wonderful, and I will try them.
> Have a great day!
>
> Kind regards,
> Nick
>
> On 24 January 2017 at 19:46, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> If your data is already partitioned by key, you can save writing to a
>> topic by doing a dummy reduce instead:
>>
>> stream
>> .groupByKey()
>> .reduce(new Reducer() {
>> V apply(V value1, V value2) {
>> return value2;
>> }
>> },
>> "yourStoreName");
>>
>> (replace V with your actuall value type)
>>
>>
>> -Matthias
>>
>> On 1/24/17 8:53 AM, Damian Guy wrote:
>> > Hi Nick,
>> >
>> > I guess there is some reason why you can't just build it as a table to
>> > begin with?
>> > There isn't a convenient method for doing this right now, but you could
>> do
>> > something like:
>> >
>> > stream.to("some-other-topic");
>> > builder.table("some-other-topic");
>> >
>> > Thanks,
>> > Damian
>> >
>> > On Tue, 24 Jan 2017 at 16:32 Nick DeCoursin <n....@foodpanda.com>
>> > wrote:
>> >
>> >> Hello,
>> >>
>> >> How can I simply table a Kafka Stream? I have a Kafka Stream, and I
>> want to
>> >> create a table from it backed by a state store. The key of the stream
>> could
>> >> be the same as the table.
>> >>
>> >> I've tried following examples, but it seems all examples use `groupBy`
>> or
>> >> `count` to convert `KStream`s into `KTable`s. Is there any other way?
>> >>
>> >> Thank you very much,
>> >> Nick DeCoursin
>> >>
>> >> --
>> >>
>> >> Nick DeCoursin
>> >> Software Engineer
>> >> foodpanda
>> >>
>> >> Tel | +1 920 450 5434 <(920)%20450-5434>
>> >>
>> >> Mail | n.decoursin@foodpanda.com
>> >>
>> >> Skype | nick.foodpanda
>> >>
>> >> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
>> >> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
>> >> USt-ID-Nr | DE 283789080
>> >> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
>> >>
>> >> CONFIDENTIALITY NOTICE: This message (including any attachments) is
>> >> confidential and may be privileged. It may be read, copied and used
>> only by
>> >> the intended recipient. If you have received it in error please
>> contact the
>> >> sender (by return e-mail) immediately and delete this message. Any
>> >> unauthorized use or dissemination of this message in whole or in parts
>> is
>> >> strictly prohibited.
>> >>
>> >
>>
>>
>
>
> --
>
> Nick DeCoursin
> Software Engineer
> foodpanda
>
> Tel | +1 920 450 5434 <(920)%20450-5434>
>
> Mail | n.decoursin@foodpanda.com
>
> Skype | nick.foodpanda
>
> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
> USt-ID-Nr | DE 283789080
> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
>
> CONFIDENTIALITY NOTICE: This message (including any attachments) is
> confidential and may be privileged. It may be read, copied and used only by
> the intended recipient. If you have received it in error please contact the
> sender (by return e-mail) immediately and delete this message. Any
> unauthorized use or dissemination of this message in whole or in parts is
> strictly prohibited.
>
--
Nick DeCoursin
Software Engineer
foodpanda
Tel | +1 920 450 5434
Mail | n.decoursin@foodpanda.com
Skype | nick.foodpanda
Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
USt-ID-Nr | DE 283789080
Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
CONFIDENTIALITY NOTICE: This message (including any attachments) is
confidential and may be privileged. It may be read, copied and used only by
the intended recipient. If you have received it in error please contact the
sender (by return e-mail) immediately and delete this message. Any
unauthorized use or dissemination of this message in whole or in parts is
strictly prohibited.
Re: Table a KStream
Posted by Nick DeCoursin <n....@foodpanda.com>.
Thank you very much, both suggestions are wonderful, and I will try them.
Have a great day!
Kind regards,
Nick
On 24 January 2017 at 19:46, Matthias J. Sax <ma...@confluent.io> wrote:
> If your data is already partitioned by key, you can save writing to a
> topic by doing a dummy reduce instead:
>
> stream
> .groupByKey()
> .reduce(new Reducer() {
> V apply(V value1, V value2) {
> return value2;
> }
> },
> "yourStoreName");
>
> (replace V with your actuall value type)
>
>
> -Matthias
>
> On 1/24/17 8:53 AM, Damian Guy wrote:
> > Hi Nick,
> >
> > I guess there is some reason why you can't just build it as a table to
> > begin with?
> > There isn't a convenient method for doing this right now, but you could
> do
> > something like:
> >
> > stream.to("some-other-topic");
> > builder.table("some-other-topic");
> >
> > Thanks,
> > Damian
> >
> > On Tue, 24 Jan 2017 at 16:32 Nick DeCoursin <n....@foodpanda.com>
> > wrote:
> >
> >> Hello,
> >>
> >> How can I simply table a Kafka Stream? I have a Kafka Stream, and I
> want to
> >> create a table from it backed by a state store. The key of the stream
> could
> >> be the same as the table.
> >>
> >> I've tried following examples, but it seems all examples use `groupBy`
> or
> >> `count` to convert `KStream`s into `KTable`s. Is there any other way?
> >>
> >> Thank you very much,
> >> Nick DeCoursin
> >>
> >> --
> >>
> >> Nick DeCoursin
> >> Software Engineer
> >> foodpanda
> >>
> >> Tel | +1 920 450 5434 <(920)%20450-5434>
> >>
> >> Mail | n.decoursin@foodpanda.com
> >>
> >> Skype | nick.foodpanda
> >>
> >> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
> >> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
> >> USt-ID-Nr | DE 283789080
> >> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
> >>
> >> CONFIDENTIALITY NOTICE: This message (including any attachments) is
> >> confidential and may be privileged. It may be read, copied and used
> only by
> >> the intended recipient. If you have received it in error please contact
> the
> >> sender (by return e-mail) immediately and delete this message. Any
> >> unauthorized use or dissemination of this message in whole or in parts
> is
> >> strictly prohibited.
> >>
> >
>
>
--
Nick DeCoursin
Software Engineer
foodpanda
Tel | +1 920 450 5434
Mail | n.decoursin@foodpanda.com
Skype | nick.foodpanda
Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
USt-ID-Nr | DE 283789080
Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
CONFIDENTIALITY NOTICE: This message (including any attachments) is
confidential and may be privileged. It may be read, copied and used only by
the intended recipient. If you have received it in error please contact the
sender (by return e-mail) immediately and delete this message. Any
unauthorized use or dissemination of this message in whole or in parts is
strictly prohibited.
Re: Table a KStream
Posted by "Matthias J. Sax" <ma...@confluent.io>.
If your data is already partitioned by key, you can save writing to a
topic by doing a dummy reduce instead:
stream
.groupByKey()
.reduce(new Reducer() {
V apply(V value1, V value2) {
return value2;
}
},
"yourStoreName");
(replace V with your actuall value type)
-Matthias
On 1/24/17 8:53 AM, Damian Guy wrote:
> Hi Nick,
>
> I guess there is some reason why you can't just build it as a table to
> begin with?
> There isn't a convenient method for doing this right now, but you could do
> something like:
>
> stream.to("some-other-topic");
> builder.table("some-other-topic");
>
> Thanks,
> Damian
>
> On Tue, 24 Jan 2017 at 16:32 Nick DeCoursin <n....@foodpanda.com>
> wrote:
>
>> Hello,
>>
>> How can I simply table a Kafka Stream? I have a Kafka Stream, and I want to
>> create a table from it backed by a state store. The key of the stream could
>> be the same as the table.
>>
>> I've tried following examples, but it seems all examples use `groupBy` or
>> `count` to convert `KStream`s into `KTable`s. Is there any other way?
>>
>> Thank you very much,
>> Nick DeCoursin
>>
>> --
>>
>> Nick DeCoursin
>> Software Engineer
>> foodpanda
>>
>> Tel | +1 920 450 5434 <(920)%20450-5434>
>>
>> Mail | n.decoursin@foodpanda.com
>>
>> Skype | nick.foodpanda
>>
>> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
>> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
>> USt-ID-Nr | DE 283789080
>> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
>>
>> CONFIDENTIALITY NOTICE: This message (including any attachments) is
>> confidential and may be privileged. It may be read, copied and used only by
>> the intended recipient. If you have received it in error please contact the
>> sender (by return e-mail) immediately and delete this message. Any
>> unauthorized use or dissemination of this message in whole or in parts is
>> strictly prohibited.
>>
>
Re: Table a KStream
Posted by Damian Guy <da...@gmail.com>.
Hi Nick,
I guess there is some reason why you can't just build it as a table to
begin with?
There isn't a convenient method for doing this right now, but you could do
something like:
stream.to("some-other-topic");
builder.table("some-other-topic");
Thanks,
Damian
On Tue, 24 Jan 2017 at 16:32 Nick DeCoursin <n....@foodpanda.com>
wrote:
> Hello,
>
> How can I simply table a Kafka Stream? I have a Kafka Stream, and I want to
> create a table from it backed by a state store. The key of the stream could
> be the same as the table.
>
> I've tried following examples, but it seems all examples use `groupBy` or
> `count` to convert `KStream`s into `KTable`s. Is there any other way?
>
> Thank you very much,
> Nick DeCoursin
>
> --
>
> Nick DeCoursin
> Software Engineer
> foodpanda
>
> Tel | +1 920 450 5434 <(920)%20450-5434>
>
> Mail | n.decoursin@foodpanda.com
>
> Skype | nick.foodpanda
>
> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
> USt-ID-Nr | DE 283789080
> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
>
> CONFIDENTIALITY NOTICE: This message (including any attachments) is
> confidential and may be privileged. It may be read, copied and used only by
> the intended recipient. If you have received it in error please contact the
> sender (by return e-mail) immediately and delete this message. Any
> unauthorized use or dissemination of this message in whole or in parts is
> strictly prohibited.
>