You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mikael Högqvist <ho...@gmail.com> on 2016/12/16 10:45:24 UTC

KTable#through from window store to key value store

Hi,

I have a small example topology that count words per minute (scala):

    words
      .map { (key, word) =>
        new KeyValue(word, Long.box(1L))
      }
      .groupByKey(Serdes.String, Serdes.Long)
      .count(TimeWindows.of(5 * 60 * 1000L), tableOne)
      .through(new WindowedSerde, Serdes.Long, s"$appId-count-topic",
tableTwo)

The first table, tableOne, is a WindowStore and can be accessed using fetch
on the key and time range. After using .through to forward the data to
another topic and table, tableTwo becomes a KeyValueStore. Is it possible
to keep tableTwo as a WindowStore also?

Best,
Mikael

Re: KTable#through from window store to key value store

Posted by Eno Thereska <en...@gmail.com>.
Ok, makes sense. We'll be putting out a KIP on updating the KTable semantics and one of the things we'll add is the ability to materialize any KTable without explicitely needing to call through. So you could do a .mapValues, you'll get a KTable back and then we'll probably have a .materialize API to allow you to query that. Stay tuned.

Thanks
Eno


> On 16 Dec 2016, at 18:44, Mikael Högqvist <ho...@gmail.com> wrote:
> 
> Hi Eno,
> 
> in this example it doesn't make much sense, but I could add a .mapValues
> that transforms the values and the use .through to materialize a table with
> the updated values.
> 
> It is possible to the access the new table as a KeyValueStore also, but it
> is much less convenient and I also expected that if I have a window store
> it would still be a window store after transformation/through.
> 
> Thanks,
> Mikael
> 
> On Fri, Dec 16, 2016 at 6:20 PM Eno Thereska <en...@gmail.com> wrote:
> 
>> Hi Mikael,
>> 
>> Currently that is not possible. Could you elaborate why you'd need that
>> since you can query from tableOne.
>> 
>> Thanks
>> Eno
>>> On 16 Dec 2016, at 10:45, Mikael Högqvist <ho...@gmail.com> wrote:
>>> 
>>> Hi,
>>> 
>>> I have a small example topology that count words per minute (scala):
>>> 
>>>   words
>>>     .map { (key, word) =>
>>>       new KeyValue(word, Long.box(1L))
>>>     }
>>>     .groupByKey(Serdes.String, Serdes.Long)
>>>     .count(TimeWindows.of(5 * 60 * 1000L), tableOne)
>>>     .through(new WindowedSerde, Serdes.Long, s"$appId-count-topic",
>>> tableTwo)
>>> 
>>> The first table, tableOne, is a WindowStore and can be accessed using
>> fetch
>>> on the key and time range. After using .through to forward the data to
>>> another topic and table, tableTwo becomes a KeyValueStore. Is it possible
>>> to keep tableTwo as a WindowStore also?
>>> 
>>> Best,
>>> Mikael
>> 
>>

Re: KTable#through from window store to key value store

Posted by Mikael Högqvist <ho...@gmail.com>.
Hi Eno,

in this example it doesn't make much sense, but I could add a .mapValues
that transforms the values and the use .through to materialize a table with
the updated values.

It is possible to the access the new table as a KeyValueStore also, but it
is much less convenient and I also expected that if I have a window store
it would still be a window store after transformation/through.

Thanks,
Mikael

On Fri, Dec 16, 2016 at 6:20 PM Eno Thereska <en...@gmail.com> wrote:

> Hi Mikael,
>
> Currently that is not possible. Could you elaborate why you'd need that
> since you can query from tableOne.
>
> Thanks
> Eno
> > On 16 Dec 2016, at 10:45, Mikael Högqvist <ho...@gmail.com> wrote:
> >
> > Hi,
> >
> > I have a small example topology that count words per minute (scala):
> >
> >    words
> >      .map { (key, word) =>
> >        new KeyValue(word, Long.box(1L))
> >      }
> >      .groupByKey(Serdes.String, Serdes.Long)
> >      .count(TimeWindows.of(5 * 60 * 1000L), tableOne)
> >      .through(new WindowedSerde, Serdes.Long, s"$appId-count-topic",
> > tableTwo)
> >
> > The first table, tableOne, is a WindowStore and can be accessed using
> fetch
> > on the key and time range. After using .through to forward the data to
> > another topic and table, tableTwo becomes a KeyValueStore. Is it possible
> > to keep tableTwo as a WindowStore also?
> >
> > Best,
> > Mikael
>
>

Re: KTable#through from window store to key value store

Posted by Eno Thereska <en...@gmail.com>.
Hi Mikael,

Currently that is not possible. Could you elaborate why you'd need that since you can query from tableOne.

Thanks
Eno
> On 16 Dec 2016, at 10:45, Mikael Högqvist <ho...@gmail.com> wrote:
> 
> Hi,
> 
> I have a small example topology that count words per minute (scala):
> 
>    words
>      .map { (key, word) =>
>        new KeyValue(word, Long.box(1L))
>      }
>      .groupByKey(Serdes.String, Serdes.Long)
>      .count(TimeWindows.of(5 * 60 * 1000L), tableOne)
>      .through(new WindowedSerde, Serdes.Long, s"$appId-count-topic",
> tableTwo)
> 
> The first table, tableOne, is a WindowStore and can be accessed using fetch
> on the key and time range. After using .through to forward the data to
> another topic and table, tableTwo becomes a KeyValueStore. Is it possible
> to keep tableTwo as a WindowStore also?
> 
> Best,
> Mikael