You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Eno Thereska <en...@gmail.com> on 2016/09/08 17:07:21 UTC

Re: [VOTE] KIP-63: Unify store and downstream caching in streams

There have been a couple of changes to KIP-63 since the voting started, after more feedback, most notably the fact that this KIP applies to the DSL only, and not to the Processor API.

At this point I'd like to restart the voting process. 

Thanks
Eno

> On 31 Aug 2016, at 17:16, Jim Jagielski <ji...@jaguNET.com> wrote:
> 
> +1
>> On Aug 25, 2016, at 6:57 AM, Eno Thereska <en...@gmail.com> wrote:
>> 
>> Hi folks,
>> 
>> We'd like to start the vote for KIP-63. At this point the Wiki addresses
>> all previous questions and we believe the PoC is feature-complete.
>> 
>> Thanks
>> Eno
>

Re: [VOTE] KIP-63: Unify store and downstream caching in streams

Posted by Eno Thereska <en...@gmail.com>.
Guozhang,

I think the info is there. The next sentence says "The optimization is applicable to aggregations. It is not applicable to other operators like joins."
Also the bit about the per-thread cache is one paragraph lower: "Specifically, for a streams instance with T threads and cache size C, each thread will have an even C/T bytes of cache, to use as it sees fit among its tasks. No sharing of caches across threads will happen"

Thanks
Eno


> On 8 Sep 2016, at 18:51, Guozhang Wang <wa...@gmail.com> wrote:
> 
> Just read the KIP wiki again:
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-63%3A+Unify+store+and+downstream+caching+in+streams
> 
> Just one minor comments but otherwise I'm +1:
> 
> In "proposed changes" section: "The cache has two functions. First, it
> continues to serve as a read buffer for data that is sent to the state
> store, just like today. Second, it serves as a write deduplicator for the
> state store (just like today) as well as for the downstream processor
> node(s). "
> 
> I feel this is still a bit confusing. The caching layer is only turned on
> for state stores used in aggregate operators in the DSL, right? For example
> in KStream-KStream joins, we will not turn on caching since there is no
> updates on the state store on the same key.
> 
> So could we just propose "removing the caching layer inside the persistent
> state store engines (i.e. RocksDB), and instead add a per-thread global
> cache which will only be activated for state stores used in Streams DSL's
> aggregate operator as a write deduplicator for both the state store and to
> the downstream operators"?
> 
> Guozhang
> 
> 
> On Thu, Sep 8, 2016 at 10:07 AM, Eno Thereska <en...@gmail.com>
> wrote:
> 
>> There have been a couple of changes to KIP-63 since the voting started,
>> after more feedback, most notably the fact that this KIP applies to the DSL
>> only, and not to the Processor API.
>> 
>> At this point I'd like to restart the voting process.
>> 
>> Thanks
>> Eno
>> 
>>> On 31 Aug 2016, at 17:16, Jim Jagielski <ji...@jaguNET.com> wrote:
>>> 
>>> +1
>>>> On Aug 25, 2016, at 6:57 AM, Eno Thereska <en...@gmail.com>
>> wrote:
>>>> 
>>>> Hi folks,
>>>> 
>>>> We'd like to start the vote for KIP-63. At this point the Wiki addresses
>>>> all previous questions and we believe the PoC is feature-complete.
>>>> 
>>>> Thanks
>>>> Eno
>>> 
>> 
>> 
> 
> 
> -- 
> -- Guozhang

Re: [VOTE] KIP-63: Unify store and downstream caching in streams

Posted by Guozhang Wang <wa...@gmail.com>.
Just read the KIP wiki again:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-63%3A+Unify+store+and+downstream+caching+in+streams

Just one minor comments but otherwise I'm +1:

In "proposed changes" section: "The cache has two functions. First, it
continues to serve as a read buffer for data that is sent to the state
store, just like today. Second, it serves as a write deduplicator for the
state store (just like today) as well as for the downstream processor
node(s). "

I feel this is still a bit confusing. The caching layer is only turned on
for state stores used in aggregate operators in the DSL, right? For example
in KStream-KStream joins, we will not turn on caching since there is no
updates on the state store on the same key.

So could we just propose "removing the caching layer inside the persistent
state store engines (i.e. RocksDB), and instead add a per-thread global
cache which will only be activated for state stores used in Streams DSL's
aggregate operator as a write deduplicator for both the state store and to
the downstream operators"?

Guozhang


On Thu, Sep 8, 2016 at 10:07 AM, Eno Thereska <en...@gmail.com>
wrote:

> There have been a couple of changes to KIP-63 since the voting started,
> after more feedback, most notably the fact that this KIP applies to the DSL
> only, and not to the Processor API.
>
> At this point I'd like to restart the voting process.
>
> Thanks
> Eno
>
> > On 31 Aug 2016, at 17:16, Jim Jagielski <ji...@jaguNET.com> wrote:
> >
> > +1
> >> On Aug 25, 2016, at 6:57 AM, Eno Thereska <en...@gmail.com>
> wrote:
> >>
> >> Hi folks,
> >>
> >> We'd like to start the vote for KIP-63. At this point the Wiki addresses
> >> all previous questions and we believe the PoC is feature-complete.
> >>
> >> Thanks
> >> Eno
> >
>
>


-- 
-- Guozhang