You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Boris Lublinsky <bo...@lightbend.com> on 2017/11/13 18:24:54 UTC

Custom state store

It looks like for the custom state store implementation the only option is to use Topology APIs.
The problem is that in the case of DSL, Kafka streams does not provide any option to create Store Builder for a custom store.
Am I missing something? 

Boris Lublinsky
FDP Architect
boris.lublinsky@lightbend.com
https://www.lightbend.com/

Re: Custom state store

Posted by "Matthias J. Sax" <ma...@confluent.io>.

If you want to plug in a store into a DSL operator like aggregate, it
must be a key-value store as the aggregation is key-based (similar it
must be a windowed-store or session-store for window/session aggregation).

Not sure what code you wrote in 0.11 -- there, is must also be
key-value/window/session store if you plug it into a DSL operator. Can
you elaborate?

Btw: we have example for 1.0.0 API here:
https://github.com/confluentinc/kafka-streams-examples/tree/4.0.x

CP 4.0.0 is not release yet, but the examples should be up-to-date for
Kafka 1.0.0 already.


-Matthias



On 11/13/17 10:49 AM, Boris Lublinsky wrote:
> Is there an example code for this somewhere?
> Also does it have to be key/value.
> In my case a store is just a state, so key is not exposed.
> It was working fine in the 11.0, but now semantics is very different
> 
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com
> https://www.lightbend.com/
> 
>> On Nov 13, 2017, at 12:45 PM, Matthias J. Sax <ma...@confluent.io> wrote:
>>
>> You can plug in a custom store via `Materialized` parameter that allows
>> to specify a custom `KeyValueBytesStoreSupplier` (and others)
>>
>>
>> -Matthias
>>
>> On 11/13/17 10:26 AM, Boris Lublinsky wrote:
>>>
>>>> On Nov 13, 2017, at 12:24 PM, Boris Lublinsky <bo...@lightbend.com> wrote:
>>>>
>>>> It looks like for the custom state store implementation the only option is to use Topology APIs.
>>>> The problem is that in the case of DSL, Kafka streams does not provide any option to create Store Builder for a custom store.
>>>> Am I missing something? 
>>>>
>>>> Boris Lublinsky
>>>> FDP Architect
>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>> https://www.lightbend.com/
>>>>
>>>
>>>
>>
> 
>

Re: Custom state store

Posted by Boris Lublinsky <bo...@lightbend.com>.

Is there an example code for this somewhere?
Also does it have to be key/value.
In my case a store is just a state, so key is not exposed.
It was working fine in the 11.0, but now semantics is very different

Boris Lublinsky
FDP Architect
boris.lublinsky@lightbend.com
https://www.lightbend.com/

> On Nov 13, 2017, at 12:45 PM, Matthias J. Sax <ma...@confluent.io> wrote:
> 
> You can plug in a custom store via `Materialized` parameter that allows
> to specify a custom `KeyValueBytesStoreSupplier` (and others)
> 
> 
> -Matthias
> 
> On 11/13/17 10:26 AM, Boris Lublinsky wrote:
>> 
>>> On Nov 13, 2017, at 12:24 PM, Boris Lublinsky <bo...@lightbend.com> wrote:
>>> 
>>> It looks like for the custom state store implementation the only option is to use Topology APIs.
>>> The problem is that in the case of DSL, Kafka streams does not provide any option to create Store Builder for a custom store.
>>> Am I missing something? 
>>> 
>>> Boris Lublinsky
>>> FDP Architect
>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>> https://www.lightbend.com/
>>> 
>> 
>> 
>

Re: Custom state store

Posted by "Matthias J. Sax" <ma...@confluent.io>.

You can plug in a custom store via `Materialized` parameter that allows
to specify a custom `KeyValueBytesStoreSupplier` (and others)


-Matthias

On 11/13/17 10:26 AM, Boris Lublinsky wrote:
> 
>> On Nov 13, 2017, at 12:24 PM, Boris Lublinsky <bo...@lightbend.com> wrote:
>>
>> It looks like for the custom state store implementation the only option is to use Topology APIs.
>> The problem is that in the case of DSL, Kafka streams does not provide any option to create Store Builder for a custom store.
>> Am I missing something? 
>>
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>> https://www.lightbend.com/
>>
> 
>

Re: Custom state store

Posted by Boris Lublinsky <bo...@lightbend.com>.

> On Nov 13, 2017, at 12:24 PM, Boris Lublinsky <bo...@lightbend.com> wrote:
> 
> It looks like for the custom state store implementation the only option is to use Topology APIs.
> The problem is that in the case of DSL, Kafka streams does not provide any option to create Store Builder for a custom store.
> Am I missing something? 
> 
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com <ma...@lightbend.com>
> https://www.lightbend.com/
>