You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Navneeth Krishnan <re...@gmail.com> on 2020/04/02 19:08:44 UTC

Global state store: Lazy loading

Hi All,

Is there a recommend way for lazy loading the global state store. I'm using
PAPI and I have the local KV state stores in lazy load fashion so that I
don't end up loading unnecessary data. Similarly I want to get the global
state store to be loaded only when the request has the id for which the
data is not present.

Thanks

Re: Global state store: Lazy loading

Posted by "Matthias J. Sax" <mj...@apache.org>.

Global state store are designed to load data upfront from a topics. I
don't think you can "bend" it easily.

You can keep the topic empty before starting the application, and put
data into that topic on demand, and the store would pick it up
afterwards. Not sure if this would be feasible for your use case.


-Matthias

On 4/9/20 11:51 PM, Navneeth Krishnan wrote:
> Hi All,
> 
> Any suggestions on how I can achieve this?
> 
> Thanks
> 
> On Fri, Apr 3, 2020 at 12:49 AM Navneeth Krishnan <re...@gmail.com>
> wrote:
> 
>> Hi Boyang,
>>
>> Basically I don’t want to load all the states upfront. For local kv store,
>> when the very first message arrives I basically do a http request to an
>> external service and load the data.
>>
>> I can do the same for global state store but since the global states are
>> read only I couldn’t figure out a better way to do it.
>>
>> Thanks
>>
>> On Thu, Apr 2, 2020 at 1:38 PM Boyang Chen <re...@gmail.com>
>> wrote:
>>
>>> Hey Navneeth,
>>>
>>> could you clarify a bit on what you mean by `lazy load`, specifically how
>>> you make it happen with local KV store?
>>>
>>> On Thu, Apr 2, 2020 at 12:09 PM Navneeth Krishnan <
>>> reachnavneeth2@gmail.com>
>>> wrote:
>>>
>>>> Hi All,
>>>>
>>>> Is there a recommend way for lazy loading the global state store. I'm
>>> using
>>>> PAPI and I have the local KV state stores in lazy load fashion so that I
>>>> don't end up loading unnecessary data. Similarly I want to get the
>>> global
>>>> state store to be loaded only when the request has the id for which the
>>>> data is not present.
>>>>
>>>> Thanks
>>>>
>>>
>>
>

Re: Global state store: Lazy loading

Posted by Navneeth Krishnan <re...@gmail.com>.

Hi All,

Any suggestions on how I can achieve this?

Thanks

On Fri, Apr 3, 2020 at 12:49 AM Navneeth Krishnan <re...@gmail.com>
wrote:

> Hi Boyang,
>
> Basically I don’t want to load all the states upfront. For local kv store,
> when the very first message arrives I basically do a http request to an
> external service and load the data.
>
> I can do the same for global state store but since the global states are
> read only I couldn’t figure out a better way to do it.
>
> Thanks
>
> On Thu, Apr 2, 2020 at 1:38 PM Boyang Chen <re...@gmail.com>
> wrote:
>
>> Hey Navneeth,
>>
>> could you clarify a bit on what you mean by `lazy load`, specifically how
>> you make it happen with local KV store?
>>
>> On Thu, Apr 2, 2020 at 12:09 PM Navneeth Krishnan <
>> reachnavneeth2@gmail.com>
>> wrote:
>>
>> > Hi All,
>> >
>> > Is there a recommend way for lazy loading the global state store. I'm
>> using
>> > PAPI and I have the local KV state stores in lazy load fashion so that I
>> > don't end up loading unnecessary data. Similarly I want to get the
>> global
>> > state store to be loaded only when the request has the id for which the
>> > data is not present.
>> >
>> > Thanks
>> >
>>
>

Re: Global state store: Lazy loading

Posted by Navneeth Krishnan <re...@gmail.com>.

Hi Boyang,

Basically I don’t want to load all the states upfront. For local kv store,
when the very first message arrives I basically do a http request to an
external service and load the data.

I can do the same for global state store but since the global states are
read only I couldn’t figure out a better way to do it.

Thanks

On Thu, Apr 2, 2020 at 1:38 PM Boyang Chen <re...@gmail.com>
wrote:

> Hey Navneeth,
>
> could you clarify a bit on what you mean by `lazy load`, specifically how
> you make it happen with local KV store?
>
> On Thu, Apr 2, 2020 at 12:09 PM Navneeth Krishnan <
> reachnavneeth2@gmail.com>
> wrote:
>
> > Hi All,
> >
> > Is there a recommend way for lazy loading the global state store. I'm
> using
> > PAPI and I have the local KV state stores in lazy load fashion so that I
> > don't end up loading unnecessary data. Similarly I want to get the global
> > state store to be loaded only when the request has the id for which the
> > data is not present.
> >
> > Thanks
> >
>

Re: Global state store: Lazy loading

Posted by Boyang Chen <re...@gmail.com>.

Hey Navneeth,

could you clarify a bit on what you mean by `lazy load`, specifically how
you make it happen with local KV store?

On Thu, Apr 2, 2020 at 12:09 PM Navneeth Krishnan <re...@gmail.com>
wrote:

> Hi All,
>
> Is there a recommend way for lazy loading the global state store. I'm using
> PAPI and I have the local KV state stores in lazy load fashion so that I
> don't end up loading unnecessary data. Similarly I want to get the global
> state store to be loaded only when the request has the id for which the
> data is not present.
>
> Thanks
>