You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Frank Lyaruu <fl...@gmail.com> on 2017/05/17 05:08:58 UTC

Creating read-only state stores from a compacted topic

Hi Kafka people,

I'm using the low level API that often creates a simple state store based
on a (single partition, compacted) topic (It feeds all messages and simply
stores all those messages as-is). Now all these stores get their own
'changelog' topic, to back the state store.

This is fine, but in this case all those changelogs are identical to the
original topic, so that wastes quite a bit of space.

I do see an option to disableLogging, that would take care of the changelog
but then I guess the store wouldn't know to re-read the entire source topic
when the local state gets destroyed.

Is there a workaround for that? In other words, could I supply a state
store with a topic it should use to restore from?

regards, Frank

Re: Creating read-only state stores from a compacted topic

Posted by "Matthias J. Sax" <ma...@confluent.io>.
Hey,

at the moment there is no official API for that -- however, using
`KSTreamBuilder#table()` we internally do the exact some thing -- we
don't create an additional changelog topic, but use the original input
topic for that.

Thus, it might make sense to expose this as official API at Processor
API level. Would you like to create a JIRA for this? You can also work
on this if you like :)

This change would require a KIP btw:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals


-Matthias



On 5/17/17 2:37 AM, Frank Lyaruu wrote:
> Definitely! Thanks, I'll try this.
> 
> On Wed, May 17, 2017 at 10:59 AM, Damian Guy <da...@gmail.com> wrote:
> 
>> Sorry misread your question!
>> If the local state is destroyed there will be no checkpoint file and the
>> input topic will be read from the earliest offset. So it will restore all
>> state.
>>
>> On Wed, 17 May 2017 at 09:57 Damian Guy <da...@gmail.com> wrote:
>>
>>> Hi Frank,
>>>
>>> Stores.create("store")
>>>         .withKeys(Serdes.String())
>>>         .withValues(Serdes.String())
>>>         .persistent()
>>>         .disableLogging()
>>>         .build();
>>>
>>>
>>> Does that help?
>>>
>>> Thanks,
>>> Damian
>>>
>>>
>>>
>>> On Wed, 17 May 2017 at 06:09 Frank Lyaruu <fl...@gmail.com> wrote:
>>>
>>>> Hi Kafka people,
>>>>
>>>> I'm using the low level API that often creates a simple state store
>> based
>>>> on a (single partition, compacted) topic (It feeds all messages and
>> simply
>>>> stores all those messages as-is). Now all these stores get their own
>>>> 'changelog' topic, to back the state store.
>>>>
>>>> This is fine, but in this case all those changelogs are identical to the
>>>> original topic, so that wastes quite a bit of space.
>>>>
>>>> I do see an option to disableLogging, that would take care of the
>>>> changelog
>>>> but then I guess the store wouldn't know to re-read the entire source
>>>> topic
>>>> when the local state gets destroyed.
>>>>
>>>> Is there a workaround for that? In other words, could I supply a state
>>>> store with a topic it should use to restore from?
>>>>
>>>> regards, Frank
>>>>
>>>
>>
> 


Re: Creating read-only state stores from a compacted topic

Posted by Frank Lyaruu <fl...@gmail.com>.
Definitely! Thanks, I'll try this.

On Wed, May 17, 2017 at 10:59 AM, Damian Guy <da...@gmail.com> wrote:

> Sorry misread your question!
> If the local state is destroyed there will be no checkpoint file and the
> input topic will be read from the earliest offset. So it will restore all
> state.
>
> On Wed, 17 May 2017 at 09:57 Damian Guy <da...@gmail.com> wrote:
>
> > Hi Frank,
> >
> > Stores.create("store")
> >         .withKeys(Serdes.String())
> >         .withValues(Serdes.String())
> >         .persistent()
> >         .disableLogging()
> >         .build();
> >
> >
> > Does that help?
> >
> > Thanks,
> > Damian
> >
> >
> >
> > On Wed, 17 May 2017 at 06:09 Frank Lyaruu <fl...@gmail.com> wrote:
> >
> >> Hi Kafka people,
> >>
> >> I'm using the low level API that often creates a simple state store
> based
> >> on a (single partition, compacted) topic (It feeds all messages and
> simply
> >> stores all those messages as-is). Now all these stores get their own
> >> 'changelog' topic, to back the state store.
> >>
> >> This is fine, but in this case all those changelogs are identical to the
> >> original topic, so that wastes quite a bit of space.
> >>
> >> I do see an option to disableLogging, that would take care of the
> >> changelog
> >> but then I guess the store wouldn't know to re-read the entire source
> >> topic
> >> when the local state gets destroyed.
> >>
> >> Is there a workaround for that? In other words, could I supply a state
> >> store with a topic it should use to restore from?
> >>
> >> regards, Frank
> >>
> >
>

Re: Creating read-only state stores from a compacted topic

Posted by Damian Guy <da...@gmail.com>.
Sorry misread your question!
If the local state is destroyed there will be no checkpoint file and the
input topic will be read from the earliest offset. So it will restore all
state.

On Wed, 17 May 2017 at 09:57 Damian Guy <da...@gmail.com> wrote:

> Hi Frank,
>
> Stores.create("store")
>         .withKeys(Serdes.String())
>         .withValues(Serdes.String())
>         .persistent()
>         .disableLogging()
>         .build();
>
>
> Does that help?
>
> Thanks,
> Damian
>
>
>
> On Wed, 17 May 2017 at 06:09 Frank Lyaruu <fl...@gmail.com> wrote:
>
>> Hi Kafka people,
>>
>> I'm using the low level API that often creates a simple state store based
>> on a (single partition, compacted) topic (It feeds all messages and simply
>> stores all those messages as-is). Now all these stores get their own
>> 'changelog' topic, to back the state store.
>>
>> This is fine, but in this case all those changelogs are identical to the
>> original topic, so that wastes quite a bit of space.
>>
>> I do see an option to disableLogging, that would take care of the
>> changelog
>> but then I guess the store wouldn't know to re-read the entire source
>> topic
>> when the local state gets destroyed.
>>
>> Is there a workaround for that? In other words, could I supply a state
>> store with a topic it should use to restore from?
>>
>> regards, Frank
>>
>

Re: Creating read-only state stores from a compacted topic

Posted by Damian Guy <da...@gmail.com>.
Hi Frank,

Stores.create("store")
        .withKeys(Serdes.String())
        .withValues(Serdes.String())
        .persistent()
        .disableLogging()
        .build();


Does that help?

Thanks,
Damian



On Wed, 17 May 2017 at 06:09 Frank Lyaruu <fl...@gmail.com> wrote:

> Hi Kafka people,
>
> I'm using the low level API that often creates a simple state store based
> on a (single partition, compacted) topic (It feeds all messages and simply
> stores all those messages as-is). Now all these stores get their own
> 'changelog' topic, to back the state store.
>
> This is fine, but in this case all those changelogs are identical to the
> original topic, so that wastes quite a bit of space.
>
> I do see an option to disableLogging, that would take care of the changelog
> but then I guess the store wouldn't know to re-read the entire source topic
> when the local state gets destroyed.
>
> Is there a workaround for that? In other words, could I supply a state
> store with a topic it should use to restore from?
>
> regards, Frank
>