Re: Streams changelog topic retention is high

[Sameer Kumar <sa...@gmail.com>]

Thanks Damian, I worked. I changed the  StreamsConfig.WINDOW_STORE_
CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG  and was able to reduce the
retention time of the changelog topic.

-Sameer.

On Mon, Oct 30, 2017 at 9:38 PM, Damian Guy <da...@gmail.com> wrote:

> The retention for the joins is as specified above. With until set to 240
> and StreamsConfig.WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG
> set to 1 day (the default) this would be 100800000
>
> For plain key value stores, there should be no retention period as the
> topics are compacted only.
>
> On Mon, 30 Oct 2017 at 15:48 Sameer Kumar <sa...@gmail.com> wrote:
>
> > Actually I am using Key Value store, I do use join as part of my
> DAG(until
> > for the same has been set at 240 mins). The sink processor is key-value,
> is
> > there any option to control it.
> >
> > -Sameer.
> >
> > On Mon, Oct 30, 2017 at 6:33 PM, Damian Guy <da...@gmail.com>
> wrote:
> >
> > > The topics in question are both changelogs for window stores. The
> > retention
> > > period for them is calculated as the Window retention period, which is
> > the
> > > value that is passed to `JoinWindows.until(...)` (default is 1 day)
> plus
> > > the value of the config
> > > StreamsConfig.WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG (
> > > default is 1 day)
> > >
> > >
> > >
> > > On Mon, 30 Oct 2017 at 12:49 Sameer Kumar <sa...@gmail.com>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > I have configured my settings to be the following:-
> > > >
> > > > log.retention.hours=3
> > > > delete.topic.enable=true
> > > > delete.retention.ms=10800000
> > > > min.cleanable.dirty.ratio=0.20
> > > > segment.ms=180000
> > > >
> > > > Howsoever, the changelog topic created as part of stream has the
> > > > rentention.ms to be 100800000, the source topic has it to be 3
> hours.
> > > >
> > > > [root@dmpkafka6591 kafka-11-single]# bin/kafka-topics.sh --describe
> > > > --zookeeper 172.29.65.91:2181 --topic
> > > > c-7-aq7-KSTREAM-JOINOTHER-0000000019-store-changelog
> > > >     Topic:c-7-aq7-KSTREAM-JOINOTHER-0000000019-store-changelog
> > > > PartitionCount:60       ReplicationFactor:1     Configs:retention.ms
> =
> > > > 100800000,cleanup.policy=delete,compact
> > > >
> > > > Can someone please explain this behavior.
> > > >
> > > > -Sameer.
> > > >
> > >
> >
>