You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sameer Kumar <sa...@gmail.com> on 2017/10/30 12:49:09 UTC
Streams changelog topic retention is high
Hi,
I have configured my settings to be the following:-
log.retention.hours=3
delete.topic.enable=true
delete.retention.ms=10800000
min.cleanable.dirty.ratio=0.20
segment.ms=180000
Howsoever, the changelog topic created as part of stream has the
rentention.ms to be 100800000, the source topic has it to be 3 hours.
[root@dmpkafka6591 kafka-11-single]# bin/kafka-topics.sh --describe
--zookeeper 172.29.65.91:2181 --topic
c-7-aq7-KSTREAM-JOINOTHER-0000000019-store-changelog
Topic:c-7-aq7-KSTREAM-JOINOTHER-0000000019-store-changelog
PartitionCount:60 ReplicationFactor:1 Configs:retention.ms=
100800000,cleanup.policy=delete,compact
Can someone please explain this behavior.
-Sameer.
Re: Streams changelog topic retention is high
Posted by Sameer Kumar <sa...@gmail.com>.
Thanks Damian, I worked. I changed the StreamsConfig.WINDOW_STORE_
CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG and was able to reduce the
retention time of the changelog topic.
-Sameer.
On Mon, Oct 30, 2017 at 9:38 PM, Damian Guy <da...@gmail.com> wrote:
> The retention for the joins is as specified above. With until set to 240
> and StreamsConfig.WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG
> set to 1 day (the default) this would be 100800000
>
> For plain key value stores, there should be no retention period as the
> topics are compacted only.
>
> On Mon, 30 Oct 2017 at 15:48 Sameer Kumar <sa...@gmail.com> wrote:
>
> > Actually I am using Key Value store, I do use join as part of my
> DAG(until
> > for the same has been set at 240 mins). The sink processor is key-value,
> is
> > there any option to control it.
> >
> > -Sameer.
> >
> > On Mon, Oct 30, 2017 at 6:33 PM, Damian Guy <da...@gmail.com>
> wrote:
> >
> > > The topics in question are both changelogs for window stores. The
> > retention
> > > period for them is calculated as the Window retention period, which is
> > the
> > > value that is passed to `JoinWindows.until(...)` (default is 1 day)
> plus
> > > the value of the config
> > > StreamsConfig.WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG (
> > > default is 1 day)
> > >
> > >
> > >
> > > On Mon, 30 Oct 2017 at 12:49 Sameer Kumar <sa...@gmail.com>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > I have configured my settings to be the following:-
> > > >
> > > > log.retention.hours=3
> > > > delete.topic.enable=true
> > > > delete.retention.ms=10800000
> > > > min.cleanable.dirty.ratio=0.20
> > > > segment.ms=180000
> > > >
> > > > Howsoever, the changelog topic created as part of stream has the
> > > > rentention.ms to be 100800000, the source topic has it to be 3
> hours.
> > > >
> > > > [root@dmpkafka6591 kafka-11-single]# bin/kafka-topics.sh --describe
> > > > --zookeeper 172.29.65.91:2181 --topic
> > > > c-7-aq7-KSTREAM-JOINOTHER-0000000019-store-changelog
> > > > Topic:c-7-aq7-KSTREAM-JOINOTHER-0000000019-store-changelog
> > > > PartitionCount:60 ReplicationFactor:1 Configs:retention.ms
> =
> > > > 100800000,cleanup.policy=delete,compact
> > > >
> > > > Can someone please explain this behavior.
> > > >
> > > > -Sameer.
> > > >
> > >
> >
>
Re: Streams changelog topic retention is high
Posted by Damian Guy <da...@gmail.com>.
The retention for the joins is as specified above. With until set to 240
and StreamsConfig.WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG
set to 1 day (the default) this would be 100800000
For plain key value stores, there should be no retention period as the
topics are compacted only.
On Mon, 30 Oct 2017 at 15:48 Sameer Kumar <sa...@gmail.com> wrote:
> Actually I am using Key Value store, I do use join as part of my DAG(until
> for the same has been set at 240 mins). The sink processor is key-value, is
> there any option to control it.
>
> -Sameer.
>
> On Mon, Oct 30, 2017 at 6:33 PM, Damian Guy <da...@gmail.com> wrote:
>
> > The topics in question are both changelogs for window stores. The
> retention
> > period for them is calculated as the Window retention period, which is
> the
> > value that is passed to `JoinWindows.until(...)` (default is 1 day) plus
> > the value of the config
> > StreamsConfig.WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG (
> > default is 1 day)
> >
> >
> >
> > On Mon, 30 Oct 2017 at 12:49 Sameer Kumar <sa...@gmail.com>
> wrote:
> >
> > > Hi,
> > >
> > > I have configured my settings to be the following:-
> > >
> > > log.retention.hours=3
> > > delete.topic.enable=true
> > > delete.retention.ms=10800000
> > > min.cleanable.dirty.ratio=0.20
> > > segment.ms=180000
> > >
> > > Howsoever, the changelog topic created as part of stream has the
> > > rentention.ms to be 100800000, the source topic has it to be 3 hours.
> > >
> > > [root@dmpkafka6591 kafka-11-single]# bin/kafka-topics.sh --describe
> > > --zookeeper 172.29.65.91:2181 --topic
> > > c-7-aq7-KSTREAM-JOINOTHER-0000000019-store-changelog
> > > Topic:c-7-aq7-KSTREAM-JOINOTHER-0000000019-store-changelog
> > > PartitionCount:60 ReplicationFactor:1 Configs:retention.ms=
> > > 100800000,cleanup.policy=delete,compact
> > >
> > > Can someone please explain this behavior.
> > >
> > > -Sameer.
> > >
> >
>
Re: Streams changelog topic retention is high
Posted by Sameer Kumar <sa...@gmail.com>.
Actually I am using Key Value store, I do use join as part of my DAG(until
for the same has been set at 240 mins). The sink processor is key-value, is
there any option to control it.
-Sameer.
On Mon, Oct 30, 2017 at 6:33 PM, Damian Guy <da...@gmail.com> wrote:
> The topics in question are both changelogs for window stores. The retention
> period for them is calculated as the Window retention period, which is the
> value that is passed to `JoinWindows.until(...)` (default is 1 day) plus
> the value of the config
> StreamsConfig.WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG (
> default is 1 day)
>
>
>
> On Mon, 30 Oct 2017 at 12:49 Sameer Kumar <sa...@gmail.com> wrote:
>
> > Hi,
> >
> > I have configured my settings to be the following:-
> >
> > log.retention.hours=3
> > delete.topic.enable=true
> > delete.retention.ms=10800000
> > min.cleanable.dirty.ratio=0.20
> > segment.ms=180000
> >
> > Howsoever, the changelog topic created as part of stream has the
> > rentention.ms to be 100800000, the source topic has it to be 3 hours.
> >
> > [root@dmpkafka6591 kafka-11-single]# bin/kafka-topics.sh --describe
> > --zookeeper 172.29.65.91:2181 --topic
> > c-7-aq7-KSTREAM-JOINOTHER-0000000019-store-changelog
> > Topic:c-7-aq7-KSTREAM-JOINOTHER-0000000019-store-changelog
> > PartitionCount:60 ReplicationFactor:1 Configs:retention.ms=
> > 100800000,cleanup.policy=delete,compact
> >
> > Can someone please explain this behavior.
> >
> > -Sameer.
> >
>
Re: Streams changelog topic retention is high
Posted by Damian Guy <da...@gmail.com>.
The topics in question are both changelogs for window stores. The retention
period for them is calculated as the Window retention period, which is the
value that is passed to `JoinWindows.until(...)` (default is 1 day) plus
the value of the config
StreamsConfig.WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG (
default is 1 day)
On Mon, 30 Oct 2017 at 12:49 Sameer Kumar <sa...@gmail.com> wrote:
> Hi,
>
> I have configured my settings to be the following:-
>
> log.retention.hours=3
> delete.topic.enable=true
> delete.retention.ms=10800000
> min.cleanable.dirty.ratio=0.20
> segment.ms=180000
>
> Howsoever, the changelog topic created as part of stream has the
> rentention.ms to be 100800000, the source topic has it to be 3 hours.
>
> [root@dmpkafka6591 kafka-11-single]# bin/kafka-topics.sh --describe
> --zookeeper 172.29.65.91:2181 --topic
> c-7-aq7-KSTREAM-JOINOTHER-0000000019-store-changelog
> Topic:c-7-aq7-KSTREAM-JOINOTHER-0000000019-store-changelog
> PartitionCount:60 ReplicationFactor:1 Configs:retention.ms=
> 100800000,cleanup.policy=delete,compact
>
> Can someone please explain this behavior.
>
> -Sameer.
>