Re: Kafka per topic retention.bytes and global log.retention.bytes not working

[Raghav <ra...@gmail.com>]

We are on Kafka 10.2.1 and facing similar issue. Upgrading to 1.0 is
disruptive. Any other way this can be circumvented ?

Thanks.

On Fri, Jan 12, 2018 at 1:24 AM, Wim Van Leuven <
wim.vanleuven@highestpoint.biz> wrote:

> awesome!
>
> On Thu, 11 Jan 2018 at 23:48 Thunder Stumpges <th...@gmail.com>
> wrote:
>
> > Thanks, yes we upgraded to 1.0.0 and that has indeed fixed the issue.
> > Thanks for the pointer!
> > -Thunder
> >
> >
> > On Tue, Jan 9, 2018 at 9:50 PM Wim Van Leuven <
> > wim.vanleuven@highestpoint.biz> wrote:
> >
> > > Upgrade?
> > >
> > > On Wed, Jan 10, 2018, 00:26 Thunder Stumpges <
> thunder.stumpges@gmail.com
> > >
> > > wrote:
> > >
> > > > How would I know if we are seeing that issue? We are running 0.11.0.0
> > so
> > > we
> > > > would not have this fix.
> > > >
> > > > On Tue, Jan 9, 2018 at 11:07 AM Wim Van Leuven <
> > > > wim.vanleuven@highestpoint.biz> wrote:
> > > >
> > > > > What minor version of Kafka are you running? Might you be impacted
> by
> > > > > https://issues.apache.org/jira/browse/KAFKA-6030?
> > > > > -w
> > > > >
> > > > > On Tue, 9 Jan 2018 at 19:02 Thunder Stumpges <
> > > thunder.stumpges@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hello, I posted this on StackOverflow
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> > https://stackoverflow.com/questions/47948399/kafka-per-
> topic-retention-bytes-and-global-log-retention-bytes-not-working
> > > > > > >also
> > > > > > but haven't gotten any response.
> > > > > >
> > > > > > thanks in advance,
> > > > > > Thunder
> > > > > > __________
> > > > > >
> > > > > > We are running a 6 node cluster of kafka 0.11.0. We have set a
> > global
> > > > as
> > > > > > well as a per-topic retention in bytes, neither of which is being
> > > > > applied.
> > > > > > There are no errors that I can see in the logs, just nothing
> being
> > > > > deleted
> > > > > > (by size; the time retention does seem to be working)
> > > > > >
> > > > > > See relevant configs below:
> > > > > >
> > > > > > *./config/server.properties* :
> > > > > >
> > > > > > # global retention 75GB or 60 days, segment size 512MB
> > > > > > log.retention.bytes=75000000000
> > > > > > log.retention.check.interval.ms=60000
> > > > > >
> > > > > > log.retention.hours=1440
> > > > > >
> > > > > > log.cleanup.policy=delete
> > > > > >
> > > > > > log.segment.bytes=536870912
> > > > > >
> > > > > > *topic configuration (30GB):*
> > > > > >
> > > > > > [tstumpges@kafka-02 kafka]$ bin/kafka-topics.sh  --zookeeper
> > > > > > zk-01:2181/kafka --describe --topic stg_logtopic
> > > > > > Topic:stg_logtopic    PartitionCount:12       ReplicationFactor:3
> > > > > > Configs:retention.bytes=30000000000
> > > > > >         Topic: stg_logtopic   Partition: 0    Leader: 4
> > > > > > Replicas: 4,5,6 Isr: 4,5,6
> > > > > >         Topic: stg_logtopic   Partition: 1    Leader: 5
> > > > > > Replicas: 5,6,1 Isr: 5,1,6
> > > > > >         ...
> > > > > >
> > > > > > And, disk usage showing 910GB usage for one partition!
> > > > > >
> > > > > > [tstumpges@kafka-02 kafka]$ sudo du -s -h /data1/kafka-data/*
> > > > > > 82G     /data1/kafka-data/stg_logother3-2
> > > > > > 155G    /data1/kafka-data/stg_logother2-9
> > > > > > 169G    /data1/kafka-data/stg_logother1-6
> > > > > > 910G    /data1/kafka-data/stg_logtopic-4
> > > > > >
> > > > > > I can see there are plenty of segment log files (512MB each) in
> the
> > > > > > partition directory... what is going on?!
> > > > > >
> > > > > > Thanks in advance, Thunder
> > > > > >
> > > > >
> > > >
> > >
> >
>



-- 
Raghav