You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Vinay Kumar <vi...@gmail.com> on 2019/09/17 19:02:13 UTC
log.retention.bytes not working
Hi,
I have the same problem mentioned here by Neha. Attached the comment.
https://grokbase.com/t/kafka/users/138hje002v/questtion-about-log-retention-bytes
Can someone please provide how effectively to impose the retention for all
topics.
Re: log.retention.bytes not working
Posted by Thameem Ansari <th...@gmail.com>.
To reliably delete the logs you have to follow this.
In Kafka server.properties set the following properties (adjust per your need)
log.segment.bytes=10485760
log.retention.check.interval.ms=120000
log.retention.ms=600000
I believe the documentation should be clear enough to explain the priority of these parameters.
Thanks.
> On Sep 18, 2019, at 10:42 AM, Vinay Kumar <vi...@gmail.com> wrote:
>
> Yes, log.cleanup.policy is set to delete.
>
> On Wed, Sep 18, 2019, 15:12 M. Manna <ma...@gmail.com> wrote:
>
>> And is your log.cleanup.policy set to delete ?
>>
>> On Wed, 18 Sep 2019 at 06:19, Vinay Kumar <vi...@gmail.com> wrote:
>>
>>> I don't see log.retention.bytes only is not working. Even after the
>>> specified size in log.retention.bytes reached, the topic partition with
>>> segments grows much beyond.
>>>
>>> On Wed, Sep 18, 2019, 1:31 AM M. Manna <ma...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> This is a very old post. Kafka has come far.
>>>>
>>>> Are you saying that log.retention.bytes (broker leave - all topics)
>>> and/or
>>>> retention.bytes (topic level - individual topic) isn’t working for you
>> ?
>>>>
>>>> Thanks,
>>>>
>>>> On Tue, 17 Sep 2019 at 20:02, Vinay Kumar <vi...@gmail.com>
>> wrote:
>>>>
>>>>> Hi,
>>>>> I have the same problem mentioned here by Neha. Attached the comment.
>>>>>
>>>>>
>>>>>
>>>>
>>>
>> https://grokbase.com/t/kafka/users/138hje002v/questtion-about-log-retention-bytes
>>>>>
>>>>> Can someone please provide how effectively to impose the retention
>> for
>>>> all
>>>>> topics.
>>>>>
>>>>
>>>
>>
Re: log.retention.bytes not working
Posted by Vinay Kumar <vi...@gmail.com>.
Yes, log.cleanup.policy is set to delete.
On Wed, Sep 18, 2019, 15:12 M. Manna <ma...@gmail.com> wrote:
> And is your log.cleanup.policy set to delete ?
>
> On Wed, 18 Sep 2019 at 06:19, Vinay Kumar <vi...@gmail.com> wrote:
>
> > I don't see log.retention.bytes only is not working. Even after the
> > specified size in log.retention.bytes reached, the topic partition with
> > segments grows much beyond.
> >
> > On Wed, Sep 18, 2019, 1:31 AM M. Manna <ma...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > This is a very old post. Kafka has come far.
> > >
> > > Are you saying that log.retention.bytes (broker leave - all topics)
> > and/or
> > > retention.bytes (topic level - individual topic) isn’t working for you
> ?
> > >
> > > Thanks,
> > >
> > > On Tue, 17 Sep 2019 at 20:02, Vinay Kumar <vi...@gmail.com>
> wrote:
> > >
> > > > Hi,
> > > > I have the same problem mentioned here by Neha. Attached the comment.
> > > >
> > > >
> > > >
> > >
> >
> https://grokbase.com/t/kafka/users/138hje002v/questtion-about-log-retention-bytes
> > > >
> > > > Can someone please provide how effectively to impose the retention
> for
> > > all
> > > > topics.
> > > >
> > >
> >
>
Re: log.retention.bytes not working
Posted by "M. Manna" <ma...@gmail.com>.
And is your log.cleanup.policy set to delete ?
On Wed, 18 Sep 2019 at 06:19, Vinay Kumar <vi...@gmail.com> wrote:
> I don't see log.retention.bytes only is not working. Even after the
> specified size in log.retention.bytes reached, the topic partition with
> segments grows much beyond.
>
> On Wed, Sep 18, 2019, 1:31 AM M. Manna <ma...@gmail.com> wrote:
>
> > Hi,
> >
> > This is a very old post. Kafka has come far.
> >
> > Are you saying that log.retention.bytes (broker leave - all topics)
> and/or
> > retention.bytes (topic level - individual topic) isn’t working for you ?
> >
> > Thanks,
> >
> > On Tue, 17 Sep 2019 at 20:02, Vinay Kumar <vi...@gmail.com> wrote:
> >
> > > Hi,
> > > I have the same problem mentioned here by Neha. Attached the comment.
> > >
> > >
> > >
> >
> https://grokbase.com/t/kafka/users/138hje002v/questtion-about-log-retention-bytes
> > >
> > > Can someone please provide how effectively to impose the retention for
> > all
> > > topics.
> > >
> >
>
Re: log.retention.bytes not working
Posted by Vinay Kumar <vi...@gmail.com>.
I don't see log.retention.bytes only is not working. Even after the
specified size in log.retention.bytes reached, the topic partition with
segments grows much beyond.
On Wed, Sep 18, 2019, 1:31 AM M. Manna <ma...@gmail.com> wrote:
> Hi,
>
> This is a very old post. Kafka has come far.
>
> Are you saying that log.retention.bytes (broker leave - all topics) and/or
> retention.bytes (topic level - individual topic) isn’t working for you ?
>
> Thanks,
>
> On Tue, 17 Sep 2019 at 20:02, Vinay Kumar <vi...@gmail.com> wrote:
>
> > Hi,
> > I have the same problem mentioned here by Neha. Attached the comment.
> >
> >
> >
> https://grokbase.com/t/kafka/users/138hje002v/questtion-about-log-retention-bytes
> >
> > Can someone please provide how effectively to impose the retention for
> all
> > topics.
> >
>
Re: log.retention.bytes not working
Posted by "M. Manna" <ma...@gmail.com>.
Hi,
This is a very old post. Kafka has come far.
Are you saying that log.retention.bytes (broker leave - all topics) and/or
retention.bytes (topic level - individual topic) isn’t working for you ?
Thanks,
On Tue, 17 Sep 2019 at 20:02, Vinay Kumar <vi...@gmail.com> wrote:
> Hi,
> I have the same problem mentioned here by Neha. Attached the comment.
>
>
> https://grokbase.com/t/kafka/users/138hje002v/questtion-about-log-retention-bytes
>
> Can someone please provide how effectively to impose the retention for all
> topics.
>