You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by JIEFU GONG <jg...@berkeley.edu> on 2015/07/25 01:12:26 UTC

Log Deletion Behavior

Hi all,

I have a few broad questions on how log deletion works, specifically in
conjunction with the log.retention.time setting. Say I published some
messages to some topics when the configuration was originally set to
something like log.retention.hours=168 (default). If I publish these
messages successfully, then later set the configuration to something like
log.retention.minutes=1, are those logs supposed to persist for the newest
settings or the old settings? Right now my logs are refusing to delete
themselves unless I specifically mark them for deletion -- is this the
correct/anticipated/wanted behavior?

Thanks for the help!

-- 

Jiefu Gong
University of California, Berkeley | Class of 2017
B.A Computer Science | College of Letters and Sciences

jgong@berkeley.edu <el...@berkeley.edu> | (925) 400-3427

Re: Log Deletion Behavior

Posted by JIEFU GONG <jg...@berkeley.edu>.

Okay, I will look into only using the hours setting, but I think that means
the minimum a log can be stored is 1 hour right? I think last time I tried
there Kafka failed to parse decimals.
ᐧ

On Fri, Jul 24, 2015 at 6:47 PM, Mayuresh Gharat <gharatmayuresh15@gmail.com
> wrote:

> Yes. It should. Do not set other retention settings. Just use the "hours"
> settings.
> Let me know about this :)
>
> Thanks,
>
> Mayuresh
>
> On Fri, Jul 24, 2015 at 6:43 PM, JIEFU GONG <jg...@berkeley.edu> wrote:
>
> > Mayuresh, thanks for your comment. I won't be able to change these
> settings
> > until next Monday, but just so confirm you are saying that if I restart
> the
> > brokers my logs should delete themselves with respect to the newest
> > settings, correct?
> > ᐧ
> >
> > On Fri, Jul 24, 2015 at 6:29 PM, Mayuresh Gharat <
> > gharatmayuresh15@gmail.com
> > > wrote:
> >
> > > No. This should not happen. At Linkedin we just use the log retention
> > > hours. Try using that. Chang e it and bounce the broker. It should
> work.
> > > Also looking back at the config's I am not sure why we had 3 different
> > > configs for the same property :
> > >
> > > "log.retention.ms"
> > > "log.retention.minutes"
> > > "log.retention.hours"
> > >
> > > We should probably be having just the milliseconds.
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > > On Fri, Jul 24, 2015 at 4:12 PM, JIEFU GONG <jg...@berkeley.edu>
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have a few broad questions on how log deletion works, specifically
> in
> > > > conjunction with the log.retention.time setting. Say I published some
> > > > messages to some topics when the configuration was originally set to
> > > > something like log.retention.hours=168 (default). If I publish these
> > > > messages successfully, then later set the configuration to something
> > like
> > > > log.retention.minutes=1, are those logs supposed to persist for the
> > > newest
> > > > settings or the old settings? Right now my logs are refusing to
> delete
> > > > themselves unless I specifically mark them for deletion -- is this
> the
> > > > correct/anticipated/wanted behavior?
> > > >
> > > > Thanks for the help!
> > > >
> > > > --
> > > >
> > > > Jiefu Gong
> > > > University of California, Berkeley | Class of 2017
> > > > B.A Computer Science | College of Letters and Sciences
> > > >
> > > > jgong@berkeley.edu <el...@berkeley.edu> | (925) 400-3427
> > > >
> > >
> > >
> > >
> > > --
> > > -Regards,
> > > Mayuresh R. Gharat
> > > (862) 250-7125
> > >
> >
> >
> >
> > --
> >
> > Jiefu Gong
> > University of California, Berkeley | Class of 2017
> > B.A Computer Science | College of Letters and Sciences
> >
> > jgong@berkeley.edu <el...@berkeley.edu> | (925) 400-3427
> >
>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>



-- 

Jiefu Gong
University of California, Berkeley | Class of 2017
B.A Computer Science | College of Letters and Sciences

jgong@berkeley.edu <el...@berkeley.edu> | (925) 400-3427

Re: Log Deletion Behavior

Posted by Grant Henke <gh...@cloudera.com>.

Also this stackoverflow answer may help: http://stackoverflow.com/a/29672325

On Fri, Jul 24, 2015 at 9:36 PM, Grant Henke <gh...@cloudera.com> wrote:

> I would actually suggest only using the ms versions of the retention
> config. Be sure to check/set all the configs below and look at the
> documented explanations here
> http://kafka.apache.org/documentation.html#brokerconfigs.
>
> I am guessing the default log.retention.check.interval.ms of 5 minutes is
> too long for your case or you may have changed from the default
> log.cleanup.policy and it is no longer set to delete.
>
> log.retention.ms
> log.retention.check.interval.ms
> log.cleanup.policy
>
> On Fri, Jul 24, 2015 at 9:14 PM, Mayuresh Gharat <
> gharatmayuresh15@gmail.com> wrote:
>
>> To add on, the main thing here is you should be using only one of these
>> properties.
>>
>> Thanks,
>>
>> Mayuresh
>>
>> On Fri, Jul 24, 2015 at 6:47 PM, Mayuresh Gharat <
>> gharatmayuresh15@gmail.com
>> > wrote:
>>
>> > Yes. It should. Do not set other retention settings. Just use the
>> "hours"
>> > settings.
>> > Let me know about this :)
>> >
>> > Thanks,
>> >
>> > Mayuresh
>> >
>> > On Fri, Jul 24, 2015 at 6:43 PM, JIEFU GONG <jg...@berkeley.edu> wrote:
>> >
>> >> Mayuresh, thanks for your comment. I won't be able to change these
>> >> settings
>> >> until next Monday, but just so confirm you are saying that if I restart
>> >> the
>> >> brokers my logs should delete themselves with respect to the newest
>> >> settings, correct?
>> >> ᐧ
>> >>
>> >> On Fri, Jul 24, 2015 at 6:29 PM, Mayuresh Gharat <
>> >> gharatmayuresh15@gmail.com
>> >> > wrote:
>> >>
>> >> > No. This should not happen. At Linkedin we just use the log retention
>> >> > hours. Try using that. Chang e it and bounce the broker. It should
>> work.
>> >> > Also looking back at the config's I am not sure why we had 3
>> different
>> >> > configs for the same property :
>> >> >
>> >> > "log.retention.ms"
>> >> > "log.retention.minutes"
>> >> > "log.retention.hours"
>> >> >
>> >> > We should probably be having just the milliseconds.
>> >> >
>> >> > Thanks,
>> >> >
>> >> > Mayuresh
>> >> >
>> >> > On Fri, Jul 24, 2015 at 4:12 PM, JIEFU GONG <jg...@berkeley.edu>
>> wrote:
>> >> >
>> >> > > Hi all,
>> >> > >
>> >> > > I have a few broad questions on how log deletion works,
>> specifically
>> >> in
>> >> > > conjunction with the log.retention.time setting. Say I published
>> some
>> >> > > messages to some topics when the configuration was originally set
>> to
>> >> > > something like log.retention.hours=168 (default). If I publish
>> these
>> >> > > messages successfully, then later set the configuration to
>> something
>> >> like
>> >> > > log.retention.minutes=1, are those logs supposed to persist for the
>> >> > newest
>> >> > > settings or the old settings? Right now my logs are refusing to
>> delete
>> >> > > themselves unless I specifically mark them for deletion -- is this
>> the
>> >> > > correct/anticipated/wanted behavior?
>> >> > >
>> >> > > Thanks for the help!
>> >> > >
>> >> > > --
>> >> > >
>> >> > > Jiefu Gong
>> >> > > University of California, Berkeley | Class of 2017
>> >> > > B.A Computer Science | College of Letters and Sciences
>> >> > >
>> >> > > jgong@berkeley.edu <el...@berkeley.edu> | (925) 400-3427
>> >> > >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > -Regards,
>> >> > Mayuresh R. Gharat
>> >> > (862) 250-7125
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> Jiefu Gong
>> >> University of California, Berkeley | Class of 2017
>> >> B.A Computer Science | College of Letters and Sciences
>> >>
>> >> jgong@berkeley.edu <el...@berkeley.edu> | (925) 400-3427
>> >>
>> >
>> >
>> >
>> > --
>> > -Regards,
>> > Mayuresh R. Gharat
>> > (862) 250-7125
>> >
>>
>>
>>
>> --
>> -Regards,
>> Mayuresh R. Gharat
>> (862) 250-7125
>>
>
>
>
> --
> Grant Henke
> Solutions Consultant | Cloudera
> ghenke@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>



-- 
Grant Henke
Solutions Consultant | Cloudera
ghenke@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke