Re: Interaction of retention settings for broker and topic plus partitions

[Jason Rosenberg <jb...@squareup.com>]

Jun,

To clarify though, is it correct that a per topic limit will always
override the default limit of the same type?  (e.g. a large per-topic
retention hours vs. a small default retention hours)?

Jason

On Sat, Sep 20, 2014 at 12:28 AM, Jun Rao <ju...@gmail.com> wrote:

> That's right. The rule is that a log segment is deleted if either the size
> or the time limit is reached. Log sizes are per partition.
>
> Thanks,
>
> Jun
>
> On Thu, Sep 18, 2014 at 2:55 PM, Cory Watson <gp...@keen.io> wrote:
>
> > Hello all!
> >
> > I'm curious about the interaction of server and topic level retention
> > settings. It's not clear to me the precedence of the follow:
> >
> >    - broker's default log.retention.bytes
> >    - topic's retention.bytes (which defaults to broker's
> >    log.retention.bytes)
> >    - broker's log.retention.hours and log.retention.minutes (if both are
> >    specified then it seems to be the lower of the two, since it's when
> >    "either" is exceeded)
> >
> > It seems that the rule is that when any of these are violated then the
> log
> > segment is deleted. Is this right?
> >
> > Also, just to be clear: The log sizes in questions are for a single
> > partitions logs?
> >
> > I have a situation where my per-topic retention.bytes is very high, but
> my
> > default log.retention.hours is lower (the default @ 168 hours). It seems
> > that it's truncating at the log.retention.hours instead of the topic's
> > retention.bytes.
> >
> > Am I understanding this correctly? :)
> >
> > --
> > Cory Watson
> > Principal Infrastructure Engineer // Keen IO
> >
>

Re: Interaction of retention settings for broker and topic plus partitions

[Neha Narkhede <ne...@gmail.com>]

To clarify though, is it correct that a per topic limit will always
override the default limit of the same type?  (e.g. a large per-topic
retention hours vs. a small default retention hours)?

That's correct.

On Thu, Nov 6, 2014 at 9:34 AM, Jason Rosenberg <jb...@squareup.com> wrote:

> Jun,
>
> To clarify though, is it correct that a per topic limit will always
> override the default limit of the same type?  (e.g. a large per-topic
> retention hours vs. a small default retention hours)?
>
> Jason
>
> On Sat, Sep 20, 2014 at 12:28 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > That's right. The rule is that a log segment is deleted if either the
> size
> > or the time limit is reached. Log sizes are per partition.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Sep 18, 2014 at 2:55 PM, Cory Watson <gp...@keen.io> wrote:
> >
> > > Hello all!
> > >
> > > I'm curious about the interaction of server and topic level retention
> > > settings. It's not clear to me the precedence of the follow:
> > >
> > >    - broker's default log.retention.bytes
> > >    - topic's retention.bytes (which defaults to broker's
> > >    log.retention.bytes)
> > >    - broker's log.retention.hours and log.retention.minutes (if both
> are
> > >    specified then it seems to be the lower of the two, since it's when
> > >    "either" is exceeded)
> > >
> > > It seems that the rule is that when any of these are violated then the
> > log
> > > segment is deleted. Is this right?
> > >
> > > Also, just to be clear: The log sizes in questions are for a single
> > > partitions logs?
> > >
> > > I have a situation where my per-topic retention.bytes is very high, but
> > my
> > > default log.retention.hours is lower (the default @ 168 hours). It
> seems
> > > that it's truncating at the log.retention.hours instead of the topic's
> > > retention.bytes.
> > >
> > > Am I understanding this correctly? :)
> > >
> > > --
> > > Cory Watson
> > > Principal Infrastructure Engineer // Keen IO
> > >
> >
>